So, I had a theory, which was that the spinning wheel is caused by a failure to get past some intermediate proxy. And as I wanted to post about it, the spinning wheel appeared.
So I tried to play with timeout on the proxy, no luck (my theory being “too much people are connected and polling, our proxy is using some default config that could be too low”) Then I looked at the graph, nothing egregious (as I really think haproxy should handle 5 clients…). Then I tried restarting as I did before, and no luck again.
My last attempt was to switch from 2 puma workers to 1 (puma being, if I am not wrong, the application server we use for Ruby ?), assuming some weird deadlock that get triggered only when lots of people use it (eg, something we wouldn’t have seen during test, but we would notice as more and more people use the forum, as @bmbouter noted elsewhere).
And it seems to have fixed the issue, somehow ?
I do not like magic fix I can’t explain, so for now, I am not touching to anything (and because it is late), but please ping me if it break again.
Maybe it got fixed because the pod need to be restarted, but only after some time as there is a lock somewhere that expire, or something like that (and my puma worker change is not why it got fixed).