Yo @ariznaf
Woke up this morning after being off the net for 12 hours and switched back to our socket-only.yml configuration and everything is back to normal again.
So, at least on this end of the expansive discourse-verse, all is well in two container, nginx reverse proxy to unix socket land again.
We had switched to the nginx front end configuration maybe six hours before the anomaly (was noticed) and all was fine.
Based on this helpful hint from @riking (as always, much appreciated Kane)
The various resized avatar images aren’t included in a backup, just the originals. It’ll take some time for the scheduled job to go through and generate the resized versions of every avatar.
My best guess is that when we did the switch to ngnix we did not notice any issues because the many avatar images were already cached and the regeneration process had not ended; so over time, the cache expired for those images and the anomaly started to appear.
So I drop off the net (socket-only.yml container is still running in the background, inactve) for 12 hours, wake up in the morning, and sidekiq had done it’s magic overnight (here) as @riking (great support BTW Kane, on every topic here on meta).
This scenario seems to confirm what @riking suggested.
Honestly, the more we use Discourse, the more we like it. The hiccups and anomalies are very interesting and the two-container configuration is really great.
Our containers currently looks like this:
# ls -l containers
-rw-r--r-- 1 discourse root 1124 Apr 15 11:29 data.yml
-rw-r--r-- 1 discourse root 3939 Apr 16 07:45 socket-only.yml
-rw-r--r-- 1 discourse root 3784 Apr 16 07:28 socket.yml
-rw-r--r-- 1 discourse root 3921 Apr 15 11:50 web-only.yml
What I like about this is even when we see an issue, for example this avatar regeneration anomaly, we can easily switch back-and-forth from socket-only.yml to web-only.yml.
In this case, we switched back to web-only(during this regeneration) and switched back after the process was done (because all the containers are still running). When we perform a container rebuild, we can just switch between these containers and config very easily.
Coming from two decades of running a LAMP forum, we are more and more impressed with Discourse, on the sys admin side.
Sidebar (Editorial);
Of course, it is way above my page-grade here on meta, but I think the basic two container configuration (without the reverse proxy) should be the default, as it is very easy to setup and we gain much more from this configuration than any perceived “penalty” for having two yml files.
