I’d like to open a discussion on the best practices for performing core maintenance tasks on a Discourse instance while minimizing or eliminating downtime.
Tasks like changing critical resource settings (e.g., UNICORN_WORKERS, DISCOURSE_SIDEKIQ_WORKERS, DISCOURSE_DB_POOL) or applying major updates typically require a launcher rebuild app which can take a significant amount of time, sometimes 30 minutes or more.
My question is: What are the recommended strategies for system administrators to perform these essential updates and configuration changes with the least amount of user-facing downtime?
Are there any advanced techniques, like blue/green deployments or other zero-downtime deployment strategies, that are supported or recommended for Discourse? Or is the standard rebuild process the only supported method, and the focus should be on optimizing the rebuild time itself?
I’m interested in hearing from anyone who has experience managing large or high-traffic instances and what their workflow looks like for maintenance.
If you have a two container install, the new container builds while the old one runs. Downtime is just the amount of time it takes to launch the new container. The only issue is that you need enough ram to build a container while the other runs.
If you want zero down time then you need a load balancer that keeps the old container running until the new one has fully started. Then you shut down the old container and do the post update migrations.
Discourse is so stable this is pretty unnecessary for most installs (but I guess you might consider it for very high availability requirements or if you are hosting others?!)
I don’t think I’ve had a single outage in 7 years due to a production “glitch” …
The riskiest moments in a Discourse’s life is always at rebuild.
the two container setup gives you the ability to bootstrap a new build before committing to it though that won’t catch some runtime errors of course.
The issue is that if your migrations have run, you might need to commit to the new build and so you would usually try to track down and fix the source of those errors rather than roll back.