After config change some settings not in sync on sibling nodes in multi-node discourse setup

Support
1

Hello,

In multi node setup with web_only container behind load-balancer noticed that some settings when changed are applied only on single node (probably the one to which I was connected through load-balancer when making changes). So noticed these specifically: Global notice, Globally pinned posts and now the latest, when changing theme to update custom one.

So in the end changes are represented to users only if load-balancer connect them to same node where they were made. Which leads to mixed page loads and messy site render in some cases.

Now the question. Do I need to run some config reload command in rake on all nodes after such changes or maybe some specific env variable needs to be added to container when running it, for auto-reloads/cluster-mode for config to be propagated to sibling nodes automatically?

2

Yeah, you could try the following:

su discourse -c 'bundle exec rake cache:clear'

Alternatively if you want to go into the rails console, SiteSetting.refresh! would likely do the same thing but is more specific to site settings.

3

Thanks! I had a feeling I was missing something there. Are there maybe some documentation article that mentions after which config changes such actions are required? Did a quick look at documentation articles and didn’t notice anything for that or HA setup specifically.

Edit: seems that the cache:clear is not available

docker exec -it web_only bash
root@discourse-build-web-only:/# cd /var/www/discourse
root@discourse-build-web-only:/var/www/discourse# su discourse -c 'bundle exec rake cache:clear'
rake aborted!
Don't know how to build task 'cache:clear' (See the list of available tasks with `rake --tasks`)
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.0/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
(See full trace by running task with --trace)

When checking with that --tasks I also do not see it.

4

Just checking: do you have a shared redis database between all nodes? That is essential for Discourse to work.

(some other applications work with one-redis-per-node, but trying to do that with Discourse will cause the kinds of issues you describe)

5

Yes. Redis is shared by all discourse node instances. I created the setup to be HA, so there is separate S3/Redis/PostgreSQL.

6

Do you observe any error messages akin to Global messages on xx timed out, message bus is no longer functioning correctly within /logs?

Previously, I discovered that when Redis and the message bus run on separate hosts, timeouts occur, resulting in a failure to synchronize among different Unicorn workers.

My workaround was to periodically reload the entire Unicorn server.

7

Grep’ed through logs on all nodes, not seeing such log entries.

8

Ah. After all there were those Global messages on xx timed out, message bus is no longer functioning correctly messages. But I mistakenly looked in actual logs dir. Now when looking in web interface logs error section I actually did notice entries which you mentioned. Need to get used to fact that different errors show up in different places for discourse. Still nice to have features in discourse web side.