설정 변경 후 멀티노드 Discourse 환경에서 자매 노드 간 일부 설정이 동기화되지 않음

지원
1

Hello,

In multi node setup with web_only container behind load-balancer noticed that some settings when changed are applied only on single node (probably the one to which I was connected through load-balancer when making changes). So noticed these specifically: Global notice, Globally pinned posts and now the latest, when changing theme to update custom one.

So in the end changes are represented to users only if load-balancer connect them to same node where they were made. Which leads to mixed page loads and messy site render in some cases.

Now the question. Do I need to run some config reload command in rake on all nodes after such changes or maybe some specific env variable needs to be added to container when running it, for auto-reloads/cluster-mode for config to be propagated to sibling nodes automatically?

2

Yeah, you could try the following:

su discourse -c 'bundle exec rake cache:clear'

Alternatively if you want to go into the rails console, SiteSetting.refresh! would likely do the same thing but is more specific to site settings.

3

Thanks! I had a feeling I was missing something there. Are there maybe some documentation article that mentions after which config changes such actions are required? Did a quick look at documentation articles and didn’t notice anything for that or HA setup specifically.

Edit: seems that the cache:clear is not available

docker exec -it web_only bash
root@discourse-build-web-only:/# cd /var/www/discourse
root@discourse-build-web-only:/var/www/discourse# su discourse -c 'bundle exec rake cache:clear'
rake aborted!
Don't know how to build task 'cache:clear' (See the list of available tasks with `rake --tasks`)
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.0/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
(See full trace by running task with --trace)

When checking with that --tasks I also do not see it.

4

Just checking: do you have a shared redis database between all nodes? That is essential for Discourse to work.

(some other applications work with one-redis-per-node, but trying to do that with Discourse will cause the kinds of issues you describe)

5

Yes. Redis is shared by all discourse node instances. I created the setup to be HA, so there is separate S3/Redis/PostgreSQL.

6

Do you observe any error messages akin to Global messages on xx timed out, message bus is no longer functioning correctly within /logs?

Previously, I discovered that when Redis and the message bus run on separate hosts, timeouts occur, resulting in a failure to synchronize among different Unicorn workers.

My workaround was to periodically reload the entire Unicorn server.

7

Grep’ed through logs on all nodes, not seeing such log entries.

8

Ah. After all there were those Global messages on xx timed out, message bus is no longer functioning correctly messages. But I mistakenly looked in actual logs dir. Now when looking in web interface logs error section I actually did notice entries which you mentioned. Need to get used to fact that different errors show up in different places for discourse. Still nice to have features in discourse web side.

9

안녕하세요 @david, PR을 열었습니다: fix: subscriber thread never self heals after half open tcp connection by Mubramaj · Pull Request #388 · discourse/message_bus · GitHub

이 PR은 Global messages on xx timed out 문제를 해결하고, @pangbo 님의 전체 Unicorn 서버를 주기적으로 재로드하는 임시 해결책이 필요 없도록 합니다.