Hello, We’ve had our internal Discourse site for around 3 months now and suddenly it has gone into read only mode. Based on other topics (Stuck in 'Read Only' Mode) I have already tried the following: Login to docker instance: ./launcher enter app Login to rails: rails c Disable read only: Discourse.disable_readonly_mode(Discourse::USER_READONLY_MODE_KEY) Quit rails: quit Exit container: exit After doing this, our application seems to come out of read only mode, then goes back to read only mode. I tried to rebuild a container but now I am getting the following error: "Caused by: PG::ReadOnlySqlTransaction: ERROR: cannot execute ALTER TABLE in a read-only transaction /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-mini-profiler-1.0.0/lib/patches/db/pg.rb:92:in `async_exec' /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-mini-profiler-1.0.0/lib/patches/db/pg.rb:92:in `async_exec'" I guess because it’s stuck in read only mode for some reason? Current setup is: 4 containers (2 cannot rebuild based on above, 2 are still running from 2 weeks ago) AWS Elasticache Redis RDS PostgreSQL Regards, amoncadot

Hi Jeff, Thanks for the compliment. It’s nice to hear that the co-founder of Discourse/Stackoverflow considers our environment as enterprise even before it has been released :wink: Problem solved. The issue was that Amazon Aurora was being used as RDS and by default this creates a cluster with two database instances inside - one primary and one replica. Sometime yesterday an auto failover occurred and within our app.yml under the DISCOURSE_DB_HOST: parameter I had specified the DATABASE endpoint. Not the CLUSTER endpoint. The failover made the database endpoint specified in app.yml a read only replica, thus Discourse being locked into Read Only mode. If anyone is running a similar setup: EC2 instances with docker containers Redis ElastiCache Amazon Aurora RDS (PostgreSQL underneath) Check that /var/discourse/containers/app.yml contains: DISCOURSE_DB_HOST: RDS Cluster Endpoint (Go to RDS > Clusters > Access your cluster > Under Cluster endpoint is the endpoint you need to specify) DISCOURSE_DB_PORT: RDS Cluster Port DISCOURSE_REDIS_HOST: ElastiCache Primary Endpoint (Go to ElastiCache > Redis > Toggle the “play” shape button beside your Redis Cluster Name > Under Primary Endpoint is the endpoint you need to specify) DISCOURSE_REDIS_PORT: Redis Cluster Port Hope this helps someone! @codinghorror Is there a way to run ./launcher rebuild app without pulling down the latest codebase from Discourse?

Hello, Just followed the steps via Inheriting discourse install - need some assistance - #17 by jerdog and still no luck. I have two containers still running while the other two are offline. My worry is that if I stop those two containers then I won’t be able to log back into Discourse. I’d like to fix this while those two containers are still running. Kind regards, amoncadot

Can you check whether you’re out of disk space?

Hi, The web servers are not out of disk space and RDS is fine also. Regards, amoncadot

It says that Redis is in a read only state: I, [2018-06-19T18:01:55.777804 #13] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' Preformatted textNo connection to db, unable to retrieve site settings! (normal when running db:create) WARN: Redis is in a readonly state.' Performed a noop Failed to report error: Connection lost (ECONNRESET) 2 Dropping undeliverable message: ERR Error running script (call to f_b06356ba4628144e123b652c99605b873107c9be): @user_script:14: @user_script: 14: -READONLY You can't write against a read only slave. I have rebooted my Elasticache and failed over… yet Redis remains in a read only state… any ideas?

[image] amoncadot: You can’t write against a read only slave. Is your ElastiCache setup Multi-AZ? That message suggests you are connecting to a secondary node in a cluster. Double check that the hostname you are using is the Primary Endpoint of the cluster.

Yes ElastiCache is Multi-AZ. Setup is: [image] I’ve just tried to use an entirely different Redis cache and again the build failed. My app.yml specifies the primary endpoint of the cluster. So far I have no containers running, just postgreSQL and redis cache. I am going to try and use a snapshot of RDS this morning with a new Redis cache but if that fails I am not sure what else I can do since I have no containers running to access the UI. Why did Discourse suddenly go into read only mode without manual intervention?

[image] amoncadot: Why did Discourse suddenly go into read only mode without manual intervention? The site can switch to read-only mode when databases return errors like “Redis is in a readonly state”. There are multiple types of READONLY depending on the trigger. Discourse.disable_readonly_mode(Discourse::USER_READONLY_MODE_KEY) will only turn off one, you have to pass the other keys to turn off the other types.

Ah okay. When I had containers running, I disabled all three modes/keys you listed and it temporarily removed read only access and then returned back immediately. Hence why I have now moved onto trying to rebuild a different cache. I had seen that the keys solution had worked for other people but for some reason it did not work for our application.

Note that this is wildly on the enterprise side of complex setups, so there’s a limited amount we can help here.

AWS installation stuck in Read only mode

Product Support

itsbhanusharma (Bhanu Sharma) June 20, 2018, 7:24am 12

You can pin the build number in your yml

Topic		Replies	Views
Unable to get Redis out of read only mode post upgrade Self-hosting docker , read-only	12	6412	March 27, 2019
Stuck in 'Read Only' Mode Product Support read-only	16	8819	December 31, 2018
Site automatically exits read-only mode when enabled by 'discourse enable_readonly' Development read-only	9	757	February 29, 2024
Read-only mode please! Feature read-only	20	7935	May 7, 2015
This site is in read only mode Product Support	2	295	April 25, 2025

AWS installation stuck in Read only mode

Related topics