Unable to get out of read only mode post upgrade


(Andrew Stroup) #1

Hey Discourse community! This may be a quick fix, but I couldn’t find anything that solved this problem.

I recently updated my Discourse server (docker > AWS setup > EC2, postgres, Redis, SES) and have been stuck in read only mode. I’ve tried ssh’ing into my server and running the following commands

sudo ./launcher enter app
discourse disable_readonly

which returns

WARN: Redis is in a readonly state. Performed a noop
The site is now fully operable.

OR entering the rails console

sudo ./launcher enter app
rails c

which returns …

WARN: Redis is in a readonly state. Performed a noop
=> true

but even after a rebuild I’m still unable to log into the Discourse server. Not super familiar with Redis, any help would be awesome!

(Jeff Atwood) #2

@sam perhaps you can comment on this?

(Andrew Stroup) #3

hey @codinghorror and @sam any thoughts on this? At a stand still as I can’t make any changes to the server AND the users can’t upload anything either :frowning:

(Robin Ward) #4

You should only be seeing the redis readonly error if redis is replying with a readonly result. Is there any chance your redis is running read only?

(Andrew Stroup) #5

Hi @eviltrout thanks for the response! When I initially setup the Redis cluster (2 nodes) it was working fine BUT with the in app update it put my Redis server in read only.

Any guidance on how to get it out of read only mode? Thanks!

(Sam Saffron) #6

Redis cluster, I am a bit confused, only setup that would work is a master slave, to support active-active clustering we would need application changes.

The error you have definitely indicates its talking to a slave redis:

The following would kick a live instance off slave mode.

slaveof no one

Perhaps you are talking to a slave redis instead of master?

(Andrew Stroup) #7

Hey @sam thanks for the response!

So I setup two clusters within a replication group so that there was a backup cluster in case the primary failed (discourse-001 and discourse-002). What’s interesting is that I think the upgrade forced a switch/swap of the clusters as discourse-001 WAS setup to the primary BUT now discourse-002 is now set as primary.

Let me dig deeper …

Here’s my setup below:

(Andrew Stroup) #8

So quick update, I was able to get out of read only mode by doing the following:

  • it looks like the upgrade switched the primary redis cluster within the replication group from -001 to -002 SO by switching to discourse-002 in the app.yml file, it pulled me out of read-only mode.

  • I tried to switching to the replication group end point instead (removing -001 or -002 from the endpoint address) to see if it’d auto-default to one or the other, BUT it failed

  • HOWEVER this means that anytime I do an upgrade, I’ll have to go into the app.yml file and switch between the redis clusters for it to get out of read-only mode

This isn’t ideal, but workable. May be something to look into down the road for future builds, thanks!

(Sam Saffron) #9

There is no reason the upgrade would cause that, what is your failover check and process?

(Andrew Stroup) #10

hm interesting, noob question, where do I check those settings? Thanks!

(Sam Saffron) #11

I am really unsure about how AWS goes about this, can you ask support at AWS for some help?

(Andrew Stroup) #12

Yeah totally, I’ll dig into this more this week and will keep you guys posted. Glad it’s back up and running and a work around for the time being, more to come!

(Andrew Stroup) #13

So not sure if there was a fix that came out from you guys (happy to say there was) or something weird happened last time, but the docker and discourse upgrade I just did worked find and didn’t kick me into read only mode :smile:

(cosy) #14

I don’t recognized earlier but I needed to disable read only in backups section /admin/backups/