Unable to get Redis out of read only mode post upgrade

Installation
,
1

Hey Discourse community! This may be a quick fix, but I couldn’t find anything that solved this problem.

I recently updated my Discourse server (docker > AWS setup > EC2, postgres, Redis, SES) and have been stuck in read only mode. I’ve tried ssh’ing into my server and running the following commands

sudo ./launcher enter app
discourse disable_readonly

which returns

WARN: Redis is in a readonly state. Performed a noop
The site is now fully operable.

OR entering the rails console

sudo ./launcher enter app
rails c
Discourse.disable_readonly_mode

which returns …

WARN: Redis is in a readonly state. Performed a noop
=> true

but even after a rebuild I’m still unable to log into the Discourse server. Not super familiar with Redis, any help would be awesome!

1 Like
4

You should only be seeing the redis readonly error if redis is replying with a readonly result. Is there any chance your redis is running read only?

5

Hi @eviltrout thanks for the response! When I initially setup the Redis cluster (2 nodes) it was working fine BUT with the in app update it put my Redis server in read only.

Any guidance on how to get it out of read only mode? Thanks!

6

Redis cluster, I am a bit confused, only setup that would work is a master slave, to support active-active clustering we would need application changes.

The error you have definitely indicates its talking to a slave redis:

The following would kick a live instance off slave mode.

redis-cli 
slaveof no one

Perhaps you are talking to a slave redis instead of master?

1 Like
7

Hey @sam thanks for the response!

So I setup two clusters within a replication group so that there was a backup cluster in case the primary failed (discourse-001 and discourse-002). What’s interesting is that I think the upgrade forced a switch/swap of the clusters as discourse-001 WAS setup to the primary BUT now discourse-002 is now set as primary.

Let me dig deeper …

Here’s my setup below:

Pasted image1182×632 44.3 KB

Pasted image1174×440 34.3 KB

Pasted image1173×398 33.1 KB

8

So quick update, I was able to get out of read only mode by doing the following:

  • it looks like the upgrade switched the primary redis cluster within the replication group from -001 to -002 SO by switching to discourse-002 in the app.yml file, it pulled me out of read-only mode.

  • I tried to switching to the replication group end point instead (removing -001 or -002 from the endpoint address) to see if it’d auto-default to one or the other, BUT it failed

  • HOWEVER this means that anytime I do an upgrade, I’ll have to go into the app.yml file and switch between the redis clusters for it to get out of read-only mode

This isn’t ideal, but workable. May be something to look into down the road for future builds, thanks!

2 Likes
9

There is no reason the upgrade would cause that, what is your failover check and process?

10

hm interesting, noob question, where do I check those settings? Thanks!

11

I am really unsure about how AWS goes about this, can you ask support at AWS for some help?

12

Yeah totally, I’ll dig into this more this week and will keep you guys posted. Glad it’s back up and running and a work around for the time being, more to come!

1 Like
13

So not sure if there was a fix that came out from you guys (happy to say there was) or something weird happened last time, but the docker and discourse upgrade I just did worked find and didn’t kick me into read only mode :smile:

2 Likes
14

I don’t recognized earlier but I needed to disable read only in backups section /admin/backups/

16

Hope all is well and that you are still enjoying Discourse.

In the past few years we have done a lot to make this readonly redis mode much more fault tolerant, we have not really heard complaints recently about redis being stuck in secret read only mode.

2 Likes
17

This topic was automatically closed after 3 days. New replies are no longer allowed.