Sidekiq, using elasticache, not processing queued jobs since discourse upgrade

Installation
1

I recently upgraded my docker instance of discourse running in an ubuntu ec2. I used launcher rebuild for the upgrade. I previously had version 2.4.0 and I upgraded to 2.6.0.beta1 (I also upgraded again to 2.6.0.beta2). I am using elasticache instead of redis and it had been working without a problem before the upgrade. Immediately after upgrade I noticed that sidekiq was not sending out emails. I restarted the app and cleared the cache and it seemed to be working, but that was a false positive.

Observed indications of malfunction

  • Sidekiq is scheduling and enqueuing jobs at a reasonably steady pace (as indicated in the sidekiq dashboard), but it is not running any
  • The admin page is complaining about sidekiq not running
  • The logs show several instances of Sidekiq heartbeat test failed, restarting
  • Of course: emails are not going out to users

Things I have tried (after reading posts here and SO)

  • Successfully sent a test email with discourse-doctor to verify that part of the flow is working
  • Verified at the sidekiq dashboard that the footer shows the correct elasticache instance
  • Verified that the docker logs are not indicating an error (ok: run: redis: (pid XX) 0s)
  • Cleared the cache and restarted discourse (./launcher restart app)
  • Stopped the app, cleared the cache, started the app
  • Rebuilt the app again

Steps I haven’t tried

  • Creating a new elasticache instance
  • Mothballing this installation and creating a clean one

Has anybody experienced this and have any suggestions?
Are there any logs I have missed?
Is there a way I can coerce sidekiq to process its jobs?

2

At some point discourse started requiring redis 5, I believe what version of redis does your elasticache support? I thought that there was a clear message on the logs, but could be wrong.

I’ll check my site that’s using elasticache to be sure that I’m not having trouble. I’m fairly certain that this site uses elasticache.

2 Likes
3

That is useful information, thank you. I am on version 3 of redis so looks like I am going to be doing an update tonight.

I thought that there was a clear message on the logs

There is the distinct possibility that I am not looking in the correct logs, too. Neither /logs nor ./launcher logs app provide any indication of it, so maybe there is another place I should be looking?

1 Like
4 
/var/discourse/shared/logs/rails/production.log

if my memory and fingers worked.

1 Like
5

Really close. I found it at:

/var/discourse/shared/web-only/log/rails/production.log

Very clear message, once I found it though :+1:t2::

You are connecting to Redis v3.2.10, Sidekiq requires Redis v4.0.0 or greater

I upgraded to Redis 5 and it processed my backlogged queue. All good now! Thank you!

2 Likes
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.