We recently had to bring our Discourse forum on prem, and overall had a pretty painless experience. We have noticed, though, a pretty massive (10-15x) increase in time spent in Redis per transaction. This is the last three days:
We aren’t quite sure what to make of this. Overall the hosts are still pretty snappy at 64ms, but previously we were sitting at around 40ms. This, translates, though, into some real slowdowns on the front end, with our page load time increasing over a second on average.
There are a number of things that are different about these installs, including the version of Discourse - previously we were on 1.8, now we’re on 2.0.3 - and I am not sure how alarmed I should be about these numbers. Most of the changes are in the hosting environment- we are using Discourse’s official image. There has been some custom work done to integrate discourse into our internal auth, but outside of that all of our other plugins are the same.
So, my questions are:
Are these numbers alarming? (I think they are at the very least concerning)
What’s your (Discourse team, any interested party) immediate intuition on where to start investigating/troubleshooting the increase?
Before we migrated, a user had to explicitly create an account in Discourse, and had to ‘log in’ to Discourse, even if already logged into their service account. Our auth changes integrate into Discourse’s SSO and when a user visits the forum, an account is automatically created, or they are auto-logged-in if they’re already logged into the service. This has created a surge in both new users and logins.
Is this redis inside the default Discourse docker container, self hosted Redis elsewhere, or Elasticache Redis? If using Elasticache Redis can you share the instance type?
Prometheus exporter plugin for Discourse can give you redis time per page type, like on the topic page and on the category page. Your graph can give you this? That way we can compare numbers.
Alarming indeed.
Here in Meta (topics show controller) running in AWS Elasticache t2.medium we get around 20ms for p99 and 10ms median.
I agree, I am not casting a ton of suspicion on the message bus, but since a lot of time is spent there is demonstrative of the change. This is from the week before we migrated:
Fair. So to clarify, since most of this increase in redis time is in the message-bus, and with similar results to meta in TopicsController#show, maybe the increase in redis isn’t the source of the front end time?
a slightly slower message bus should not result in slower page load time. It’s for background tasks.