More details on how the Redis cache is utilized?

Dev
1

Hey there!

I’ve been performing some load tests on a Discourse instance and I noticed that when repeatedly fetching the same comment thread over and over the Redis cache hit rate went down, rather than up, which was a bit unexpected (for a mixture of reads/writes we saw a cache hit rate of up to 85%, for 100% reads it dipped down to as low as 22%).

I’ve done some searching around in the codebase and on the forums here, and how exactly the Redis cache is utilized is a bit unclear to me. The README states this:

We use Redis as a cache and for transient data.

I used redis-cli to dump the commands that were sent to the Redis cache during the aforementioned load test, and I primarily saw “get” commands for scheduled jobs and for keys prefixed with “__mb_backlog_id_n_” (I believe this is referring to MessageBus stuff).

I have the following questions:

  • Is there an “easy” way for me to search the codebase for which bits of data are cached in Redis? I’d love to be able to answer these questions on my own, but unfortunately I’m not super familiar with Ruby on Rails applications (or Ruby in general, really). :slight_smile:
  • Does being logged-in/logged-out impact cache hit rates? For reference, the load test mentioned above was using an admin API key.
  • Are frequently queried/fairly static data such as post contents cached in Redis? Or is Redis primarily used for job scheduling and background processing with Sidekiq and whatnot?

Thanks in advance!

3 Likes
2

This is the main thing here. The most aggressive cache happens for anonymous requests, so I’d suggest retrying the load test with some anon robots.

There are a handful of grepable methods, like Discourse.cache.fetch and DistributedCache.new.

We do cache some infrequent config blobs, but the approach to topics is mostly caching the whole response for the anons, allowing the app to craft a response while barely touching the DB.

Redis is heavily used for Sidekiq and MessageBus.

7 Likes
3

Awesome, thanks for the super helpful response!

1 Like
4

I just re-ran the load test but with anonymous requests this time around, and we saw a huge performance improvement! Previously we were only able to hit around 25 requests per second on a single host, now we can hit 380! The Redis cache hit rate also went up from ~22% to ~66%. :slight_smile:

Just thought I’d circle back with the results in case anyone was curious.

Thanks again for the help!

1 Like
5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.