`/srv/status` returns OK even if database is broken

Development

AI-generated summary

The /srv/status endpoint in Discourse returns “OK” even if the database is broken, as it only checks the local process and not its dependencies. michaeld and riking explained that “OK” means the Unicorn process is working, but it does not verify the status of Postgres or Redis. This is equivalent to a Kubernetes livenessProbe, not a readinessProbe. A possible alternative for monitoring is using a different endpoint, such as /about.json for sites that do not require login. As suggested by michaeld, a plugin could be used to perform more comprehensive checks, like User.find(1) and a Redis get, but this might not cover all cases, such as a broken database consistency.

1

I had an upgrade on a 2-container install go sideways this morning. Bootstrapping a new container migrated the database into a broken state (perhaps it wouldn’t have if I’d used SKIP_POST_DEPLOYMENT_MIGRATIONS=1, but that’s another issue), so that the running container got the “oops, this site is broken” message.

That much is expected, but my monitor is checking /srv/status and it merrily returns an OK even when rails is pretty broken.

Is this a bug? I really want my monitors to know if there is a problem, should I instead be pulling something else like /about.json (for sites that don’t require login).

2

How exactly was it broken? Can you be more specific? What did the homepage look like?

3

Yeah, “ok” means “unicorn is working”. You can bring Postgres and Redis down and it still says “ok” if I remember correctly.

4

I’m fairly certain that’s correct. It makes sense, it’s just not what I thought.

5

I’m not sure what the status of this is @sam @eviltrout? I vaguely remember a discussion of this in the past.

6

Yup, doesn’t check for redis or PG. I think we use a plugin that does User.find(1) and a $redis.get instead. That still does not catch @pfaffman his case but that might be a bit too much, you can’t expect this to do a complete db consistency check.
discourse/app/controllers/forums_controller.rb at main · discourse/discourse · GitHub

7

The /srv/status endpoint checks only the local process, not any dependencies. “Is the HTTP stack wedged?”, plus “am I in lame-duck?”. In Kubernetes terminology, this is the livenessProbe, not the readinessProbe.

If we want to introduce a readinessProbe, it should live at a different URL.

Probably Discourse.system_user.id instead of 1.