Stylesheet cache in multi-server setup

hosting
1

Hello,

We’re running Discourse using AWS Elastic Beanstalk’s Docker environment, with a single Docker container running on each of 2 (or more) servers.

We noticed that the main stylesheet is 404ing about 50% of the time. I think this is because of the stylesheet cache implementation:

  • A rebuild of the main stylesheet is triggered by e.g. adding a new category.
  • One of the servers recompiles the stylesheet, writes it to its local filesystem, and updates the stylesheet path in the distributed cache.
  • Both servers now serve the updated stylesheet path, but only one of the servers has the updated stylesheet in its filesystem.

Is my understanding of the issue correct? If so, it seems like this could potentially be solved by writing the stylesheet to the distributed cache, or a shared storage service like S3, rather than the local filesystem.

3 Likes
2

you need Gluster or Ceph or some other shared storage for the /public/uploads directory otherwise you are hosed.

3

I see. Does that mean Discourse can’t be used on services like Elastic Beanstalk, Heroku, etc?

I’ve configured our installation to store uploads in S3, but the stylesheet cache seems to be a special case.

Would you be open to a patch that uses the distributed cache to store the compiled stylesheet?

4

I definitely think this is plugin territory, pretty sure you could monkey patch rails/discourse to work this way.

5

OK. I’ll look into that. Thanks!

6

It would be great to see this resolved in Discourse itself.

Eliminating ad-hoc local filesystem writes in favor of persisting to well-defined backing services along the lines of http://12factor.net/backing-services would allowing easier deployments and horizontal scaling across different platforms. AWS Elastic Beanstalk may be a little less common, but the ability to run on Heroku seems a good litmus test for the deployability of an app.

Also, depending on this single local filesystem seems to be throwing out much of the benefit of the docker-by-default approach that I’ve been impressed Discourse is taking.

Given Discourse already depends on Redis, that seems like a decent candidate for storing runtime-rendered stylesheets. Or since S3 is already supported for uploads, that would be another option that doesn’t introduce new dependencies.

1 Like
7

This is very temporary, we are having all sorts of nightmarish issues with this, edge cases where it is failing and so on. We are going to be dropping support for it in our next release.

I am curious to see a template that add ceph or some other distributed fs into discourse docker, it would be an interesting thing to play with and gives for way better isolation of infrastructure

8

We are having this exact same error on Heroku and it’s driving us mad. Have you ever come up with a solution?

9

We use glusterfs and it works fine for us.

10

Our solution was to pay for hosted Discourse :wink:

1 Like
11

I went with Digital Ocean and discourse_docker - we are happy now.

1 Like