Defining DISCOURSE_S3_CDN_URL links to assets in S3 CDN URL

Support
1

I had this problem before and decided that I was crazy, confused, or the database on the site was suspect, but this is on a brand new site. Also, I was on Digital Ocean spaces, so I thought it might be a problem somehow.

I’m trying again to figure out how to keep images on AWS S3 like I think the Big Boys do.

Here’s what I have in the env section:

  DISCOURSE_S3_ACCESS_KEY_ID: 'key'
  DISCOURSE_S3_SECRET_ACCESS_KEY: 'lock'
  DISCOURSE_BACKUP_LOCATION: 's3'
  DISCOURSE_S3_BUCKET: 'lc-xyz'
  DISCOURSE_S3_BUCKET_NAME: 'lc-xyz'
  DISCOURSE_S3_BACKUP_BUCKET: 'lc-xyz/backups'
  DISCOURSE_S3_UPLOAD_BUCKET: 'lc-xyz/uploads'
  DISCOURSE_S3_CDN_URL: 'https://lc-rbx.s3.amazonaws.com'

When I include the s3 cdn url the site breaks because all of the links to the assets are on S3 like https://lc-xyz.s3.amazonaws.com/assets/plugin-third-party-01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b.br.js.

Because if you define s3 cdn url Discourse looks for the assets on the s3_cdn_url.

I did a rebuild, but the assets are still missing. I can do a

rake s3:upload_assets

Is there an after_bundling_assets stanza that I could add that to? (I know about after_db_migrate and after_bundle_exec, but don’t know if those would work.)

Do assets get produced some other time? (It would seem like when themes get modified assets would change.)

If I also had a “push CDN like normal”, would that keep this from happening?

Is there some best practice that I’m missing?

1 Like
2

You can use after_assets_precompile as the hook for that rake task.

4 Likes
3

Aha! Thanks very much for that. Where does the list of those things live?

I’m still confused why this is necessary, especially since you seemed confused before. (Or maybe I should be happy that I can push all this stuff to S3, now that I know how to do it–And I think that it was you who said that one could use CloudFlare’s free CDN to front an AWS bucket.)

4

I’m trying this again. I’ve got uploads going to s3. That works.

I configured cloudflare to CDN the site. If I enter https://lc-XXX.literatehosting.com/uploads in the s3_cdn_url site setting, new uploads go to the S3 bucket and the image is linked to the CDN url and it works.

BUT if I try to set the s3_cdn_url with an ENV in app.yml (or edit config/discourse.conf by hand and sv restart unicorn, all of the assets are loaded from S3 (where they aren’t). That might be OK, but if I try to rake s3:upload_assets, it complains that S3 isn’t configured.

1 Like
5

I happened across this while figuring out some clashing settings names.

I think (and hope, because it means I understand this and didn’t write absolute nonsense in the thread linked above) that if you remove these two lines:

DISCOURSE_S3_BUCKET: 'lc-xyz'
DISCOURSE_S3_BUCKET_NAME: 'lc-xyz'

Then you can set DISCOURSE_S3_CDN_URL, and it will only be used for uploads, not assets.

3 Likes
6

I think that you do understand, and, thanks to you, I’m at least closer to understanding myself! Thanks very much!

1 Like
7

Running into this and still very confused after reading this thread…

I’m trying to server s3 uploads (not compiled assets) from CDN (Cloudfront).

If I configure “s3 cdn url” via the settings screen, it works as expected (well…except for System upload not using s3 cdn url)

However, if I configure via DISCOURSE_S3_CDN_URL and rebuild, the frontend is broken because its trying to load compiled assets from my s3 cdn url.

Seems DISCOURSE_S3_CDN_URL / s3_cdn_url should only affect uploads, and DISCOURSE_CDN_URL should only affect assets.

2 Likes
8

That is my experience too. I ended up making a plugin to set the S3_CDN_URL.

2 Likes
9

@pfaffman seems like this should be filed as a bug then, yes?

10

Perhaps. It’s not a feature likely to be used by normal self hosters,so it’s going to be a low priority. Also, I think that there is soon to be a change in how global settings and shadowed by global work, so it’ll likely get worked out then.

11

seems the same issue turn me down here
@pfaffman need your help
do you use cloudfront to achieve the custom domain cdn url or just use the public prefix of bucket objects in cdn url

12

I’ve just hit this. @pfaffman did you ever get this resolved?

I’ve got uploads working by setting the s3_cdn_url manually in site settings, but ideally need to be able to set the ENV variable globally when deploying. When I do, I get the same issue, that Discourse tries to look for assets at the s3_cdn_url which seems like a bug to me in https://github.com/discourse/discourse/blob/master/lib/content_security_policy/default.rb#L36

13

I think that I just set the value in the database.

My guess is that the solution is to find the rake task that will push the assets to S3.

14

I was wondering if any of the @team could confirm whether this is expected behaviour, that you can’t use an S3 CDN (set globally) for uploads without Discourse also looking for assets there too. It seems like unexpected behaviour unless I’m misunderstanding something.

15

Yes, that is the behavior.

Setting up S3 CDN will use it for every thing that would be a static file, be it uploads or JS assets.

1 Like
16

Here’s some information about this, I was dealing with this last month.

I fixed it by setting both variables (DISCOURSE_S3_CDN_URL and DISCOURSE_CDN_URL) and creating two CloudFront distributions, one for the uploads with origin the S3 bucket, and one for the assets with origin the server.

Here’s the code we use for this

Here’s our app.yml (we named it web.yml), we replace variables at build time infra/modules/services/discourse/web.yml at master · debtcollective/infra · GitHub

4 Likes
17

Amazing. Thanks very much. There doesn’t appear to be much documentation around this at all, and the settings imply that you can just use s3 for uploads only, which I think is what’s catching us all out.

2 Likes
18

It is in my list to write a howto on this. I hope to do it by the week end.

6 Likes
19

Yeah, once you understand you need two CloudFront distributions, makes a bit more sense. Also, remeber to upload assets to s3 after each rebuild. There’s an issue with upgrades from Docker Manager where you need to run bundle exec rake s3:upload_assets manually inside the docker container. If you do a rebuild instead it should work if you add these lines to your app.yml

hooks:
  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - 'bundle exec rake s3:upload_assets'
5 Likes
20

That task is complaining that I don’t have S3 configured. I’ve narrowed it down to this method in global_setting.rb:

def self.use_s3?
    (@use_s3 ||=
      begin
        s3_bucket &&
        s3_region && (
          s3_use_iam_profile || (s3_access_key_id && s3_secret_access_key)
        ) ? :true : :false
      end) == :true
  end

Where is GlobalSetting.s3_bucket expected to be defined? It looks to me like we need to set DISCOURSE_S3_UPLOAD_BUCKET and DISCOURSE_S3_BUCKET environment variables. What is the difference between the two?