Defining DISCOURSE_S3_CDN_URL links to assets in S3 CDN URL

Yep sorry that was a typo just in my previous post. DISCOURSE_S3_BACKUP_BUCKET doesn’t set s3_bucket in GlobalSetting for me. I’m not sure how you’re running that rake task without setting DISCOURSE_S3_BUCKET.

I really appreciate your help btw and realise this isn’t your problem to solve, so thanks.

No worries is not an issue. I forgot to mention we also define USE_DB_S3_CONFIG: true in our app.yml file. infra/web.yml at master · debtcollective/infra · GitHub

And I think you are right since this changes the behaviour of how S3 buckets get defined (discourse/s3.rake at 427d54b2b00fa94474c0522eaed750452c4e7f43 · discourse/discourse · GitHub) and probably it’s a bug.

Check if setting that does the trick for you.

Dear @eatcodetravel thanks for your great post.

i am trying to set up cloudfront as you.

Should i upload asset folder to my S3 bucket or will it be automatically.

What will be happen after a discourse update ? Should i upload theme again if it will not be automatically.

thank you.

Of course.

def ensure_s3_configured!
  unless GlobalSetting.use_s3? || use_db_s3_config
    STDERR.puts "ERROR: Ensure S3 is configured in config/discourse.conf or environment vars"
    exit 1
  end
end

use_db_s3_config is saving you from having to set that extra variable. That must be a bug in global_setting.rb because I should be able to just set DISCOURSE_S3_UPLOAD_BUCKET unless there is a difference between that and DISCOURSE_S3_BUCKET, but I think you’re right that the latter is supposed to be deprecated.

Irrespective of a bug in global_setting.rb I’m still seeing an issue where Discourse is looking for assets in their usual place and not on my S3 CDN, even though I’ve declared all my variables and DISCOURSE_ENABLE_S3_UPLOADS is set to true.

There’s a task for this bundle exec rake uploads:migrate_to_s3 . Once you have configured your buckets you should run that task to move uploads to s3. S3/CDN has changed in the recent few months and documentation is not up to date, so make sure you backup and prepare in case something goes wrong.

When I was enabling this for the first time we had some downtime while we figure everything out/

1 Like

I guess this is because you are missing DISCOURSE_CDN_URL. We have it like this in our config

DISCOURSE_CDN_URL: 'https://d16zv78c963s69.cloudfront.net' # this points to the server
DISCOURSE_S3_CDN_URL: 'https://community-cdn-prod.debtcollective.org' # this points to S3 bucket

Stylesheets are fetched from the server so they use DISCOURSE_CDN_URL. JavaScript it’s uploaded to the S3 bucket and uses DISCOURSE_S3_CDN_URL. @Falco explained this to me here.

Yep I saw that. I’ve set DISCOURSE_CDN_URL but I still get the same issue. However I’m tired so maybe I missed something with all the messing around. Will pick this back up tomorrow. Thanks for the help.

Will this upload both upload files as images and assets as js files to S3? And should i use a cloudfront url or S3 is enough ? thanks for you help.

bundle exec rake uploads:migrate_to_s3 will upload only attachments to S3, this task needs to be run once. After you enable S3 new uploads will be saved in the S3 buckets.

To upload assets you need to use bundle exec rake s3:upload_assets and this needs to run after every rebuild/upgrade.

1 Like

i did what you mentioned and uploads to s3 (images etc ) are successful but but for assets i get below error

ERROR: Ensure S3 is configured in config/discourse.conf or environment vars

i set both

DISCOURSE_CDN_URL: 'https://d16zv78c963s69.cloudfront.net' # this points to the server
DISCOURSE_S3_CDN_URL: 'https://community-cdn-prod.debtcollective.org' # this points to S3 bucket

Probably this is related to USE_DB_S3_CONFIG: true. Try setting that variable to see if it works.

Here are all the variables I set related to S3/CDN

3 Likes

yeah. thanks :slightly_smiling_face:. this fixed the problem.

there is an interesting problem, discourse trying to reach asset files from wrong path. it tries to reach from brotli_asset path but there is no this path in S3 bucket.

Also it tries to reach javascripts and stylesheet files from cdn url. But there are not these files in bucket also.

1 Like

@atozsoft I believe all the answers to your issues are in this thread. I’ve literally just gone through the same setup.

In short, you need to make sure you’ve uploaded your assets once they’ve been precompiled.

3 Likes

Wait
I don’t understand one thing.

Why set up this data in ENV ?

I’ll give my example.

By setting the data in admin panels to S3 everything works properly and everything loads to S3. I do not have any settings in ENV. Is that bad ? Is it necessary if everything works ?

The second example is different because I can’t migrate the old data to S3 (only new ones work) But here entering this data in app.yml (DISCOURSE_S3_ACCESS_KEY_ID: ‘key’ etc) file doesn’t change anything and it still doesn’t work (i.e.

rake uploads:migrate_to_s3

does not work

Can you explain that to me?

Nope, the environment variables provide an extra way to configure this stuff, e.g. before you have access to the admin panel.

3 Likes

thanks for help

Maybe you’ll know where the mistake I described here Migrating uploads to S3 for 'default'... rake aborted! comes from ??

NameError: uninitialized constant FileStore::ToS3Migration::Aws

What am I supposed to do with this? Edit files this way and then rebuild app ? I don’t understand.

There is a guide now about how to configure this properly at Using Object Storage for Uploads (S3 Clones)

4 Likes