Using Object Storage for Uploads (S3 & Clones)

Jagster · June 30, 2022, 1:44pm

What happends if you comment out DISCOURSE_CDN_URL?

Victor_Ashiedu · June 30, 2022, 4:13pm

I haven’t done that. But I have been troubleshooting and I strongly believe that I did not set up the S3 CDN correctly. I am using StackPath and I am not sure what to use in the Origin address and the Host header

Victor_Ashiedu · June 30, 2022, 4:16pm

Another thing I am not sure of is how to set up the file listing feature of my DigitalOcean’s Space bucket. Now, I am using Restrict Listing. Does this affect anything?

Victor_Ashiedu · June 30, 2022, 4:39pm

I commented out DISCOURSE_S3_CDN_URL and the site loads. This confirms that my setup of the S3 CDN is the problem but I still don’t know how to set it up correctly.

I will really appreciate any help.

md-misko · June 30, 2022, 5:50pm

Did you complete all tasks in the Configuration section?

If I remember correctly, setting DISCOURSE_S3_CDN_URL in app.yml requires that assets are stored on S3.

Jagster · June 30, 2022, 11:06pm

I don’t know much how Discourse works. I’m just another copy&pasting webmaster from PHP-world, but I’m still wondering why there would be need to declare same CDN twice.

I don’t use DISCOURSE_CDN_URL because I have DISCOURSE_S3_CDN_URL (and propably any doc I red didn’t guide to use it). So I tried.

I added DISCOURSE_CDN_URL: <cdn-url>. Rebuilding gave no errors, but I got error 502. When I removed it my forum worked again.

Still your issues can happend because of some misconfiguration somewhere else. But I like easy solutions, even when I’m wrong, and I’m blaiming situation when you are telling twice CDN for different needs. Or something.

Victor_Ashiedu · July 1, 2022, 3:55pm

DISCOURSE_CDN_URL and DISCOURSE_S3_CDN_URL are different. While DISCOURSE_CDN_URL points to your forum, DISCOURSE_S3_CDN_URL points to your S3 storage URL.

The idea is for DISCOURSE_CDN_URL to serve pullable assets like CSS while DISCOURSE_S3_CDN_URL serves pushable assets like JS, images and user uploads.

I have been able to get everything working but my S3 storage CDN. Whenever I enable that bit in the app.yml file, my forum displays blank.

A look at the browser inspect element network shows that assets delivered by DISCOURSE_S3_CDN_URL are showing error 404.

This is where I am stock! I am not sure why the S3 CDN cannot deliver the assets.

Jagster · July 1, 2022, 4:14pm

Thanks for patient explanation!

That is quite… unfamiliar for me. But as I said, I have very limited experiense. I’m using AWS as CDN thru S3/CloudFlare in WordPress but only for static files — scripts would need another setup. I reckon there is no point to compare WP and Discourse because those are fundamentally so different platforms. But I’m trying to understand principles of he concept here.

Well, I got much more now than you

EDIT: Aaaand I was totally wrong at my setup too All JS, images, etc are served thru CDN. But I couldn’t find CSS’ there.

Victor_Ashiedu · July 1, 2022, 4:21pm

Yes, they are. But thanks for the update.

Victor_Ashiedu · July 1, 2022, 4:27pm

Hi @Jagster I really will appreciate your help.

The problem I am having now is that my site is trying to load .gz.js assets from the Object Storage CDN but the files are not there. How do I force Discours to upload those .gz.js assets to the Object Storage?

Falco · July 1, 2022, 4:30pm

That’s what this bit in the guide does:

Falco:

In order to store Discourse static assets in your Object Storage add this configuration on your app.yml under the hooks section:
  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets

Victor_Ashiedu · July 1, 2022, 4:41pm

I already have that line in my app.yml. I put the above code after defining the plugins. Or do I move them above the plugin lists?

Falco · July 1, 2022, 4:45pm

That’s wrong. The after_assets_precompile YAML key must be under the hooks key.

By default we ship as:

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git

and you want

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git
  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets

Victor_Ashiedu · July 1, 2022, 4:46pm

Thank you so much, @Falco. I’ll move them now and try again.

Victor_Ashiedu · July 1, 2022, 4:47pm

But what about the other plugins? Is it okay to list them below

sudo -E -u discourse bundle exec rake s3:upload_assets ?

Falco · July 1, 2022, 4:48pm

No.

It will become:

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/docker_manager.git
  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets

YAML is tree-like data structure and white space indentation defines the structure.

Victor_Ashiedu · July 1, 2022, 5:38pm

With the guidance you (@Falco) and @pfaffman provided, I have just successfully completed this configuration.

For other community members that wish to see my configurations. Here are the screenshots. I also wrote a comprehensive guide while I was configuring all these.

I used StackPath and DigitalOcean Spaces. My guide includes a step by step of the whole configuration. If anybody needs this guide, I will be happy to post the URL when requested.

Victor_Ashiedu · July 1, 2022, 5:39pm

@Falco and @pfaffman thank you both so much for your help. This is my first time configuring Discourse but I have just completed the most difficult bits (I think!). I am very grateful!.

pfaffman · July 1, 2022, 5:40pm

So it’s working now?

Victor_Ashiedu · July 1, 2022, 5:41pm

Perfectly. It took me so long but apart from the fact that it is working, I have also learned so much about Discourse configuration!

Thank you!