Cloudflare R2: Navigating Setup and Handling Configuration Errors

[pfaffman (Jay Pfaffman)]

@kuaza

after_assets_precompile:

That line is not indented properly. It should have 2 spaces exactly like after_code

[kuaza]

Is this a problem? because some settings start without spaces. and when I compile it doesn’t give any error. Example:

[pfaffman (Jay Pfaffman)]

That all looks fine. What does the after_assets_precompile part that I asked about look like?

[kuaza]

[pfaffman (Jay Pfaffman)]

@pfaffman

That line is not indented properly. It should have 2 spaces exactly like after_code

You need to insert two spaces in front of “after_assets_precompile”.

[kuaza]

I will do this and let you know. Thank you so much for your time <3

[kuaza]

When I apply this, it gives the following error and it does not load, the site cannot be reached:

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets failed with return #<Process::Status: pid 4803 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"cd"=>"$home", "cmd"=>["sudo -E -u discourse bundle exec rake s3:upload_assets", "sudo -E -u discourse bundle exec rake s3:expire_missing_assets"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.

./discourse-doctor I ran this, then recompiled, got the same error.

When I enter the container and run the following codes, I get an error:

rake uploads:migrate_to_s3

Error:

rake uploads:migrate_to_s3 --trace
Plugin name is 'DiscourseMatheditor', but plugin directory is named 'discourse-matheditor'
Plugin name is 'discourse-post-voting', but plugin directory is named 'discourse-question-answer'
Plugin name is 'discourse-topic-voting', but plugin directory is named 'discourse-voting'
** Invoke uploads:migrate_to_s3 (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute uploads:migrate_to_s3
Please note that migrating to S3 is currently not reversible!
[CTRL+c] to cancel, [ENTER] to continue

Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Running the migration, this may take a while...
rake aborted!
FileStore::ToS3MigrationError: Some uploads could not be migrated to the new scheme. You need to fix this manually. (FileStore::ToS3MigrationError)
/var/www/discourse/lib/file_store/to_s3_migration.rb:156:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:59:in `migrate'
/var/www/discourse/lib/tasks/uploads.rake:126:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:106:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-6.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-6.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-6.0.0/lib/rails_multisite/connection_management.rb:21:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:104:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:100:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `block in execute'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `execute'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:188:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:188:in `invoke_task'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `block (2 levels) in top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `block in top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:147:in `run_with_threads'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:132:in `top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:83:in `block in run'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:214:in `standard_exception_handling'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:80:in `run'
bin/rake:13:in `<top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:58:in `load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:58:in `kernel_load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:23:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:451:in `exec'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:34:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:28:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/exe/bundle:28:in `block in <top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/exe/bundle:20:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => uploads:migrate_to_s3

Some similar unresolved issues:

S3 error trying to run rake task

I installed my Discourse, added my S3 creds and added the relevant S3 code into the hooks When rebuilding, I get this error: FAILED -------------------- Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets failed with return #<Process::Status: pid 1532 exit 1> Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn' exec failed with the params {"cd"=>"$home", "cmd"=>["sudo -E -u discourse bundle exec…

How to debug S3 uploads

For a brief moment I had S3 uploads to a bucket working. Then I followed the several posts here to make sure it is all correct and now all uploads go local. I cant see any clues in the error log. How does one go about debugging this complex setup? One post suggests you have to do it with mostly environment variables, but the other does not, and the admin panel seems to have all the S3 parameters necessary. I am basically confused. I have double checked all the values for S3 in the system.

Forum rendered unusable

Mmm, I’d noted previously that someone else had talked about putting stuff in the wrong section, then I went ahead and did the same thing anyway, eek (!) So after reading more carefully, I still seem to have an issue, I have; hooks: after_code: - exec: cd: $home/plugins cmd: - git clone https://github.com/discourse/docker_manager.git - git clone https://github.com/discourse/discourse-whos-online - git clone https://github.com/discourse/discou…

Defining DISCOURSE_S3_CDN_URL links to assets in S3 CDN URL

What am I supposed to do with this? Edit files this way and then rebuild app ? I don’t understand.

[Firepup650 (Firepup Sixfifty)]

@kuaza

Some uploads could not be migrated to the new scheme. You need to fix this manually

Do any of the other topics you mentioned happen to have instructions for finishing the uploads manually?

@kuaza

it gives the following error

That’s just the final fail layer, the actual errors from that would be farther up as it mentions:

@kuaza

please scroll up and look for earlier error messages, there may be more than one.

[kuaza]

@Firepup650

Do any of the other topics you mentioned happen to have instructions for finishing the uploads manually?

No I did not see. However, there must be a reason why it doesn’t add it after compilation and we should be able to see this error. Maybe I can take a look if it is stated how to see this.

[kuaza]

I wanted to try again. I added the necessary S3 connections to app.yml and recomplied them. this time, i removed after_assets_precompile. All urls changed to cdn and style files were not loaded. When I manually enter the container and try it: rake s3:upload_assets, I get the following error.

Attempting to apply ASSETS S3 CORS ruleset in bucket cloudflarebucketname.
rake aborted!
Aws::S3::Errors::AccessDenied: Access Denied (Aws::S3::Errors::AccessDenied)

---

If I try this: rake uploads:migrate_to_s3 --trace

The error I got:

Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Running the migration, this may take a while...
rake aborted!
FileStore::ToS3MigrationError: Some uploads could not be migrated to the new scheme. You need to fix this manually. (FileStore::ToS3MigrationError)

All error message for “rake s3:upload_assets”:

root@domain-app:/var/www/discourse# rake s3:upload_assets --trace
Plugin name is 'DiscourseMatheditor', but plugin directory is named 'discourse-matheditor'
Plugin name is 'discourse-post-voting', but plugin directory is named 'discourse-question-answer'
Plugin name is 'discourse-topic-voting', but plugin directory is named 'discourse-voting'
** Invoke s3:upload_assets (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke s3:ensure_cors_rules (first_time)
** Invoke environment
** Execute s3:ensure_cors_rules
Installing CORS rules...
Attempting to apply ASSETS S3 CORS ruleset in bucket cloudflarebucketname.
rake aborted!
Aws::S3::Errors::AccessDenied: Access Denied (Aws::S3::Errors::AccessDenied)
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/dualstack.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/accelerate.rb:43:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/response_target.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/request.rb:72:in `send_request'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/client.rb:5424:in `get_bucket_cors'
/var/www/discourse/lib/s3_helper.rb:385:in `fetch_bucket_cors_rules'
/var/www/discourse/lib/s3_helper.rb:166:in `ensure_cors!'
/var/www/discourse/lib/s3_cors_rulesets.rb:60:in `sync'
/var/www/discourse/lib/tasks/s3.rake:184:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `block in execute'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `execute'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:243:in `block in invoke_prerequisites'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:241:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:241:in `invoke_prerequisites'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:188:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:188:in `invoke_task'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `block (2 levels) in top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `block in top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:147:in `run_with_threads'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:132:in `top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:83:in `block in run'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:214:in `standard_exception_handling'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:80:in `run'
bin/rake:13:in `<top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:58:in `load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:58:in `kernel_load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:23:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:451:in `exec'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:34:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:28:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/exe/bundle:28:in `block in <top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/exe/bundle:20:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => s3:upload_assets => s3:ensure_cors_rules

It’s hard to get out of this situation.

[kuaza]

Something tells me that the S3 codes run after compilation only support Amazon S3. What are you thinking?

[pfaffman (Jay Pfaffman)]

It works for me. There’s a setting to skip the cors rules, I think. You might look at Configure an S3 compatible object storage provider for uploads to see what it is.

[kuaza]

this?: DISCOURSE_S3_INSTALL_CORS_RULE: false

[kuaza]

This worked great, but there is one problem. theme-javascripts files are not uploaded to S3. Naturally, the design of the site is damaged. Only the assets folder is loaded. What can we do to upload other design files?

[pfaffman (Jay Pfaffman)]

This is the last time I’ll tell you that your include the part that calls the rake task to upload to s3.

Good luck.

[kuaza]

What am I missing? The code below is attached and when I rebuild it loads the assets files but not the theme files.

@kuaza

after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets
          - sudo -E -u discourse bundle exec rake s3:expire_missing_assets

---

When I run this by entering the container: rake uploads:migrate_to_s3 I get the following error:

Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
1001 posts were flagged for a rebake
rake aborted!
FileStore::ToS3MigrationError: 9 of 3769 uploads are not migrated to S3. S3 migration failed for db 'default'. (FileStore::ToS3MigrationError)

My app.yml file is as follows. I turned off this setting because it did not upload the theme-javascripts and stylesheets: DISCOURSE_CDN_URL.

I’ll try to open it again at midnight and compile it, but I don’t know what I missed. I searched to see what else you might mean by this, but I couldn’t find anything different: “the rake task to upload to s3”

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: 'auto'
  DISCOURSE_S3_ENDPOINT: 'https://xxxxx.eu.r2.cloudflarestorage.com'
  DISCOURSE_S3_ACCESS_KEY_ID: 'xxxxx'
  DISCOURSE_S3_SECRET_ACCESS_KEY: 'xxxxx'
  DISCOURSE_S3_CDN_URL: 'https://cdn.xxxxx.com'
  #DISCOURSE_CDN_URL: 'https://cdn.xxxxx.com'
  DISCOURSE_S3_BUCKET: 'cloudflarer2xxxxx'
  #DISCOURSE_S3_BACKUP_BUCKET: 'cloudflarer2xxxxxbackups'
  #DISCOURSE_BACKUP_LOCATION: 's3'
  DISCOURSE_INCLUDE_S3_UPLOADS_IN_BACKUPS: true
  DISCOURSE_S3_INSTALL_CORS_RULE: false

I’m sorry I wasted your time, but it would be nice if the user documentation included these issues. Now we know that r2 works. Just add the following two settings:

DISCOURSE_S3_REGION: 'auto'
DISCOURSE_S3_INSTALL_CORS_RULE: false

We can move the images and assets folder on the site to the S3 area. There is only the problem that the theme files and style files are not transferred and I will try it again at midnight. I hope I will be successful and I will mark this issue as solved, of course, after explaining the solution.

Note: Postgresql server is separate, I do not use the one on localhost. I guess there is no problem with this.
Thank you to everyone who took the time.

[pfaffman (Jay Pfaffman)]

@kuaza

rake s3:upload_assets

That is the command to run to upload assets. My guess is that you still have something wrong that’s keeping assets from grin uploaded.

[kuaza]

Good, if I’m not doing anything wrong then I need to find the problem. When I enter the container and run the “rake s3:upload_assets” code, everything seems fine. Even if I turn this setting on or off, it does not load the style files (theme-javascripts and stylesheets): “DISCOURSE_CDN_URL: ‘https://cdn.xxxxx.com’”.

rake s3:upload_assets --trace
Plugin name is 'DiscourseMatheditor', but plugin directory is named 'discourse-matheditor'
Plugin name is 'discourse-post-voting', but plugin directory is named 'discourse-question-answer'
Plugin name is 'discourse-topic-voting', but plugin directory is named 'discourse-voting'
** Invoke s3:upload_assets (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke s3:ensure_cors_rules (first_time)
** Invoke environment
** Execute s3:ensure_cors_rules
Installing CORS rules...
skipping
** Execute s3:upload_assets Skipping: assets/break_string-cc617154cd04790e.js
....
....
....

As I said, if I’m not doing something wrong, I try to find the error and how to fix it. If there is any improvement, I will add it here, thank you.