I do not seem to have wiki edit permission, but I did successfully use a different provider.
OVHcloud
Service name: Object Storage
Regions correspond to data centers which are identified by a three letters code. If you don’t know where you created your bucket, check in the object storage tab in your customer portal.
Initially both in my test version but I went to prod with only backups since the prod machine has a ton of space anyway.
It’s very compatible S3, but there are some things missing like lifecycle rules to delete old files or move them to cold storage which are being actively worked on by OVH. It works fine to serve files though.
So for backups I just used Discourseʼs option to delete old backups on its own.
This is a really disappointing and completely unhelpful response. What exactly is the issue? Linking to a support document that changes means that no one can actually tell what the heck the “awful brokenness” mentioned in this thread here.
You mention “metadata” and that the CDN “doesn’t know about it”. What metadata? It would be helpful to know what isn’t working.
I just wanted to add to this post of mine with instructions on how to use iDrive’s E2.
There appears to be a thing that iDrive has recently switched on that causes access keys that are assigned only to one bucket to fail unless a bucket auth check is bypassed.
You can bypass this when using rclone with no_check_bucket = true in the rclone.conf file, but I’m not sure such an ENV setting exists for Discourse building.
As a result, with iDrive E2, you currently must use a key that has access to write to all your buckets, rather than just one.
Why would someone else know the exact issue for a provider they don’t use?
Anyway, it looks like we’re almost there with Cloudflare R2. Per:
When I entered all the info in the Web UI new uploads went to S3 storage correctly, and backups are going to S3 correctly. Current uploads did not move obviously.
So I then went to the app.yml and entered this info:
## This set of lines allows R2 S3 hosted files to be uploaded and downloaded..
DISCOURSE_CDN_URL: https://eufiles.technospider.com
DISCOURSE_USE_S3: true
DISCOURSE_S3_ENDPOINT: https://randomnumber.r2.cloudflarestorage.com
DISCOURSE_S3_CDN_URL: https://eufiles.technospider.com
DISCOURSE_S3_BACKUP_BUCKET: exotics-unlimited-backups
DISCOURSE_INCLUDE_S3_UPLOADS_IN_BACKUPS: true
DISCOURSE_BACKUP_LOCATION: s3
DISCOURSE_S3_BUCKET: exotics-unlimited
DISCOURSE_S3_REGION: auto
DISCOURSE_S3_ACCESS_KEY_ID: randomnumbers
DISCOURSE_S3_SECRET_ACCESS_KEY: randomnumbers
DISCOURSE_S3_INSTALL_CORS_RULE: false
## The Docker container is stateless; all data is stored in /shared
volumes:
- volume:
host: /var/discourse/shared/standalone
guest: /shared
- volume:
host: /var/discourse/shared/standalone/log/var-log
guest: /var/log
## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
after_code:
- exec:
cd: $home/plugins
cmd:
- git clone https://github.com/discourse/docker_manager.git
# - git clone https://github.com/discourse/discourse-subscriptions.git
- git clone https://github.com/discourse/discourse-follow.git
- git clone https://github.com/discourse/discourse-solved.git
- git clone https://github.com/communiteq/discourse-private-topics.git
# - git clone https://github.com/discourse/discourse-assign.git
- git clone https://github.com/tknospdr/discourse-auto-remove-group.git
- git clone https://github.com/discourse/discourse-topic-voting.git
- git clone https://github.com/discourse/discourse-livestream.git
# - git clone https://github.com/discourse/discourse-calendar.git
- git clone https://github.com/jannolii/discourse-topic-trade-buttons.git
## - git clone https://github.com/tknospdr/force-tag-group-order.git
## Hooks for S3
after_assets_precompile:
- exec:
cd: $home
cmd:
- sudo -E -u discourse bundle exec rake s3:upload_assets
- sudo -E -u discourse bundle exec rake s3:expire_missing_assets
After a rebuild my site was broken due to a bunch of missing files, so I tried to rake them to the server and got this error for each file:
root@talk-app:/var/www/discourse# rake uploads:migrate_to_s3
Please note that migrating to S3 is currently not reversible!
[CTRL+c] to cancel, [ENTER] to continue
Migrating uploads to S3 for 'default'...
Uploading files to S3...
- Listing local files
=> 31 files
- Listing S3 files
. => 4 files
- Syncing files to S3
#<Thread:0x00007ff89dcbcb20 /var/www/discourse/lib/file_store/to_s3_migration.rb:212 run> terminated with exception (report_on_exception is true):
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.226.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': You can only specify one non-default checksum at a time. (Aws::S3::Errors::InvalidRequest)
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/dualstack.rb:21:in `call'
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/accelerate.rb:43:in `call'
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.226.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:169:in `call'
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.226.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.226.0/lib/aws-sdk-core/plugins/invocation_id.rb:16:in `call'
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.226.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.226.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.226.0/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.226.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.226.0/lib/seahorse/client/plugins/response_target.rb:24:in `call'
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.226.0/lib/aws-sdk-core/plugins/telemetry.rb:39:in `block in call'
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.226.0/lib/aws-sdk-core/telemetry/no_op.rb:29:in `in_span'
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.226.0/lib/aws-sdk-core/plugins/telemetry.rb:53:in `span_wrapper'
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.226.0/lib/aws-sdk-core/plugins/telemetry.rb:39:in `call'
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.226.0/lib/seahorse/client/request.rb:72:in `send_request'
from /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/client.rb:17315:in `put_object'
from /var/www/discourse/lib/file_store/to_s3_migration.rb:215:in `block (2 levels) in migrate_to_s3'
Would love some advice to get this finished up one way or the other.
The missing files are likely assets, so you need the rake task that pushes those to s3 (s3:upload_assets – it’s near the top)
But your errors are likely due to the new aws s3 library breaking a bunch of services. So you’ll need to contrive to downgrade the aws gem or use another service.
I think there may be a topic about how to do that. I’ve done it on a site or two, but am not sure how or where it’s been documented
I need to run that even though that’s part of the changes I put in the app.yml file? I thought that was there to automatically push the files during a rebuild.
You shouldn’t, but those are the things that would make your site be “broken” (or does “broken” mean “images are missing” to you?) Uploading images with migrate_to_s3 just affects images.
I can’t quite explain how the assets were all uploaded already (what was broken on the site? How did those get uploaded if your S3 setup is broken?)
Since everything looks like it’s there in my bucket, there’s probably some env variable that needs to be updated to make everything work right.
If we can crack this we can say R2 is ready for prime time.
Where is “extra-locales” supposed to be? I don’t see it under ‘public’ or ‘assets’ even inside the container. Where should I look?
Sorry about the dumb questions, but this is a totally new venture for me.
Is WHAT my CDN or my bucket?
With the leading dashes, and all the single and double quotes? I don’t have any of those in my app.yml file. Should I add those and rebuild to test?
This is exactly how my file looks, indented 2 spaces each:
## This set of lines allows R2 S3 hosted files to be uploaded and downloaded..
DISCOURSE_CDN_URL: https://eufiles.technospider.com
DISCOURSE_USE_S3: true
DISCOURSE_S3_ENDPOINT: https://randomnumber.r2.cloudflarestorage.com
DISCOURSE_S3_CDN_URL: https://eufiles.technospider.com
DISCOURSE_S3_BACKUP_BUCKET: exotics-unlimited-backups
DISCOURSE_INCLUDE_S3_UPLOADS_IN_BACKUPS: true
DISCOURSE_BACKUP_LOCATION: s3
DISCOURSE_S3_BUCKET: exotics-unlimited
DISCOURSE_S3_REGION: auto
DISCOURSE_S3_ACCESS_KEY_ID: randomnumbers
DISCOURSE_S3_SECRET_ACCESS_KEY: randomnumbers
DISCOURSE_S3_INSTALL_CORS_RULE: false
It worked when I had the settings entered via the web UI, then when I put everything into the app.yml file I got what you saw when you went to the site.
Here’s what I see when I access one of your assets:
This object does not exist or is not publicly accessible at this URL. Check the URL of the object that you're looking for or contact the owner to enable Public access.
You can see if https://eufiles.technospider.com/extra-locales/ca382c69f8e6b85162b2ba58f2ce100bfe741966/en/mf.js?__ws=eu.technospider.com works if you change the hostname to your cloudflare endpoint
Can you find that file in your bucket? Can you access it?
Don’t know but that’s a URL I copied from the developer tools in my browser.
So first find that file in your bucket and see if it’s there at all and then you can go about figuring out why the CDN can’t find it.
No. I use a different tool for configuration, but those are the ENV settings that I’m pretty sure worked for me at one time.
This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Error>
<Code>InvalidArgument</Code>
<Message>Authorization</Message>
</Error>
A bucket is that ”main directory” you’ve created and where all files are. CDN is plenty of servers all around the world and those get copy of that bucket. Your is indentified using the url you gave when that connection was created, as i.e. cdn.example.com.
It seems that with R2 the CDN is automatically created if you give the bucket a custom domain. That was a little confusing as I didn’t have to actually do anything with CDN.
This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Error>
<Code>InvalidArgument</Code>
<Message>Authorization</Message>
</Error>
