Can't enable Uploads in Digital Ocean Spaces

[lucasbasquerotto]

I followed the tutorial about setting up Uploads in Digital Ocean Spaces :

(Superseded) Set up backup and image uploads to DigitalOcean Spaces

This topic is superseded by Configure an S3 compatible object storage provider for uploads.

I did it in a new discourse instance in a $5 droplet.

Here is what my settings look like:

After enabling uploads in s3 (spaces), I went to my profile page and tried to upload a image, but received the error Bucket already exists:

Additional info:

1) Discourse 2.3.0.beta10 - https://github.com/discourse/discourse version cbaa9c4fcaa09b6e36ec04e6d05cb6bcecac33d5

2) The spaces in my Digital Ocean account have no buckets (from the topic I linked above, I assume that the buckets are created automatically).

3) I did a rebuild and still didn’t work (the rebuild worked, but the upload didn’t).

4) If I uncheck the option to enable s3 uploads the uploads work again (locally).

5) If I use environment variables in the app.yml file (tested in a brand new droplet, in the first launch), like DISCOURSE_ENABLE_S3_UPLOADS, DISCOURSE_S3_ACCESS_KEY_ID, etc, I receive the error Aws::S3::Errors::BucketAlreadyExists: Bucket already exists (that I assume is the same).

Show details

INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
rake aborted!
Aws::S3::Errors::BucketAlreadyExists: Bucket already exists
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/plugins/location_constraint.rb:16:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/plugins/dualstack.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/seahorse/client/plugins/response_target.rb:23:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/seahorse/client/request.rb:70:in `send_request'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/client.rb:802:in `create_bucket'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/bucket.rb:250:in `create'
/var/www/discourse/lib/s3_helper.rb:256:in `s3_bucket'
/var/www/discourse/lib/s3_helper.rb:34:in `upload'
/var/www/discourse/lib/file_store/s3_store.rb:55:in `store_file'
/var/www/discourse/lib/file_store/s3_store.rb:30:in `store_optimized_image'
/var/www/discourse/app/models/optimized_image.rb:114:in `block (2 levels) in create_for'
/var/www/discourse/app/models/optimized_image.rb:113:in `open'
/var/www/discourse/app/models/optimized_image.rb:113:in `block in create_for'
/var/www/discourse/app/models/optimized_image.rb:29:in `block (2 levels) in lock'
/var/www/discourse/lib/distributed_mutex.rb:42:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:12:in `synchronize'
/var/www/discourse/app/models/optimized_image.rb:28:in `block in lock'
/var/www/discourse/lib/distributed_mutex.rb:42:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:12:in `synchronize'
/var/www/discourse/app/models/optimized_image.rb:27:in `lock'
/var/www/discourse/app/models/optimized_image.rb:64:in `create_for'
/var/www/discourse/lib/site_icon_manager.rb:28:in `block in ensure_optimized!'
/var/www/discourse/lib/site_icon_manager.rb:24:in `each'
/var/www/discourse/lib/site_icon_manager.rb:24:in `ensure_optimized!'
/var/www/discourse/lib/tasks/db.rake:43:in `block in <top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate

6) Making a local backup work, but enabling s3 backup (to spaces) also gives an error.

shared/app/log/rails/unicorn.stdout.log

...
pg_dump: executing SEQUENCE SET onceoff_logs_id_seq
pg_dump: executing SEQUENCE SET optimized_images_id_seq
pg_dump: executing SEQUENCE SET permalinks_id_seq
Unpausing sidekiq...
Finalizing backup...
Creating archive: discourse-2019-06-06-013103-v20190529002752.tar.gz
Making sure archive does not already exist...
Creating empty archive...
Archiving data dump...
Archiving uploads...
Removing tmp '/var/www/discourse/tmp/backups/default/2019-06-06-013103' directory...
Gzipping archive, this may take a while...
Uploading archive...
EXCEPTION: 
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/plugins/dualstack.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/plugins/accelerate.rb:35:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/seahorse/client/plugins/response_target.rb:23:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/seahorse/client/request.rb:70:in `send_request'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/client.rb:5907:in `put_object'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/file_uploader.rb:42:in `block in put_object'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/file_uploader.rb:48:in `block in open_file'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/file_uploader.rb:48:in `open'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/file_uploader.rb:48:in `open_file'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/file_uploader.rb:41:in `put_object'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/file_uploader.rb:34:in `upload'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/customizations/object.rb:308:in `upload_file'
/var/www/discourse/lib/backup_restore/s3_backup_store.rb:44:in `upload_file'
/var/www/discourse/lib/backup_restore/backuper.rb:268:in `upload_archive'
/var/www/discourse/lib/backup_restore/backuper.rb:51:in `run'
/var/www/discourse/lib/backup_restore/backup_restore.rb:169:in `block in start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:166:in `fork'
/var/www/discourse/lib/backup_restore/backup_restore.rb:166:in `start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:20:in `backup!'
/var/www/discourse/app/controllers/admin/backups_controller.rb:40:in `create'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/base.rb:194:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:132:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/callbacks.rb:41:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/rescue.rb:22:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/base.rb:134:in `process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionview-5.2.3/lib/action_view/rendering.rb:32:in `process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.0.2/lib/mini_profiler/profiling_methods.rb:104:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal.rb:191:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal.rb:252:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:840:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-protection-2.0.5/lib/rack/protection/frame_options.rb:31:in `call'
/var/www/discourse/lib/middleware/omniauth_bypass_middleware.rb:32:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/tempfile_reaper.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/conditional_get.rb:38:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'
/var/www/discourse/lib/content_security_policy/middleware.rb:12:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:220:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:232:in `context'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:226:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:98:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/logster-2.3.0/lib/logster/middleware/reporter.rb:30:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:38:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:28:in `call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:18:in `call'
/var/www/discourse/config/initializers/100-silence_logger.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/var/www/discourse/lib/middleware/enforce_hostname.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/method_override.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.0.2/lib/mini_profiler/profiler.rb:281:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-2.2.0/lib/message_bus/rack/middleware.rb:57:in `call'
/var/www/discourse/lib/middleware/request_tracker.rb:163:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:524:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:605:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:700:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:548:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:144:in `start'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/bin/unicorn:128:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `<main>'
Deleting old backups...
Cleaning stuff up...
Removing archive from local storage...
Removing '.tar' leftovers...
Marking backup as finished...
Refreshing disk stats...
Notifying 'lucasbasquerotto' of the end of the backup...
Finished!
[FAILED]

Is there someone that knows what is causing it and what I can do to make it work?

[codinghorror (Jeff Atwood)]

Can we repro this @tshenry

[tshenry (Taylor)]

I ran this by @rishabh and he spotted the issue right away (thanks Rishabh !).

The name you enter for the s3_upload_bucket setting must be globally unique. In your case “uploads” has already been taken by another Digital Ocean customer. To test if a name is available, you can go to https://cloud.digitalocean.com/spaces/new and enter the name into the field at the bottom of the page:

I’ll update the guide with a note about the name needing to be unique.

[lucasbasquerotto]

@tshenry

The name you enter for the s3_upload_bucket setting must be globally unique

I thought that was based on the buckets of the account, but that makes sense, because the bucket identifier is accessed without the account user/organization name.

That said, the error Bucket already exists was misleading (at least for me). I thought it (the code) was saying that the bucket existed under my account, because it tries to create it with my credentials (if the error was Bucket name already exists under another account or something along those lines I would have solved it immediatelly; in any case, I don’t think the error message was generated by discourse itself).

Anyway, thanks for the explanation!