I need help with moving the existing uploads to Digital Ocean Spaces (S3 is properly configured and new uploads are successfully uploaded).
I ran rake uploads:migrate_to_s3 with:
DISCOURSE_S3_ACCESS_KEY_ID="id" \
DISCOURSE_S3_SECRET_ACCESS_KEY="secret" \
DISCOURSE_S3_REGION="sfo2" \
DISCOURSE_S3_BUCKET="mybucket/uploads" \
DISCOURSE_S3_ENDPOINT="https://mybucket.sfo2.digitaloceanspaces.com" \
DRY_RUN=1 rake uploads:migrate_to_s3 --trace
but the rake is aborted with:
Seahorse::Client::NetworkingError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unspecified certificate verification error)
which indicates possible missing root certificate for DO spaces, as reported here:
Short answer is the Ruby SDK no longer bundles a SSL certificate. It relies on OpenSSL in Ruby being correctly configured. The linked issue above gives some simple work-arounds for when this is not the case.
One of the workarounds there mentions setting the
Aws.config[:ssl_ca_bundle] = '/path/to/ca-bundle.crt'
I downloaded the file to ‘/shared/ca-bundle.crt’ and set it in rails console, but I can’t seem to make this persistent (rake task fails again, and after checking the setting in rails console, it is reverted back to nil).
Full trace:
** Invoke uploads:migrate_to_s3 (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute uploads:migrate_to_s3
Checking if default already migrated...
13252 of 13261 uploads are not migrated to S3. S3 migration failed for db 'default'.
10052 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.
** Invoke posts:missing_uploads (first_time)
** Invoke environment
** Execute posts:missing_uploads
Looking for missing uploads on: default
0 post uploads are missing.
Looking for missing uploads on: default
0 post uploads are missing.
No posts require rebaking
****************************** DRY RUN ******************************
Migrating uploads to S3 for 'default'...
Uploading files to S3...
- Listing local files
.............. => 14025 files
- Listing S3 files
rake aborted!
Seahorse::Client::NetworkingError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unspecified certificate verification error)
/usr/local/lib/ruby/2.6.0/net/protocol.rb:44:in `connect_nonblock'
/usr/local/lib/ruby/2.6.0/net/protocol.rb:44:in `ssl_socket_connect'
/usr/local/lib/ruby/2.6.0/net/http.rb:996:in `connect'
/usr/local/lib/ruby/2.6.0/net/http.rb:930:in `do_start'
/usr/local/lib/ruby/2.6.0/net/http.rb:925:in `start'
/usr/local/lib/ruby/2.6.0/delegate.rb:83:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/seahorse/client/net_http/connection_pool.rb:299:in `start_session'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/seahorse/client/net_http/connection_pool.rb:97:in `session_for'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/seahorse/client/net_http/handler.rb:121:in `session'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/seahorse/client/net_http/handler.rb:73:in `transmit'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/seahorse/client/net_http/handler.rb:47:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/seahorse/client/plugins/content_length.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.60.1/lib/aws-sdk-s3/plugins/s3_signer.rb:111:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.60.1/lib/aws-sdk-s3/plugins/s3_signer.rb:59:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.60.1/lib/aws-sdk-s3/plugins/s3_host_id.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/xml/error_handler.rb:8:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/transfer_encoding.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.60.1/lib/aws-sdk-s3/plugins/s3_signer.rb:89:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.60.1/lib/aws-sdk-s3/plugins/redirects.rb:18:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/retry_errors.rb:177:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/retry_errors.rb:208:in `retry_request'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/retry_errors.rb:191:in `retry_if_possible'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/retry_errors.rb:179:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/retry_errors.rb:208:in `retry_request'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/retry_errors.rb:191:in `retry_if_possible'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/retry_errors.rb:179:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/retry_errors.rb:208:in `retry_request'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/retry_errors.rb:191:in `retry_if_possible'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/retry_errors.rb:179:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.60.1/lib/aws-sdk-s3/plugins/dualstack.rb:34:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.60.1/lib/aws-sdk-s3/plugins/accelerate.rb:50:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.60.1/lib/aws-sdk-s3/plugins/md5s.rb:30:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.60.1/lib/aws-sdk-s3/plugins/bucket_name_restrictions.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.60.1/lib/aws-sdk-s3/plugins/expect_100_continue.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.60.1/lib/aws-sdk-s3/plugins/bucket_dns.rb:33:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.60.1/lib/aws-sdk-s3/plugins/bucket_arn.rb:47:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/rest/handler.rb:8:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/user_agent.rb:11:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/seahorse/client/plugins/endpoint.rb:45:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/param_validator.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.60.1/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.60.1/lib/aws-sdk-s3/plugins/dualstack.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.60.1/lib/aws-sdk-s3/plugins/accelerate.rb:35:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/seahorse/client/plugins/response_target.rb:23:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/seahorse/client/request.rb:70:in `send_request'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.60.1/lib/aws-sdk-s3/client.rb:6637:in `list_objects_v2'
/var/www/discourse/lib/file_store/to_s3_migration.rb:178:in `block in migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:177:in `loop'
/var/www/discourse/lib/file_store/to_s3_migration.rb:177:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:61:in `migrate'
/var/www/discourse/lib/tasks/uploads.rake:239:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:218:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.0.7/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.0.7/lib/rails_multisite/connection_management.rb:73:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:216:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:212:in `block in <top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:281:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:281:in `execute'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/usr/local/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:80:in `run'
bin/rake:13:in `<top (required)>'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.1/lib/bundler/cli/exec.rb:63:in `load'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.1/lib/bundler/cli/exec.rb:63:in `kernel_load'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.1/lib/bundler/cli/exec.rb:28:in `run'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.1/lib/bundler/cli.rb:476:in `exec'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.1/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.1/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.1/lib/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.1/lib/bundler/cli.rb:30:in `dispatch'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.1/lib/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.1/lib/bundler/cli.rb:24:in `start'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.1/exe/bundle:46:in `block in <top (required)>'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.1/lib/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.1/exe/bundle:34:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_to_s3
, it should only include location, without the bucket name prefixed (the same as defined in admin—site settings—s3 endpoint):