Hi there, I’m trying to migrate to S3, but my migrate_to_s3 fails. Updating the URLs in the database... Removing old optimized images... Flagging all posts containing lightboxes for rebake... 9756 posts were flagged for a rebake 1 of 16545 uploads are not migrated to S3. S3 migration failed for db 'default'. Here’s output from rake posts:missing_uploads Looking for missing uploads on: default 10 post uploads are missing. 10 uploads are missing. 9 of 108619 posts are affected. Next I did enter rails console, checked all the posts with the missing uploads (all were .mp4 btw) and deleted them. And then after trying - once again - to do the migration: Updating the URLs in the database... Removing old optimized images... Flagging all posts containing lightboxes for rebake... 5178 posts were flagged for a rebake rake aborted! FileStore::ToS3MigrationError: 1 of 16551 uploads are not migrated to S3. S3 migration failed for db 'default'. /var/www/discourse/lib/file_store/to_s3_migration.rb:138:in `raise_or_log' /var/www/discourse/lib/file_store/to_s3_migration.rb:79:in `migration_successful?' /var/www/discourse/lib/file_store/to_s3_migration.rb:389:in `migrate_to_s3' /var/www/discourse/lib/file_store/to_s3_migration.rb:65:in `migrate' /var/www/discourse/lib/tasks/uploads.rake:127:in `migrate_to_s3' /var/www/discourse/lib/tasks/uploads.rake:106:in `block in migrate_to_s3_all_sites' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:80:in `with_connection' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:90:in `each_connection' /var/www/discourse/lib/tasks/uploads.rake:104:in `migrate_to_s3_all_sites' /var/www/discourse/lib/tasks/uploads.rake:100:in `block in ' /usr/local/bin/bundle:25:in `load' /usr/local/bin/bundle:25:in ` ' Tasks: TOP => uploads:migrate_to_s3 (See full trace by running task with --trace)

Ok, so this issue has been now fixed! Here’s how I did it (with immense help of Artur from our forum) In to_s3_migration.rb, I’ve changed this: count = Upload.by_users.where("url NOT LIKE '#{base_url}%'").count if count > 0 error_message = "#{count} of #{Upload.count} uploads are not migrated to S3. #{failure_message}" raise_or_log(error_message, should_raise) success = false end To this: count = Upload.by_users.where("url NOT LIKE '#{base_url}%'").count if count > 0 failed_uploads = Upload.by_users.where("url NOT LIKE '#{base_url}%'") failed_ids = failed_uploads.map(&:id) failed_id_list = failed_ids.join(", ") error_message = "#{count} of #{Upload.count} uploads are not migrated to S3. \n List of failed IDs: \n #{failed_id_list}; #{failure_message}" raise_or_log(error_message, should_raise) success = false end And then in rails console just destroyed problematic upload. Now I’ve got issue with FileStore::ToS3MigrationError: 3 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'. But I think I’ve encountered a bigger issue - assets after being compilation are not being uploaded to S3 bucket: [Screenshot FtgHAnTj from Arc -- 2023-07-11 at 13.21.29@2x] There obviously should be asset folder but there are none. This is something that’s preventing me from putting CDN in front of it all, but I guess that’s another issue, for a different topic?

So after all of this, turned off and on again and unfortunately: Updating the URLs in the database... Removing old optimized images... Flagging all posts containing lightboxes for rebake... 5493 posts were flagged for a rebake 1 of 16551 uploads are not migrated to S3. S3 migration failed for db 'default'. What can I do with it now?

[图片] Paweł Orzech: See full trace by running task with --trace) Did you try running it again, with the above command?

Yep, let me do it once again. I’m gonna paste here updated log after it does the thing. Updating the URLs in the database... Removing old optimized images... Flagging all posts containing lightboxes for rebake... 5672 posts were flagged for a rebake rake aborted! FileStore::ToS3MigrationError: 1 of 16551 uploads are not migrated to S3. S3 migration failed for db 'default'. /var/www/discourse/lib/file_store/to_s3_migration.rb:138:in `raise_or_log' /var/www/discourse/lib/file_store/to_s3_migration.rb:79:in `migration_successful?' /var/www/discourse/lib/file_store/to_s3_migration.rb:389:in `migrate_to_s3' /var/www/discourse/lib/file_store/to_s3_migration.rb:65:in `migrate' /var/www/discourse/lib/tasks/uploads.rake:127:in `migrate_to_s3' /var/www/discourse/lib/tasks/uploads.rake:106:in `block in migrate_to_s3_all_sites' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:80:in `with_connection' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:90:in `each_connection' /var/www/discourse/lib/tasks/uploads.rake:104:in `migrate_to_s3_all_sites' /var/www/discourse/lib/tasks/uploads.rake:100:in `block in ' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling' /var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run' bin/rake:13:in ` ' /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/cli/exec.rb:58:in `load' /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/cli/exec.rb:58:in `kernel_load' /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/cli/exec.rb:23:in `run' /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/cli.rb:491:in `exec' /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run' /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command' /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch' /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/cli.rb:34:in `dispatch' /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start' /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/cli.rb:28:in `start' /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/exe/bundle:45:in `block in ' /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors' /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/exe/bundle:33:in ` ' /usr/local/bin/bundle:25:in `load' /usr/local/bin/bundle:25:in ` ' Tasks: TOP => uploads:migrate_to_s3 CC @Timelord

Unless that 1 upload is absolutely critical, here is your solution: [图片] Migrate_to_s3 fails Self-hosting You can first try to run it with: SKIP_FAILED=1 rake uploads:migrate_to_s3. If that doesn’t work edit this line here and try again

Unfortunately I did try it with skip one and then it failed in the similar way. Could try again and get that error log. That upload is definitely not critical, in any case

Just edit your local copy of migrator as suggested in the topic. The value needs to be changed from 0 to 1 (since you have just one failing) this would fool the migrator into completion except for the one problem upload.

root@vps-9d50af6e:/var/discourse# ./launcher enter app x86_64 arch detected. root@vps-9d50af6e-app:/var/www/discourse# sed -i '85s/0/100/' lib/file_store/to_s3_migration.rb root@vps-9d50af6e-app:/var/www/discourse# sed -i '77s/0/100/' lib/file_store/to_s3_migration.rb root@vps-9d50af6e-app:/var/www/discourse# SKIP_FAILED=1 rake uploads:migrate_to_s3 Please note that migrating to S3 is currently not reversible! [CTRL+c] to cancel, [ENTER] to continue Migrating uploads to S3 for 'default'... ........................................... Updating the URLs in the database... Removing old optimized images... Flagging all posts containing lightboxes for rebake... 5953 posts were flagged for a rebake 1 of 16552 uploads are not migrated to S3. S3 migration failed for db 'default'. root@vps-9d50af6e-app:/var/www/discourse# Even though - as you see before - I’ve “sed -i” these two lines in to_s3_migration.rb

So why doesn’t the skip-failed work? What could I try next? How to check what wasn’t uploaded? Could this be because of some plugins?

Bumping, maybe someone else is going to see it and - hopefully - come up with an idea what could work

I’m gonna add - what I think is an answer - but it doesn’t work great. It is something I’m gonna try in the future though: Doing fresh installation with S3 + CDN, out of the box, Moving all of the uploads from current server to S3, Restoring from backup from old server, Changing app.yml to newer one (that works with S3 + CDN). So basically I’m gonna try and trick discourse to thing that everything was in the cloud all the way. But it’s not elegant solution to have to reinstall Discourse just because migrate_to_s3 fails miserably. Also it should let me make a decision - do I need that X of Y uploads in s3? Because in that case (and when I was trying to find an answer to my question - in other cases I found) I’d happily get rid of that upload and move on.

16551个上传中有1个未迁移到S3。数据库'default'的S3迁移失败。

产品支持安装

itsbhanusharma (Bhanu Sharma) 2023 年2 月 21 日 18:40 6

除非那一次上传至关重要，否则这是您的解决方案：

话题		回复	浏览量
Migrate_to_s3 fails Self-hosting s3	25	2467	2022 年2 月 21 日
Rake uploads:migrate_from_s3 fails Self-hosting s3	31	3616	2021 年4 月 24 日
Rake uploads:migrate_to_s3 error Self-hosting s3	8	1022	2020 年1 月 6 日
Migrate_to_S3 Fails on Rebake Self-hosting	17	2908	2022 年9 月 10 日
Uploads migration to S3 fails Product Support	26	3667	2021 年5 月 7 日

16551个上传中有1个未迁移到S3。数据库'default'的S3迁移失败。

相关话题