16551개 업로드 중 1개가 S3로 마이그레이션되지 않았습니다. 'default' DB의 S3 마이그레이션 실패

pawelorzech · 2월 21, 2023, 11:56오전

안녕하세요,

S3로 마이그레이션을 시도하고 있는데, migrate_to_s3가 실패합니다.

Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
9756 posts were flagged for a rebake
1 of 16545 uploads are not migrated to S3. S3 migration failed for db 'default'.

*rake posts:missing_uploads*의 출력 결과는 다음과 같습니다.

Looking for missing uploads on: default

10 post uploads are missing.

10 uploads are missing.
9 of 108619 posts are affected.

이후 rails 콘솔에 진입하여 업로드가 누락된 모든 게시글(참고로 모두 .mp4 파일이었습니다)을 확인하고 삭제했습니다.

그리고 다시 마이그레이션을 시도한 결과:


Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
5178 posts were flagged for a rebake
rake aborted!
FileStore::ToS3MigrationError: 1 of 16551 uploads are not migrated to S3. S3 migration failed for db 'default'.
/var/www/discourse/lib/file_store/to_s3_migration.rb:138:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:79:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:389:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:65:in `migrate'
/var/www/discourse/lib/tasks/uploads.rake:127:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:106:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:80:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:90:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:104:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:100:in `block in <main>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => uploads:migrate_to_s3
(See full trace by running task with --trace)

pawelorzech · 2월 21, 2023, 5:54오후

결국 이 모든 과정을 거치고 다시 재부팅했는데, 불행히도:

Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
5493 posts were flagged for a rebake
1 of 16551 uploads are not migrated to S3. S3 migration failed for db 'default'.

이제 어떻게 해야 할까요?

Timelord · 2월 21, 2023, 6:08오후

위 명령어로 다시 실행해 보셨나요?

pawelorzech · 2월 21, 2023, 6:08오후

네, 한 번 더 해볼게요. 작업이 끝나면 업데이트된 로그를 여기에 붙여넣을 거예요.

Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
5672 posts were flagged for a rebake
rake aborted!
FileStore::ToS3MigrationError: 1 of 16551 uploads are not migrated to S3. S3 migration failed for db 'default'.
/var/www/discourse/lib/file_store/to_s3_migration.rb:138:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:79:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:389:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:65:in `migrate'
/var/www/discourse/lib/tasks/uploads.rake:127:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:106:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:80:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:90:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:104:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:100:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
bin/rake:13:in `<top (required)>'
/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/cli/exec.rb:58:in `load'
/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/cli/exec.rb:58:in `kernel_load'
/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/cli/exec.rb:23:in `run'
/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/cli.rb:491:in `exec'
/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/cli.rb:34:in `dispatch'
/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/cli.rb:28:in `start'
/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/exe/bundle:45:in `block in <top (required)>'
/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.4.4/exe/bundle:33:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => uploads:migrate_to_s3

@Timelord 님에게 공유합니다

itsbhanusharma · 2월 21, 2023, 6:40오후

그 1개의 업로드가 절대적으로 중요하지 않다면, 아래가 해결책입니다:

pawelorzech · 2월 21, 2023, 6:41오후

불행히도 skip one 옵션을 사용하여 시도해 보았는데, 비슷한 방식으로 실패했습니다. 다시 시도해서 해당 오류 로그를 가져올 수 있습니다.

어쨌든 그 업로드는 절대 중요하지 않습니다.

itsbhanusharma · 2월 21, 2023, 6:43오후

토픽에서 제안한 대로 로컬의 migrator를 수정하기만 하면 됩니다. 값이 0에서 1로 변경되어야 합니다(실패한 항목이 하나뿐이므로). 이렇게 하면 migrator가 완료된 것으로 인식하게 되며, 문제 있는 업로드 하나만 제외됩니다.

pawelorzech · 2월 21, 2023, 6:46오후

root@vps-9d50af6e:/var/discourse# ./launcher enter app
x86_64 arch detected.
root@vps-9d50af6e-app:/var/www/discourse# sed -i '85s/0/100/' lib/file_store/to_s3_migration.rb
root@vps-9d50af6e-app:/var/www/discourse# sed -i '77s/0/100/' lib/file_store/to_s3_migration.rb
root@vps-9d50af6e-app:/var/www/discourse# SKIP_FAILED=1 rake uploads:migrate_to_s3
Please note that migrating to S3 is currently not reversible! 
[CTRL+c] to cancel, [ENTER] to continue

Migrating uploads to S3 for 'default'...

...........................................
Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
5953 posts were flagged for a rebake
1 of 16552 uploads are not migrated to S3. S3 migration failed for db 'default'.
root@vps-9d50af6e-app:/var/www/discourse#

앞에서 볼 수 있듯이 to_s3_migration.rb 파일의 해당 두 줄을 sed -i 명령으로 수정했는데도 불구하고,

pawelorzech · 2월 22, 2023, 8:11오전

그렇다면 왜 --skip-failed 옵션이 작동하지 않는 걸까요? 다음에 시도해 볼 수 있는 방법은 무엇일까요? 업로드되지 않은 항목을 확인하는 방법은 무엇일까요? 이것이 일부 플러그인 때문일 수도 있을까요?

pawelorzech · 2월 22, 2023, 11:30오후

글을 올립니다. 다른 누군가가 보고 - hopefully - 작동할 수 있는 아이디어를 내주길 바랍니다.

pawelorzech · 2월 23, 2023, 12:49오후

내가 생각하기에 정답일 수 있는 내용을 하나 더 추가해 보겠습니다. 다만 완벽하게 작동하지는 않습니다. 그래도 앞으로 시도해 볼 만한 방법입니다:

S3 + CDN을 사용한 신규 설치를 기본 상태로 수행하고,
현재 서버의 모든 업로드 파일을 S3로 이동하고,
기존 서버의 백업에서 복원하고,
app.yml을 S3 + CDN과 호환되는 최신 버전으로 변경합니다.

결국 Discourse가 처음부터 모든 것이 클라우드에 있었던 것처럼 속여보려는 시도입니다. 하지만 migrate_to_s3가 완전히 실패하기 때문에 Discourse를 다시 설치해야 하는 것은 우아한 해결책이 아닙니다.

또한 이 방법이라면 S3에 있는 Y개 업로드 중 X개가 실제로 필요한지 여부를 결정할 수 있을 것입니다. 만약 그렇게 할 수 있다면 (그리고 제 질문에 대한 답을 찾던 중 다른 사례에서 발견했던 것처럼) 해당 업로드를 기꺼이 삭제하고 넘어갈 수 있을 것입니다.

pawelorzech · 2월 26, 2023, 3:36오후

이 이슈에 대한 추가 코멘트: migrate_to_s3가 실패할 때마다 나머지 이미지 업로드 단계에서 실패하지 않습니다. 이로 인해 매 실패하는 마이그레이션마다 내 데이터의 99%가 S3에 업로드되어 막대한 양의 데이터가 소모될 수 있으며(실제로도 그랬습니다).

또한 migrate_to_s3가 되돌릴 수 없는 작업이기 때문에, 제 개인 HDD와 S3에 동시에 파일이 존재하게 됩니다. HDD의 데이터를 삭제할 수 없는 이유는 여전히 게시물에 링크되어 있기 때문입니다. 되돌릴 수 없으므로 기존 구조로 되돌아갈 수도 없고, 모든 데이터가 S3에 있는 것이 아니기 때문에 CDN을 사이에 끼워 넣는 것도 불가능합니다.

pawelorzech · 3월 25, 2023, 10:41오후

거의 한 달 만에 다시 올립니다. 아직도 해결하지 못했는데, 이제 누군가 도움이 될 만한 정보를 알고 있을지도 모르겠네요.

pawelorzech · 5월 7, 2023, 1:16오전

오늘 다시 시도해 보았습니다. 새로 설치한 뒤 백업에서 복원했습니다. 어디를 봐도 같은 오류가 나네요. 어떤 미디어 1개가 누락되어 있는지 확인할 방법이 없습니다. 1 of 16545 uploads are not migrated to S3. S3 migration failed for db 'default'. 이 메시지가 계속 따라다닐 것 같습니다.

이 스레드를 다시 올려봅니다. 혹시 도와주실 분이 있을까요…

추가 정보를 드리겠습니다. 이 문제는 Discourse(Docker 사용 중)에서 발생합니다:

root@forum-app:/var/www/discourse# rake uploads:missing
--------------------------------------------------------------------------------
WARNING! WARNING! WARNING!
--------------------------------------------------------------------------------

default has uploads on S3!
validating without inventory is likely to take an enormous amount of time.
We recommend you run SKIP_EXTERNAL=1 rake uploads:missing to skip validating if on a multisite.
rake aborted!
PG::SyntaxError: ERROR:  syntax error at end of input
LINE 1: INSERT INTO verified_ids VALUES
                                        ^
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.1.0/lib/patches/db/pg.rb:56:in `exec'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.1.0/lib/patches/db/pg.rb:56:in `exec'
/var/www/discourse/lib/file_store/s3_store.rb:423:in `list_missing'
/var/www/discourse/lib/file_store/s3_store.rb:303:in `list_missing_uploads'
/var/www/discourse/lib/tasks/uploads.rake:277:in `list_missing_uploads'
/var/www/discourse/lib/tasks/uploads.rake:270:in `block (2 levels) in <main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:80:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:90:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:255:in `block in <main>'
/var/www/discourse/lib/tasks/uploads.rake:281:in `block in <main>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => uploads:missing_files
(See full trace by running task with --trace)

그리고

root@forum-app:/var/www/discourse# rake uploads:missing_files
--------------------------------------------------------------------------------
WARNING! WARNING! WARNING!
--------------------------------------------------------------------------------

default has uploads on S3!
validating without inventory is likely to take an enormous amount of time.
We recommend you run SKIP_EXTERNAL=1 rake uploads:missing to skip validating if on a multisite.
rake aborted!
PG::SyntaxError: ERROR:  syntax error at end of input
LINE 1: INSERT INTO verified_ids VALUES
                                        ^
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.1.0/lib/patches/db/pg.rb:56:in `exec'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.1.0/lib/patches/db/pg.rb:56:in `exec'
/var/www/discourse/lib/file_store/s3_store.rb:423:in `list_missing'
/var/www/discourse/lib/file_store/s3_store.rb:303:in `list_missing_uploads'
/var/www/discourse/lib/tasks/uploads.rake:277:in `list_missing_uploads'
/var/www/discourse/lib/tasks/uploads.rake:270:in `block (2 levels) in <main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:80:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:90:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:255:in `block in <main>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => uploads:missing_files
(See full trace by running task with --trace)

그 다음:

root@forum-app:/var/www/discourse#  rake posts:missing_uploads
Looking for missing uploads on: default

0 post uploads are missing.

root@forum-app:/var/www/discourse#

그리고 - 주인공 - rake uploads:migrate_to_s3 --trace

root@forum-app:/var/www/discourse# rake uploads:migrate_to_s3 --trace
** Invoke uploads:migrate_to_s3 (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute uploads:migrate_to_s3
Please note that migrating to S3 is currently not reversible!
[CTRL+c] to cancel, [ENTER] to continue

Migrating uploads to S3 for 'default'...
Uploading files to S3...
 - Listing local files
 => 3 files
 - Listing S3 files
................... => 18784 files
 - Syncing files to S3
...
Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
10116 posts were flagged for a rebake
rake aborted!
FileStore::ToS3MigrationError: 1 of 17210 uploads are not migrated to S3. S3 migration failed for db 'default'.
/var/www/discourse/lib/file_store/to_s3_migration.rb:138:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:79:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:389:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:65:in `migrate'
/var/www/discourse/lib/tasks/uploads.rake:127:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:106:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:80:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:90:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:104:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:100:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
bin/rake:13:in `<top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/cli/exec.rb:58:in `load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/cli/exec.rb:58:in `kernel_load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/cli/exec.rb:23:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/cli.rb:491:in `exec'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/cli.rb:34:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/cli.rb:28:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/exe/bundle:45:in `block in <top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/exe/bundle:33:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => uploads:migrate_to_s3
root@forum-app:/var/www/discourse#

pawelorzech · 6월 8, 2023, 8:08오후

한 달에 한 번 정도 올려야 누군가 볼 수도 있을 것 같아요. 그래도 안 되네요!

JammyDodger · 6월 8, 2023, 8:16오후

이게 버그인지는 정확히 확신은 없는데, 더 많은 분들이 보시도록 #installation에 올려봤어요

pawelorzech · 7월 6, 2023, 10:44오전

또 하나:)!

JammyDodger · 7월 6, 2023, 11:24오전

이건 제 전문 분야는 아니에요 ^[1] 그런데 며칠 전에 이걸 봤는데 유용할 수도 있겠더라고요?

VERBOSE=1 rake posts:missing_uploads — 이 명령어로 문제 있는 게시글들을 나열하는 것 같아요.

그리고 GIVE_UP=1 VERBOSE=1 rake posts:missing_uploads도 있는데, 이건 누락된 업로드 플래그를 정리하는 것 같긴 해요 (다만 실제 누락된 업로드 자체는 복구되지 않아요).

도움이 될지 모르겠지만, 공유해 봅니다.

그리고 보니 케이크데이 동창이네요. 케이크데이 축하해요.

아직도 잘 모르겠어요 ↩︎

pawelorzech · 7월 10, 2023, 7:17오후

아쉽게도 이번에도 성공하지 못했습니다. 비표준 파일 확장명이 Discourse에 업로드되었기 때문일 것 같습니다. (불행히도 어떻게 했는지 기억나지 않지만) 한 번도 업로드되지 않은 파일들을 확인해 봤는데 전부 .mp4 파일이었습니다(이 파일들은 삭제했는데 여전히 문제가 발생하고 있습니다. 아마도 지난 2월에 그랬던 것 같은데 어떻게 그런 일이 있었는지 모르겠네요).

다시 S3에 업로드를 시도해 보았습니다:

Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
10471 posts were flagged for a rebake
rake aborted!
FileStore::ToS3MigrationError: 1 of 17777 uploads are not migrated to S3. S3 migration failed for db 'default'.
/var/www/discourse/lib/file_store/to_s3_migration.rb:138:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:79:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:389:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:65:in `migrate'
/var/www/discourse/lib/tasks/uploads.rake:127:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:106:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-5.0.0/lib/rails_multisite/connection_management.rb:82:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-5.0.0/lib/rails_multisite/connection_management.rb:92:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:104:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:100:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
bin/rake:13:in `<top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli/exec.rb:58:in `load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli/exec.rb:58:in `kernel_load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli/exec.rb:23:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli.rb:492:in `exec'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli.rb:34:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli.rb:28:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/exe/bundle:45:in `block in <top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/exe/bundle:33:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => uploads:migrate_to_s3

아쉽네요!

pawelorzech · 7월 10, 2023, 9:41오후

저도 방금 새 서버로 이동을 시도했는데, 이 - 분명히 - 깨진 백업에서 복원하는 과정에서 이 하나의 업로드 파일 때문에 실패했습니다. 이 문제를 일으키는 원인을 어떻게 찾아야 할지 전혀 감이 안 잡히네요…

새로운 로그:

root@forum-app:/var/www/discourse# rake uploads:migrate_to_s3 --trace
** Invoke uploads:migrate_to_s3 (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute uploads:migrate_to_s3
Please note that migrating to S3 is currently not reversible!
[CTRL+c] to cancel, [ENTER] to continue

Migrating uploads to S3 for 'default'...
Uploading files to S3...
 - Listing local files
find: ‘uploads/default/original’: No such file or directory
 => 0 files
 - Listing S3 files
.................... => 19385 files
 - Syncing files to S3

Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
10471 posts were flagged for a rebake
rake aborted!
FileStore::ToS3MigrationError: 1 of 17780 uploads are not migrated to S3. S3 migration failed for db 'default'.
/var/www/discourse/lib/file_store/to_s3_migration.rb:132:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:73:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:383:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:59:in `migrate'
/var/www/discourse/lib/tasks/uploads.rake:126:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:106:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-5.0.0/lib/rails_multisite/connection_management.rb:82:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-5.0.0/lib/rails_multisite/connection_management.rb:92:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:104:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:100:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
bin/rake:13:in `<top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli/exec.rb:58:in `load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli/exec.rb:58:in `kernel_load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli/exec.rb:23:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli.rb:492:in `exec'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli.rb:34:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli.rb:28:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/exe/bundle:45:in `block in <top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/exe/bundle:33:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => uploads:migrate_to_s3

글		댓글	조회수
Migrate_to_s3 fails Self-hosting s3	25	2420	2월 21, 2022
Rake uploads:migrate_from_s3 fails Self-hosting s3	31	3528	4월 24, 2021
Rake uploads:migrate_to_s3 error Self-hosting s3	8	995	1월 6, 2020
Migrate_to_S3 Fails on Rebake Self-hosting	17	2846	9월 10, 2022
Uploads migration to S3 fails Product Support	26	3591	5월 7, 2021

16551개 업로드 중 1개가 S3로 마이그레이션되지 않았습니다. 'default' DB의 S3 마이그레이션 실패

Related topics