결국 이런 일이 일어나리라 예상했다. 그동안 너무 일이 순조롭게 흘러갔다. 수년간 크루즈 컨트롤처럼 시스템을 운영해 왔는데, 시스템이 자동으로 업데이트되고 나는 몇 주에 한 번씩 Discourse를 업데이트하는 정도였다. 어젯밤 자정에 Amazon에서 시스템이 응답하지 않는다는 알림을 받았고, Discourse는 다운되어 있었으며 CPU 사용률이 100%까지 치솟아 AWS CPU 리소스가 소진될 때까지 계속됐다. 시스템에 로그인할 수 없었으며, 여러 번 재부팅 끝에 드디어 잠시 로그인에 성공했을 때 htop에서 CPU를 많이 차지하는 것을 확인했다
snap lxd activate
누군가 이런 현상을 본 적이 있다면, 왜 이런 일이 스스로 일어났는지 설명해 주시면 향후 참고할 수 있어 매우 감사하겠다.
가장 시급한 문제인 서버 복구 작업으로 넘어가겠다. AWS에서 Ubuntu 20LTS를 사용하여 새 서버를 구축했고, Discourse 설정은 매우 쉬웠다. app.yml 파일의 사본이 있어 이를 이용해 Discourse 포럼을 재구성했다. 기존 서버는 백업과 콘텐츠(이미지 등) 모두에 S3를 사용하고 있었다.
서버 생성 후, S3에서 최신 Discourse 백업 파일을 다운로드하여 수동으로 Discourse 서버에 업로드하고 복원(Restore) 버튼을 눌렀다. 몇 분 후 다음과 같은 오류가 발생했다.
[2022-06-09 09:01:56] ALTER TABLE
[2022-06-09 09:01:56] ALTER TABLE
[2022-06-09 09:01:56] Migrating the database...
[2022-06-09 09:02:11] == 20220308201942 CreateUploadReferences: migrating ===========================
-- create_table(:upload_references, {})
-> 0.0486s
-- add_index(:upload_references, [:upload_id, :target_type, :target_id], {:unique=>true, :name=>"index_upload_references_on_upload_and_target"})
-> 0.0030s
== 20220308201942 CreateUploadReferences: migrated (0.0580s) ==================
== 20220309132719 CopyPostUploadsToUploadReferences: migrating ================
-- execute("INSERT INTO upload_references(upload_id, target_type, target_id, created_at, updated_at)\nSELECT post_uploads.upload_id, 'Post', post_uploads.post_id, uploads.created_at, uploads.updated_at\nFROM post_uploads\nJOIN uploads ON uploads.id = post_uploads.upload_id\nON CONFLICT DO NOTHING\n")
-> 0.0595s
== 20220309132719 CopyPostUploadsToUploadReferences: migrated (0.0602s) =======
== 20220309132720 CopyPostUploadsToUploadReferencesForSync: migrating =========
-- execute("INSERT INTO upload_references(upload_id, target_type, target_id, created_at, updated_at)\nSELECT upload_id, 'Post', post_id, NOW(), NOW()\nFROM post_uploads\nON CONFLICT DO NOTHING\n")
-> 0.0076s
== 20220309132720 CopyPostUploadsToUploadReferencesForSync: migrated (0.0080s)
== 20220330160747 CopySiteSettingsUploadsToUploadReferences: migrating ========
-- execute("WITH site_settings_uploads AS (\n SELECT id, unnest(string_to_array(value, '|'))::integer upload_id\n FROM site_settings\n WHERE data_type = 17\n UNION\n SELECT id, value::integer\n FROM site_settings\n WHERE data_type = 18 AND value != ''\n)\nINSERT INTO upload_references(upload_id, target_type, target_id, created_at, updated_at)\nSELECT site_settings_uploads.upload_id, 'SiteSetting', site_settings_uploads.id, uploads.created_at, uploads.updated_at\nFROM site_settings_uploads\nJOIN uploads ON uploads.id = site_settings_uploads.upload_id\nON CONFLICT DO NOTHING\n")
-> 0.0034s
== 20220330160747 CopySiteSettingsUploadsToUploadReferences: migrated (0.0038s)
== 20220330160751 CopyBadgesUploadsToUploadReferences: migrating ==============
-- execute("INSERT INTO upload_references(upload_id, target_type, target_id, created_at, updated_at)\nSELECT badges.image_upload_id, 'Badge', badges.id, uploads.created_at, uploads.updated_at\nFROM badges\nJOIN uploads ON uploads.id = badges.image_upload_id\nWHERE badges.image_upload_id IS NOT NULL\nON CONFLICT DO NOTHING\n")
-> 0.0006s
== 20220330160751 CopyBadgesUploadsToUploadReferences: migrated (0.0010s) =====
== 20220330160754 CopyGroupsUploadsToUploadReferences: migrating ==============
-- execute("INSERT INTO upload_references(upload_id, target_type, target_id, created_at, updated_at)\nSELECT groups.flair_upload_id, 'Group', groups.id, uploads.created_at, uploads.updated_at\nFROM groups\nJOIN uploads ON uploads.id = groups.flair_upload_id\nWHERE groups.flair_upload_id IS NOT NULL\nON CONFLICT DO NOTHING\n")
-> 0.0050s
== 20220330160754 CopyGroupsUploadsToUploadReferences: migrated (0.0055s) =====
== 20220330160757 CopyUserExportsUploadsToUploadReferences: migrating =========
-- execute("INSERT INTO upload_references(upload_id, target_type, target_id, created_at, updated_at)\nSELECT user_exports.upload_id, 'UserExport', user_exports.id, uploads.created_at, uploads.updated_at\nFROM user_exports\nJOIN uploads ON uploads.id = user_exports.upload_id\nON CONFLICT DO NOTHING\n")
-> 0.0013s
== 20220330160757 CopyUserExportsUploadsToUploadReferences: migrated (0.0041s)
== 20220330164740 CopyThemeFieldsUploadsToUploadReferences: migrating =========
-- execute("INSERT INTO upload_references(upload_id, target_type, target_id, created_at, updated_at)\nSELECT theme_fields.upload_id, 'ThemeField', theme_fields.id, uploads.created_at, uploads.updated_at\nFROM theme_fields\nJOIN uploads ON uploads.id = theme_fields.upload_id\nWHERE type_id = 2\nON CONFLICT DO NOTHING\n")
-> 0.0006s
== 20220330164740 CopyThemeFieldsUploadsToUploadReferences: migrated (0.0010s)
== 20220404195635 CopyCategoriesUploadsToUploadReferences: migrating ==========
-- execute("INSERT INTO upload_references(upload_id, target_type, target_id, created_at, updated_at)\nSELECT categories.uploaded_logo_id, 'Category', categories.id, uploads.created_at, uploads.updated_at\nFROM categories\nJOIN uploads ON uploads.id = categories.uploaded_logo_id\nWHERE categories.uploaded_logo_id IS NOT NULL\nON CONFLICT DO NOTHING\n")
-> 0.0095s
-- execute("INSERT INTO upload_references(upload_id, target_type, target_id, created_at, updated_at)\nSELECT categories.uploaded_background_id, 'Category', categories.id, uploads.created_at, uploads.updated_at\nFROM categories\nJOIN uploads ON uploads.id = categories.uploaded_background_id\nWHERE categories.uploaded_background_id IS NOT NULL\nON CONFLICT DO NOTHING\n")
-> 0.0004s
== 20220404195635 CopyCategoriesUploadsToUploadReferences: migrated (0.0103s) =
== 20220404201949 CopyCustomEmojisUploadsToUploadReferences: migrating ========
-- execute("INSERT INTO upload_references(upload_id, target_type, target_id, created_at, updated_at)\nSELECT custom_emojis.upload_id, 'CustomEmoji', custom_emojis.id, uploads.created_at, uploads.updated_at\nFROM custom_emojis\nJOIN uploads ON uploads.id = custom_emojis.upload_id\nWHERE custom_emojis.upload_id IS NOT NULL\nON CONFLICT DO NOTHING\n")
-> 0.0032s
== 20220404201949 CopyCustomEmojisUploadsToUploadReferences: migrated (0.0036s)
== 20220404203356 CopyUserProfilesUploadsToUploadReferences: migrating ========
-- execute("INSERT INTO upload_references(upload_id, target_type, target_id, created_at, updated_at)\nSELECT user_profiles.profile_background_upload_id, 'UserProfile', user_profiles.user_id, uploads.created_at, uploads.updated_at\nFROM user_profiles\nJOIN uploads ON uploads.id = user_profiles.profile_background_upload_id\nWHERE user_profiles.profile_background_upload_id IS NOT NULL\nON CONFLICT DO NOTHING\n")
-> 0.0017s
-- execute("INSERT INTO upload_references(upload_id, target_type, target_id, created_at, updated_at)\nSELECT user_profiles.card_background_upload_id, 'UserProfile', user_profiles.user_id, uploads.created_at, uploads.updated_at\nFROM user_profiles\nJOIN uploads ON uploads.id = user_profiles.card_background_upload_id\nWHERE user_profiles.card_background_upload_id IS NOT NULL\nON CONFLICT DO NOTHING\n")
-> 0.0011s
== 20220404203356 CopyUserProfilesUploadsToUploadReferences: migrated (0.0033s)
== 20220404204439 CopyUserAvatarsUploadsToUploadReferences: migrating =========
-- execute("INSERT INTO upload_references(upload_id, target_type, target_id, created_at, updated_at)\nSELECT user_avatars.custom_upload_id, 'UserAvatar', user_avatars.id, uploads.created_at, uploads.updated_at\nFROM user_avatars\nJOIN uploads ON uploads.id = user_avatars.custom_upload_id\nWHERE user_avatars.custom_upload_id IS NOT NULL\nON CONFLICT DO NOTHING\n")
-> 0.0200s
-- execute("INSERT INTO upload_references(upload_id, target_type, target_id, created_at, updated_at)\nSELECT user_avatars.gravatar_upload_id, 'UserAvatar', user_avatars.id, uploads.created_at, uploads.updated_at\nFROM user_avatars\nJOIN uploads ON uploads.id = user_avatars.gravatar_upload_id\nWHERE user_avatars.gravatar_upload_id IS NOT NULL\nON CONFLICT DO NOTHING\n")
-> 0.0069s
== 20220404204439 CopyUserAvatarsUploadsToUploadReferences: migrated (0.0276s)
== 20220404212716 CopyThemeSettingsUploadsToUploadReferences: migrating =======
-- execute("INSERT INTO upload_references(upload_id, target_type, target_id, created_at, updated_at)\nSELECT theme_settings.value::int, 'ThemeSetting', theme_settings.id, uploads.created_at, uploads.updated_at\nFROM theme_settings\nJOIN uploads ON uploads.id = theme_settings.value::int\nWHERE data_type = 6 AND theme_settings.value IS NOT NULL AND theme_settings.value != ''\nON CONFLICT DO NOTHING\n")
-> 0.0025s
== 20220404212716 CopyThemeSettingsUploadsToUploadReferences: migrated (0.0030s)
== 20220526203356 CopyUserUploadsToUploadReferences: migrating ================
-- execute("INSERT INTO upload_references(upload_id, target_type, target_id, created_at, updated_at)\nSELECT users.uploaded_avatar_id, 'User', users.id, uploads.created_at, uploads.updated_at\nFROM users\nJOIN uploads ON uploads.id = users.uploaded_avatar_id\nWHERE users.uploaded_avatar_id IS NOT NULL\nON CONFLICT DO NOTHING\n")
-> 0.0227s
== 20220526203356 CopyUserUploadsToUploadReferences: migrated (0.0234s) =======
[2022-06-09 09:02:11] Reconnecting to the database...
[2022-06-09 09:02:12] Reloading site settings...
[2022-06-09 09:02:12] Disabling outgoing emails for non-staff users...
[2022-06-09 09:02:14] Disabling readonly mode...
[2022-06-09 09:02:14] Clearing category cache...
[2022-06-09 09:02:14] Reloading translations...
[2022-06-09 09:02:14] Remapping uploads...
[2022-06-09 09:02:14] Restoring uploads, this may take a while...
[2022-06-09 09:03:05] EXCEPTION: 509 of 1823 uploads are not migrated to S3. S3 migration failed for db 'default'.
[2022-06-09 09:03:05] /var/www/discourse/lib/file_store/to_s3_migration.rb:132:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:79:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:373:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:66:in `migrate'
/var/www/discourse/lib/file_store/s3_store.rb:328:in `copy_from'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:62:in `restore_uploads'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:44:in `restore'
/var/www/discourse/lib/backup_restore/restorer.rb:61:in `run'
/var/www/discourse/script/spawn_backup_restore.rb:23:in `restore'
/var/www/discourse/script/spawn_backup_restore.rb:36:in `block in <main>'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `fork'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `<main>'
문제가 무엇인지, 그리고 Amazon S3 서버 백업으로 서버를 복원하는 방법에 대해 조언해 주실 수 있는 분이 있으신가요?