Fout-positieven op "posts worden niet opnieuw toegewezen aan nieuwe S3-upload-URL"

Fout
1

I was migrating our Discourse instance from a server to another, and came across an interesting issue…

We use S3 to store uploads from the forum. We have enabled them several years ago, hence it’s not something we introduced in this migration.
After fixing a couple of other issues, I was able to get the backup to be imported. But, it failed on a S3-related step with the following:

Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
72038 posts were flagged for a rebake
EXCEPTION: 257 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.

After digging a bit, I was able to trace down the issue to this line:

Then, I went to the rails console, and I was able to replicate the queries with the following:

discourse(prod)> SiteSetting.cdn_path("/uploads/#{@current_db}/original").sub(/https?:/, "")
=> "/uploads//original"
discourse(prod)> RailsMultisite::ConnectionManagement.current_db
=> "default"
discourse(prod)> cdn_path = SiteSetting.cdn_path("/uploads/default/original").sub(/https?:/, "")
=> "/uploads/default/original"
discourse(prod)> Post.where("cooked LIKE '%#{cdn_path}%'")
=> ...

Then, I went to those particular posts, and they were part of the Performance Reports (screenshot is from after I run a find-and-replace script):

The image shows HTTP GET requests targeting various endpoints on a Keroiod.io educational technology community website. (Captioned by AI)
The image shows HTTP GET requests targeting various endpoints on a Keroiod.io educational technology community website. (Captioned by AI)866×685 36.9 KB

Apparently, that check is retrieving any post containing /uploads/default/original in the cooked field, despite it might not be a legitimate asset. In this case, /uploads/default/original was used as “plain text”, hence it was not missed during the migration job.

Not sure if this is expected?
Thank you!

2

I’ve seen similar issues with cooked posts, I think.

Maybe you can do a database only restore and then it skips those checks.

That’s what I would try next.

3

I was able to actually restore it fully by just replacing the text in those posts to not match the filter. It was not an issue for me, but just bringing it up in case anyone faces the same issue, as maybe it’s worth to fix it in Discourse.

2 likes
4

Yeah. Maybe the code just shouldn’t check cooked posts.

1 like
5

Ik ben er vrij zeker van dat ik dit ook ben tegengekomen bij het proberen te herstellen van een back-up en ik heb de back-up gemaakt met "uploads opnemen" als UITGESCHAKELD. Mis ik iets met het herstellen van alleen de database?

Ik ben nieuw in discourse, dus ik zal proberen uit te zoeken hoe ik dit kan omzeilen, maar dit lijkt een hogere prioriteit te hebben als de back-up/restore niet correct werkt voor gebruikers die S3-buckets als opslag voor uploads hebben.

Hier is een deel van de logbestanden van de mislukte herstelpoging.

[2025-06-22 16:02:24] /var/www/discourse/lib/file_store/to_s3_migration.rb:132:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:81:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:385:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:59:in `migrate'
/var/www/discourse/lib/file_store/s3_store.rb:352:in `copy_from'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:69:in `restore_uploads'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:49:in `restore'
/var/www/discourse/lib/backup_restore/restorer.rb:167:in `restore_uploads'
/var/www/discourse/lib/backup_restore/restorer.rb:71:in `run'
/var/www/discourse/script/spawn_backup_restore.rb:20:in `restore'
/var/www/discourse/script/spawn_backup_restore.rb:33:in `block in <main>'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `fork'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `<main>'
[2025-06-22 16:02:24] Trying to rollback...
6

Update over mijn specifieke situatie… Ik had nog steeds bestanden in mijn /var/discourse/shared/standalone/uploads, ook al was ik overgeschakeld op S3 voor opslag. Zodra ik de ‘default’-directory in die uploads-directory had verwijderd, heb ik een nieuwe back-up gemaakt en deze is succesvol aangemaakt, alleen de database (…sql.gz).

Om een of andere reden (in mijn geval) als er bestanden in die directory staan, negeert het de instelling om uploads niet mee te nemen in de back-up en maakt ze toch aan.

Laat het me weten als er meer informatie of verduidelijking nodig is voor mijn situatie. Het lijkt erop dat het misschien iets anders is dan de situatie van de OP.

Hoe dan ook, ik heb het probleem kunnen omzeilen en kan nu succesvol herstellen.