Restore process cancelled at migrating uploads to S3 step

[RGJ (Richard - Communiteq)]

If you need any help with drilling down to the root cause of issues like this, please let me know. I’m happy to help.

[ljpp]

So, tried to restore my 2.2GB backup and got this - the restore failed:

EXCEPTION: 44 posts are not remapped to new S3 upload URL. S3 migration failed for db ‘default’.

I am on the 2.6b6, so a fairly recent version. The site is fairly old, from 2016.

From time to time we run into backups were the automatic remapping and migration to S3 fails for various reasons.

This is an old topic, so any recommendation for corrective action? Having a backup I can’t actually use to restore makes me a little nervous.

Edit: I just realized I may be in the wrong thread, as this seems like the same issue

Restore failed at "EXCEPTION: x of y uploads are not migrated to S3. S3 migration failed for db 'default'."

This has been sitting as a draft on meta.discourse.org as I worked through the issue on my Discourse server. I’ve been able to get around the problem described below, but that I couldn’t do what was effectively a disaster restore (a restore without the benefit of being able to get into the old, broken server) is concerning. (The root cause was a failed Postgres 10-to-12 upgrade, but I was able to postpone the update in the Discourse app.yml file long enough to disable the enable_s3_uploads sit…

[ljpp]

@gerhard

Gave this another shot and I am still failing on the restore to the 44 posts are not remapped -error.

Temporary disabling of S3 uploads prior to the backup (proposed by @RGJ does not solve the issue, and currently I don’t have a working backup scheme, which is pretty damn serious.

Restore Failed: "Some uploads could not be migrated to the new scheme"

Have you tried turning off s3 uploads prior to creating the backup? Just turning off S3 and not rebaking should not do any harm. Then restore and switch S3 back on.

Any suggestions on what to try?

[ljpp]

@ljpp

Any suggestions on what to try?

Okay, this is the weirdest case I’ve seen so far. I stumpled upon this post.

https://meta.discourse.org/t/using-object-storage-for-uploads-s3-clones/148916/160?u=ljpp

After wasting hours on this already, I defined the DISCOURSE_CDN_URL just for kicks, a feature my site does not use. The backup was restored successfully.

What is the significance of this parameter, in this scenario? @Falco @pfaffman

EDIT:

I take that back. After success with the test, I took a fresh backup from production and once again started to migrate to a new server. It failed again, but the error message changed.

EXCEPTION: rake posts:missing_uploads identified 3 issues. S3 migration failed for db ‘default’.

EDIT2:

So I kept drilling the database and found these 3 posts.

• 2 were very old images, that were first published but then moderator replaced the images with a modified version. These were from 2016 and 2018.
• But the last 1 was odd. It was a very recent post just a couple of weeks ago by myself and it contained a oneboxed https:// link to a development server, that no longer exists. So a broken link breaks the restore process.

I simply deleted these posts manually.

Now, again, a test run shows that a backup might actually succeed. We’ll see.

[BryanV (Bryan Van de ven)]

@ljpp I seem to be in a similar situation with a restore failing with

EXCEPTION: 1 posts are not remapped to new S3 upload URL. S3 migration failed for db ‘default’.

Can you elaborate on how you found and deleted the offending posts? What commands are used?

[pfaffman (Jay Pfaffman)]

He deleted the posts in question.

But you wiped your install so you can’t do that, right?

[Saiano]

@BryanV

@ljpp Mi sembra di essere in una situazione simile con un ripristino che fallisce con

EXCEPTION: 1 posts are not remapped to new S3 upload URL. S3 migration failed for db ‘default’.

Puoi spiegare come hai trovato ed eliminato i post problematici? Quali comandi sono stati usati?

Ho questo problema nel ripristinare il mio db. Come hai trovato i post problematici?

[pfaffman (Jay Pfaffman)]

@ljpp

non rimappato al nuovo S3

Penso che il codice che esegue il controllo sia qui:

lib/file_store/to_s3_migration.rb

Quindi penso questo:

prefix = @migrate_to_multisite ? "uploads/#{@current_db}/original/" : "original/"
base_url = File.join(SiteSetting.Upload.s3_base_url, prefix)
bad = Upload.by_users.where("url NOT LIKE '#{base_url}%'")

E per sicurezza

good = Upload.by_users.where("url LIKE '#{base_url}%'")

Fammi sapere se funziona per te e vedrò di creare un topic.

[Saiano]

discourse(prod)> prefix = @migrate_to_multisite ? “uploads/#{RailsMultisite::ConnectionManagement.current_db}/original/” : “original/”
discourse(prod)> base_url = File.join(SiteSetting.Upload.s3_base_url, prefix)
discourse(prod)> bad_uploads = Upload.by_users.where(“url NOT LIKE ‘#{base_url}%’”)
discourse(prod)> bad_uploads.count
=> 0

Questo controllo mi ha dato 0 errori

[pfaffman (Jay Pfaffman)]

L’hai fatto sul server originale? O devi farlo dopo che il database è stato ripristinato prima che esegua quel controllo, usando l’opzione --pause?

[Saiano]

L’ho fatto sul vecchio server.
Ripristinando ho ottenuto lo stesso errore

[2026-01-16 13:45:52] EXCEPTION: 3 post non sono stati rimappati al nuovo URL di caricamento S3. La migrazione S3 è fallita per il db ‘default’.
[2026-01-16 13:45:52] /var/www/discourse/lib/file_store/to_s3_migration.rb:132:in `raise_or_log’

[pfaffman (Jay Pfaffman)]

@Saiano

Questo controllo non mi ha dato 0 errori

Beh. Dannazione. Non so cos’altro dirti.

Il comando good ha trovato i tuoi upload?

Probabilmente userei l’opzione --pause e interromperei il ripristino prima che esegua quel controllo.

Farei quello che ha detto Richard.

[RGJ (Richard - Communiteq)]

Non sono sicuro della tua situazione esatta, ma potresti semplicemente voler disattivare i caricamenti s3, eseguire il backup, ripristinare e riattivare. Questo mi ha salvato molte volte.

[Saiano]

Quindi, devo disattivare queste opzioni, giusto?

[RGJ (Richard - Communiteq)]

Corretto.

• deseleziona “abilita caricamenti s3”
• backup / download
• upload / ripristino
• seleziona “abilita caricamenti s3”

[Saiano]

@RGJ

Corretto.

• deseleziona “abilita caricamenti s3”
• backup / download
• upload / ripristina
• seleziona “abilita caricamenti s3”

Mentre selezionavo “abilita caricamenti s3” ho ricevuto un errore

[RGJ (Richard - Communiteq)]

Non parlo italiano quando non si tratta di cibo… forse puoi farci tutti un favore e copiarlo e incollarlo in Google Traduttore

[Saiano]

@RGJ

Non parlo italiano se non si parla di cibo… magari puoi farci un favore e copiarlo/incollarlo in Google Translate

Scusa

Abilita caricamenti S3

Memorizza i caricamenti nello spazio su disco di Amazon S3. IMPORTANTE: richiede la fornitura di credenziali S3 valide (sia l’ID chiave di accesso che la chiave segreta di accesso).

Non è possibile abilitare i caricamenti S3 perché sono già abilitati a livello globale e l’abilitazione dei caricamenti S3 a livello di sito potrebbe causare problemi critici con i caricamenti.

[RGJ (Richard - Communiteq)]

Presumo che tu l’abbia già fatto in app.yml.
I (nuovi) caricamenti funzionano correttamente? In tal caso, non c’è alcun problema.

[Saiano]

@RGJ

Presumo tu l’abbia già fatto in app.yml allora.
I (nuovi) caricamenti funzionano correttamente? In tal caso, non c’è nessun problema.

In app.yml ho configurato s3, corretto.

Quindi, cosa devo fare per eseguire il backup e il ripristino in sicurezza senza controllare i caricamenti?

Semplicemente deselezionare “abilita caricamenti s3” non funziona