Restore process cancelled at migrating uploads to S3 step

Suporte
22

If you need any help with drilling down to the root cause of issues like this, please let me know. I’m happy to help.

5 curtidas
23

So, tried to restore my 2.2GB backup and got this - the restore failed:

EXCEPTION: 44 posts are not remapped to new S3 upload URL. S3 migration failed for db ‘default’.

I am on the 2.6b6, so a fairly recent version. The site is fairly old, from 2016.

From time to time we run into backups were the automatic remapping and migration to S3 fails for various reasons.

This is an old topic, so any recommendation for corrective action? Having a backup I can’t actually use to restore makes me a little nervous.

Edit: I just realized I may be in the wrong thread, as this seems like the same issue

1 curtida
24

@gerhard

Gave this another shot and I am still failing on the restore to the 44 posts are not remapped -error.

Temporary disabling of S3 uploads prior to the backup (proposed by @RGJ does not solve the issue, and currently I don’t have a working backup scheme, which is pretty damn serious.

Any suggestions on what to try?

25

Okay, this is the weirdest case I’ve seen so far. I stumpled upon this post.

https://meta.discourse.org/t/using-object-storage-for-uploads-s3-clones/148916/160?u=ljpp

After wasting hours on this already, I defined the DISCOURSE_CDN_URL just for kicks, a feature my site does not use. The backup was restored successfully.

What is the significance of this parameter, in this scenario? @Falco @pfaffman

EDIT:

I take that back. After success with the test, I took a fresh backup from production and once again started to migrate to a new server. It failed again, but the error message changed.

EXCEPTION: rake posts:missing_uploads identified 3 issues. S3 migration failed for db ‘default’.

EDIT2:

So I kept drilling the database and found these 3 posts.

  • 2 were very old images, that were first published but then moderator replaced the images with a modified version. These were from 2016 and 2018.
  • But the last 1 was odd. It was a very recent post just a couple of weeks ago by myself and it contained a oneboxed https:// link to a development server, that no longer exists. So a broken link breaks the restore process.

I simply deleted these posts manually.

Now, again, a test run shows that a backup might actually succeed. We’ll see.

2 curtidas
26

@ljpp I seem to be in a similar situation with a restore failing with

EXCEPTION: 1 posts are not remapped to new S3 upload URL. S3 migration failed for db ‘default’.

Can you elaborate on how you found and deleted the offending posts? What commands are used?

1 curtida
27

He deleted the posts in question.

But you wiped your install so you can’t do that, right?

1 curtida
29

Eu tenho este problema restaurando meu banco de dados. Como você encontrou as postagens problemáticas?

30

Eu acho que o código que faz a verificação está aqui:

Então eu acho que isto:

prefix = @migrate_to_multisite ? "uploads/#{@current_db}/original/" : "original/"
base_url = File.join(SiteSetting.Upload.s3_base_url, prefix)
bad = Upload.by_users.where("url NOT LIKE '#{base_url}%'")

E para garantir

good = Upload.by_users.where("url LIKE '#{base_url}%'")

Me avise se isso funciona para você e eu verei como criar um tópico.

1 curtida
31

discourse(prod)> prefix = @migrate_to_multisite ? “uploads/#{RailsMultisite::ConnectionManagement.current_db}/original/” : “original/”
discourse(prod)> base_url = File.join(SiteSetting.Upload.s3_base_url, prefix)
discourse(prod)> bad_uploads = Upload.by_users.where(“url NOT LIKE ‘#{base_url}%’”)
discourse(prod)> bad_uploads.count
=> 0

Esta verificação me deu 0 erros

32

Você fez isso no servidor original? Ou você precisa fazer isso depois que o banco de dados for restaurado, antes que ele faça essa verificação, usando a chave --pause?

33

Eu fiz no servidor antigo.
Ao restaurar, obtive o mesmo erro

\[2026-01-16 13:45:52\] EXCEPTION: 3 publicações não foram remapeadas para o novo URL de upload do S3. A migração do S3 falhou para o banco de dados ‘default’.
\[2026-01-16 13:45:52\] /var/www/discourse/lib/file_store/to_s3_migration.rb:132:in `raise_or_log’

34

Bem. Droga. Não sei mais o que te dizer.

O good encontrou seus uploads?

Eu provavelmente usaria a opção --pause e pararia a restauração antes que ela fizesse essa verificação.

Eu faria o que o Richard disse.

1 curtida
35

Não tenho certeza sobre sua situação exata, mas você pode simplesmente querer desativar os uploads do S3, fazer o backup, restaurar e ativá-lo novamente. Isso me salvou muitas vezes.

2 curtidas
36

Então, eu tenho que desligar estas opções, correto?

image

37

Correto.

  • desmarque “habilitar uploads s3”
  • backup / download
  • upload / restore
  • marque “habilitar uploads s3”
3 curtidas
38

Ao marcar “habilitar uploads s3” recebi um erro

IMG_8462
IMG_84621290×859 132 KB

39

Eu não falo italiano quando não é sobre comida… talvez você possa nos fazer um favor e copiar/colar no Google Tradutor :slight_smile:

40

Desculpe

Habilitar uploads do S3

Armazena uploads no espaço em disco do Amazon S3. IMPORTANTE: requer que credenciais S3 válidas sejam fornecidas (tanto a chave de acesso ID quanto a chave de acesso secreta).

Você não pode habilitar os uploads do S3 porque eles já estão habilitados globalmente, e habilitar os uploads do S3 no nível do site pode causar problemas críticos com os uploads.

1 curtida
41

Presumo que você já o fez no app.yml então.
Os (novos) uploads estão funcionando corretamente? Se sim, então não há problema.

43

Em app.yml eu tenho s3 configurado, correto.

Então, o que eu tenho que fazer para fazer backup e restaurar com segurança sem verificar os uploads?

Simplesmente desmarcar “enable s3 uploads” não funciona