Rake uploads:migrate_from_s3 échoue

[pnoeric (Eric)]

J’ai suivi les étapes ici, sauvegardé l’intégralité de mon site, cloné mon bucket AWS S3, modifié le nom du bucket dans les paramètres de Discourse pour passer du bucket original au backup, et décoché la case « uploads to S3 » dans les paramètres.

Je suis donc enfin prêt à démarrer la migration depuis S3… et elle échoue.

Le message d’erreur

root@ubuntu:/var/www/discourse# rake uploads:migrate_from_s3
Migrating uploads from S3 to local storage for 'default'...
rake aborted!
NoMethodError: undefined method `downcase' for nil:NilClass
/var/www/discourse/app/models/global_setting.rb:107:in `s3_bucket_name'
/var/www/discourse/app/models/site_setting.rb:157:in `absolute_base_url'
/var/www/discourse/lib/tasks/uploads.rake:138:in `migrate_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:118:in `block in migrate_all_from_s3'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.2.2/lib/rails_multisite/connection_management.rb:68:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.2.2/lib/rails_multisite/connection_management.rb:78:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:118:in `migrate_all_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:93:in `block in <top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_from_s3
(See full trace by running task with --trace)

(Voici la ligne sur GitHub où l’erreur se produit — je suppose qu’il n’arrive pas à récupérer la valeur de s3_bucket ?)

Autres tentatives

• J’ai essayé d’ajouter les identifiants en ligne de commande, mais cela n’a rien changé. Par exemple :
  DISCOURSE_S3_BUCKET="dn-forum-storage-backup" DISCOURSE_S3_REGION="us-east-1" DISCOURSE_S3_ACCESS_KEY_ID="xxxxxxxxxxxxxxxxxxxx" DISCOURSE_S3_SECRET_ACCESS_KEY="xxxxxxxxxxxxxxxxxxxx" DISCOURSE_S3_CDN_URL="https://dn-forum-storage-backup.s3.us-east-1.amazonaws.com" rake uploads:migrate_from_s3

• J’ai aussi essayé de remettre le nom du bucket S3 dans mes paramètres à celui du bucket original, mais sans succès, même résultat.

• J’ai également essayé de reconstruire l’application. Même résultat.

@vinothkannans, sais-tu ce qui se passe ?

S’il vous plaît, aidez-moi, amis Discourse !

P.S. petite remarque secondaire : rake --tasks ne liste pas cette tâche ni aucune tâche commençant par uploads, je ne sais pas si cela signifie quelque chose.

[pnoeric (Eric)]

Possible related issue? cc @mcdanlj

S3 configuration expectations between global and site settings

I have a discourse instances where, months after initially creating the instance, we enabled “S3” (in our case, digital ocean spaces) for a while, only via admin settings, and never with any DISCOURSE_ config options in the app.yml file. We never migrated everything to S3, but the bulk of our images were written while we had it enabled. Now we want to migrate everything off “S3” for a variety of reasons not important here. (I know this sounds like a support post so far. But it’s not…) In a rec…

[mcdanlj (Michael K Johnson)]

@pnoeric yes that looks like exactly the same thing. I haven’t heard back in that issue on precisely what the intent is for SiteSettings vs. GlobalSettings for S3, so I don’t have any more help to give right now than to add it to SiteSettings via configuration (point 1 in my post).

[pnoeric (Eric)]

Hey, thanks for this reply… I’m not even sure what SiteSettings vs GlobalSettings means – I’m not that good a RoR coder and don’t understand the full setup well enough. I’m just following the basic instructions.

But hopefully @vinothkannans will jump in too; I think he wrote the code for the migrate tasks. Or anyone else from the @team that might know…

Let’s keep our eyes on this topic…

[mcdanlj (Michael K Johnson)]

s3_bucket is in GlobalSettings which is set from the config/discourse.conf file normally created from environment variables in the app.yml file. SiteSettings are things you change from the Admin Settings in the app.

It looks like when this was first created, you could only change S3 by rebuilding the app, and more recently it became possible to fill in the data in the Admin Settings. I can’t tell what was intended by not just doing a wholesale migration when setting S3 in SiteSettings was added.

[Edit: I inadvertently reversed the two when first posting this response]

[pnoeric (Eric)]

Hmmm @mcdanlj just to confirm, though, you haven’t been able to work out how to make the migrate_from_s3 actually work, correct?

I’m fine with editing whatever settings or low-level files or whatnot that is necessary… I just need to move off S3 ASAP since it’s costing me an arm and a leg.

[neounix (Dark Matter)]

@mcdanlj

It looks like when this was first created, you could only change S3 by rebuilding the app, and more recently it became possible to fill in the data in the Admin Settings. I can’t tell what was intended by not just doing a wholesale migration when setting S3 in SiteSettings was added.

Hi @pnoeric and @mcdanlj,

One debugging approach might be to enter into the rails console and then take a peek at the s2 site settings?

For example, in a plain ole OOTB discourse docker single container standalone app:

root@localhost:~# docker exec -it app rails c
[1] pry(main)> SiteSetting.s3_upload_bucket
=> ""
[2] pry(main)> SiteSetting.enable_s3_uploads
=> false
[3] pry(main)> 

comparing the SiteSettings (via the rails console) against the defaults, listed here:

config/site_settings.yml

Perhaps debugging in this manner might be helpful (no idea really, as we don’t use AWS or S3)? Maybe the rails console might help a bit?

[itsbhanusharma (Bhanu Sharma)]

@pnoeric

rake --tasks doesn’t list this task or any tasks that start with uploads , not sure if that means anything.

the rake --tasks only shows tasks that have a description. You can view all available tasks through rake -AT

@neounix

Perhaps debugging in this manner might be helpful (no idea really, as we don’t use AWS or S3)? Maybe the rails console might help a bit?

I don’t think it’ll help because I very recently had run these tasks on a test site. Both seem to depend on S3 variables being defined in the env however, it was a couple months ago and migrate_from_s3 didn’t really work for me.

[mcdanlj (Michael K Johnson)]

@pnoeric

Hmmm @mcdanlj just to confirm, though, you haven’t been able to work out how to make the migrate_from_s3 actually work, correct?

Tricky question. I did set s3_bucket in config/discourse.conf as mentioned in the post you linked to, which did resolve this particular error, as I noted there.

@mcdanlj

I tested this by putting s3_bucket into my config/discourse.conf in my existing container and it resolved the error for me.

This file is inside the container (./launcher enter app). Note that for this to survive ./launcher rebuild app you need also to add DISCOURSE_S3_BUCKET to the env section of your containers/app.yml file.

The fact that I fixed it was why it was a dev post, not a support request; I was asking what developers think is the right solution as I continue to hack at this.

I have about 100GB of files in S3 so I’m moving very carefully. I implemented a limit for posts to look at, and I next need to implement a limit for posts to modify. I’ve been trying one thing at a time. The fact that this appears to be rarely-used code and I’ve seen this error repeatedly makes me concerned about code rot and I don’t want to suddenly deface my entire site due to a bug, and this looks like it could be a good way to make that mistake.

• For upload:// (for me, this means non-video) uploads, so far, it appears to be working. I’m doing one at a time and then reviewing the affected post to make sure everything works.

• For uploads that don’t use the upload:// syntax (for me, this means video uploads as far as I can tell), where there is a literal reference to the URL in S3, it is mangling the URLs. That’s not a hard bug to fix as soon as I figure out for sure what I’m supposed to change them to but I haven’t done that yet. So that’s likely to be one of the PRs I post soon.

This is a spare time project for me, so no promises on timing.

[pnoeric (Eric)]

@mcdanlj

Tricky question. I did set s3_bucket in config/discourse.conf as mentioned in the post you linked to, which did resolve this particular error, as I noted there.

This file is inside the container ( ./launcher enter app ). Note that for this to survive ./launcher rebuild app you need also to add DISCOURSE_S3_BUCKET to the env section of your containers/app.yml file.

Aha, thank you! I’ll give it a crack.

[pnoeric (Eric)]

Ugh so no luck, @neounix @mcdanlj @vinothkannans. Still failing. But at least there’s a new/different error message…

Here’s what I tried today:

1. Upgrade to latest Discourse, just to be sure.

2. Add my s3_bucket in config/discourse.conf

3. ./launcher enter app

4. Edit containers/app.yml and added DISCOURSE_S3_BUCKET var

5. Tried rake uploads:migrate_from_s3 and now it fails with a new error message (before it was downcase causing the problem, now it appears to be start_with?):

/var/www/discourse# rake uploads:migrate_from_s3
Migrating uploads from S3 to local storage for 'default'...
rake aborted!
NoMethodError: undefined method `start_with?' for nil:NilClass
/var/www/discourse/app/models/site_setting.rb:161:in `absolute_base_url'
/var/www/discourse/lib/tasks/uploads.rake:138:in `migrate_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:118:in `block in migrate_all_from_s3'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:68:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:78:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:118:in `migrate_all_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:93:in `block in <main>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_from_s3
(See full trace by running task with --trace)

6. So then I tried ./launcher rebuild app

7. And again ./launcher enter app, rake uploads:migrate_from_s3

Same problem exactly:

/var/www/discourse# rake uploads:migrate_from_s3
Migrating uploads from S3 to local storage for 'default'...
rake aborted!
NoMethodError: undefined method `start_with?' for nil:NilClass
/var/www/discourse/app/models/site_setting.rb:161:in `absolute_base_url'
/var/www/discourse/lib/tasks/uploads.rake:138:in `migrate_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:118:in `block in migrate_all_from_s3'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:68:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:78:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:118:in `migrate_all_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:93:in `block in <main>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_from_s3
(See full trace by running task with --trace)

Any other ideas?

Doing this process is a real drag btw-- I have to pre-schedule and announce the forum outage days in advance, then on the day of, change the main site so folks can’t get into the forum, and then I have to shut down the forum server on Dig Ocean and take a snapshot before proceeding. That’s ~30 minutes right there. Then I start it up again and then I can try the steps above. I am SO regretting setting up Amazon S3 for media storage! I’ve burned hours trying to undo that choice and still no luck (and still a big $$$ bill from Amazon each month). I’d love to get to the bottom of this. How can I help?

[mcdanlj (Michael K Johnson)]

That line is:

        if SiteSetting.Upload.s3_region.start_with?("cn-")

Looks like it wants s3_region also; not clear to my why I didn’t run into that.

I’m not sure I follow your logic; my own migration of ~100GB of content I plan to do live, after a normal site backup. But I’m starting small, which is why I’ve been working on limiting the amount migrated at once. One warning: the code seems wrong for literal URL translations, as I see for video uploads, so if you allowed video uploads you might have a problem there with the code in its current state.

[pnoeric (Eric)]

@mcdanlj

Looks like it wants s3_region also; not clear to my why I didn’t run into that.

So maybe I should repeat all the steps I did above, but I’ll put s3_bucket, s3_region, s3_cnd_url, s3_secret_access_key etc. (basically, every variable I have) into the conf and yml files? I’d rather give it more than it (maybe) needs, just so the thing will actually work.

@mcdanlj

I’m not sure I follow your logic; my own migration of ~100GB of content I plan to do live, after a normal site backup.

I saw where someone on the Discourse team had suggested to back up the entire local site before starting this transition. Which requires me to take my Digital Ocean server offline.

@mcdanlj

But I’m starting small, which is why I’ve been working on limiting the amount migrated at once.

Right. I’m starting small too… every time I try I am migrating 0 files.

@mcdanlj

One warning: the code seems wrong for literal URL translations, as I see for video uploads, so if you allowed video uploads you might have a problem there with the code in its current state.

Luckily members are only allowed to upload JPG, GIF and PNG in my forum so it should be ok.

Fingers crossed.

[mcdanlj (Michael K Johnson)]

@pnoeric

I saw where someone on the Discourse team had suggested to back up the entire local site before starting this transition.

Backup and snapshot are not the same. A snapshot is the crudest form of backup. The Admin console has a backup facility. Make sure that you configure it to back up thumbnails in the configuration first.

@pnoeric

Right. I’m starting small too… every time I try I am migrating 0 files.

Now that you know that you don’t have to take your site down, you should be able to relax. You can use batch_migrate_from_s3 to migrate at most a certain number of uploads. Right now it limits the posts that are considered rather than the migrations done, a bug for me to resolve in a future PR. But I need to also resolve the video upload bug, and I’d like to consider printing feedback because one of the points of the limit is to be able to confirm in affected posts that the migration was successful.

I’m likely to do this all over the next 1-2 months, so if you want to wait on that it might be worth paying a few more months of S3, up to you and I’m not making promises just stating intent.

[mcdanlj (Michael K Johnson)]

@pnoeric, puisque vous êtes préoccupé par la disponibilité du site, je voulais vous transmettre ce que j’ai appris jusqu’à présent.

J’ai effectué ma migration en direct, comme je l’ai mentionné. Si je ne limite pas le débit de la migration, les files d’attente qui gèrent des tâches comme notifier les utilisateurs de l’activité des autres se saturent, ce qui dégrade l’expérience utilisateur du site.

J’ai migré environ 500 publications avec des vidéos et environ 30 000 publications avec des images, ce qui a pris environ deux semaines pour être terminé.

Si vous souhaitez essayer le code que j’ai utilisé, il se trouve actuellement à l’adresse suivante :

lib/tasks/uploads.rake

Vous pouvez le télécharger et le copier dans votre application pour remplacer le contenu actuel de lib/tasks/uploads.rake.

Avec ce code, vous pouvez faire quelque chose comme ceci :

bin/rake uploads:batch_migrate_from_s3[100,1000]

Cela ne prendra en compte que 1 000 publications au total contenant des fichiers joints, et migrera les fichiers d’un maximum de 100 publications avant de s’arrêter ; chaque fois qu’il modifie effectivement une publication après avoir migré ses fichiers joints, il attendra que la file d’attente soit vide avant de démarrer la suivante.

Si vous copiez le fichier, cela cassera les futures mises à jour du site jusqu’à ce que vous annuliez le changement. Le moyen le plus simple de l’annuler une fois satisfait est simplement ./launcher rebuild app (bien que, en tant que développeur, j’utilise git checkout HEAD lib/tasks/uploads.rake pour annuler mes modifications…).

J’ai remarqué qu’au moins avec Digital Ocean Spaces, je dois parfois réessayer plusieurs fois avant qu’une migration ne réussisse. Le script tel qu’il est actuellement ne vous donne aucun avertissement lorsque cela se produit, et vous devez simplement continuer à l’exécuter et attendre de voir. J’ai une PR en attente de révision qui affiche les erreurs dans ce cas, afin que vous sachiez au moins qu’un problème s’est produit.

…

J’ai ajouté une simple boucle de réessai courte, ainsi que le message d’erreur, et il semble que la boucle de réessai résolve le problème. De plus, la validation par rapport aux règles actuelles était effectuée sur le contenu brut des publications passées, ce qui pouvait casser la migration et laisser silencieusement des publications nécessitant une nouvelle cuisson ; j’ai également corrigé cela. Vous ne voudrez absolument pas effectuer une migration sans obtenir au moins la correction de validation, qui fait partie des commits de ma PR actuellement en cours de révision.

…

J’ai terminé ma migration, à ma connaissance. Ma PR contient tout le code que j’ai utilisé pour achever ma migration. Elle n’a pas encore été révisée. Je vous suggère de suivre l’évolution sur Migrate_from_s3 problems si vous le souhaitez.

[pnoeric (Eric)]

Merci ! Je vais essayer cela dans les prochains jours.

[mcdanlj (Michael K Johnson)]

Je viens d’ajouter une note dans ce message indiquant qu’il reste un bug que nous avons découvert aujourd’hui : les photos de profil ont disparu pour certains utilisateurs, et je ne sais pas pourquoi. Nous avons haussé les épaules et avons demandé aux utilisateurs concernés de restaurer leurs données, en nous excusant pour ce problème.

J’ai bien sûr effectué des sauvegardes fréquentes tout au long de ce processus !

Bonne chance !

[pnoeric (Eric)]

Sérieusement, est-ce que cela pourrait me rendre encore plus fou ?

Voici ce que j’ai fait :

1. J’ai copié votre nouveau code lib/tasks/upload.rake dans mon Discourse
2. J’ai ajouté TOUS mes variables Amazon s3_ à config/discourse.conf
3. Je les ai également ajoutées à app.yml (je ne suis pas sûr que cela ait un effet, mais pourquoi pas)
4. J’ai exécuté cette commande et j’ai obtenu…

root@:/var/www/discourse/config# rake uploads:batch_migrate_from_s3[100,1000]
Vous devez désactiver les uploads S3 avant d'exécuter cette tâche.

Et j’ai confirmé :

Donc, d’accord. J’ai édité le fichier uploads.rake et j’ai simplement supprimé cette vérification.

Maintenant, j’obtiens :

root@:/var/www/discourse/lib/tasks# rake uploads:batch_migrate_from_s3[100,1000]
Migration des uploads depuis S3 vers le stockage local pour 'default'...
Migration d'un maximum de 100 messages sur 1000...
... (beaucoup de sortie ici) ...
Modifié 91/100 : 28795 : 28486/1 - https://example.com/t/topic-title-here/28486/1
... (beaucoup de sortie ici) ...

Donc cela semblait fonctionner ! Hourra !

Après ce premier lot de 100, j’ai vérifié sidekiq et j’ai vu que mon message de test était en file d’attente, alors j’ai attendu qu’il soit terminé…

…puis je suis revenu vérifier… et ce message tire toujours son image depuis Amazon S3. J’ai essayé de “Reconstruire le HTML” sur le message, mais cela n’a rien changé.

Alors j’ai essayé tout le processus à nouveau, depuis la commande rake jusqu’au bout, et j’ai obtenu les mêmes résultats : les mêmes 100 messages ont été traités, les mêmes éléments mis en file d’attente dans sidekiq, et après avoir laissé tourner, l’image de ce message de test provient toujours de S3.

Hmm, je ne suis pas sûr de quoi essayer ensuite.

@mcdanlj, toute suggestion ou conseil que vous pourriez avoir serait apprécié

[mcdanlj (Michael K Johnson)]

C’est exactement ce à quoi je m’attendrais si vous supprimiez cette vérification. Je ne sais pas pourquoi vous avez décidé de la supprimer. C’est intentionnel. Désactivez les uploads vers S3 avant de lancer la migration.

[pnoeric (Eric)]

Ils étaient désactivés — complètement désactivés. (L’image de la case à cocher dans mon message correspond bien au bon paramètre, n’est-ce pas ?) Je les ai même activés puis désactivés à nouveau. Rien n’y fait.