Rake uploads:migrate_from_s3 falha

pnoeric · Junho 8, 2020, 11:20am

Segui os passos aqui, fiz backup de todo o meu site, clonei meu bucket do AWS S3, alterei o nome do bucket nas configurações do Discourse do bucket original para o backup e desmarquei a caixa de seleção “uploads para S3” nas configurações.

Então, finalmente, estou pronto para iniciar a migração do S3… e ela falha.

A mensagem de erro

root@ubuntu:/var/www/discourse# rake uploads:migrate_from_s3
Migrando uploads do S3 para armazenamento local para 'default'...
rake aborted!
NoMethodError: undefined method `downcase' for nil:NilClass
/var/www/discourse/app/models/global_setting.rb:107:in `s3_bucket_name'
/var/www/discourse/app/models/site_setting.rb:157:in `absolute_base_url'
/var/www/discourse/lib/tasks/uploads.rake:138:in `migrate_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:118:in `block in migrate_all_from_s3'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.2.2/lib/rails_multisite/connection_management.rb:68:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.2.2/lib/rails_multisite/connection_management.rb:78:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:118:in `migrate_all_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:93:in `block in <top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_from_s3
(Veja o rastreamento completo executando a tarefa com --trace)

(Aqui está a linha no GitHub onde ocorre a falha — acho que não consegue obter o valor de s3_bucket?)

Outras coisas que tentei

Tentei adicionar as credenciais na linha de comando, mas isso não fez diferença. Ou seja:
DISCOURSE_S3_BUCKET="dn-forum-storage-backup" DISCOURSE_S3_REGION="us-east-1" DISCOURSE_S3_ACCESS_KEY_ID="xxxxxxxxxxxxxxxxxxxx" DISCOURSE_S3_SECRET_ACCESS_KEY="xxxxxxxxxxxxxxxxxxxx" DISCOURSE_S3_CDN_URL="https://dn-forum-storage-backup.s3.us-east-1.amazonaws.com" rake uploads:migrate_from_s3
Também tentei alterar o nome do bucket S3 nas minhas configurações de volta para o nome do bucket original, mas sem sorte, mesmo resultado.
Também tentei reconstruir o aplicativo. Mesmo resultado.

@vinothkannans, você sabe o que está acontecendo?

Por favor, ajudem, amigos do Discourse!

P.S. pequena nota lateral: rake --tasks não lista esta tarefa nem nenhuma tarefa que comece com uploads, não sei se isso significa algo.

pnoeric · Junho 8, 2020, 11:42am

Possible related issue? cc @mcdanlj

mcdanlj · Junho 8, 2020, 12:09pm

@pnoeric yes that looks like exactly the same thing. I haven’t heard back in that issue on precisely what the intent is for SiteSettings vs. GlobalSettings for S3, so I don’t have any more help to give right now than to add it to SiteSettings via configuration (point 1 in my post).

pnoeric · Junho 8, 2020, 2:14pm

Hey, thanks for this reply… I’m not even sure what SiteSettings vs GlobalSettings means – I’m not that good a RoR coder and don’t understand the full setup well enough. I’m just following the basic instructions.

But hopefully @vinothkannans will jump in too; I think he wrote the code for the migrate tasks. Or anyone else from the @team that might know…

Let’s keep our eyes on this topic…

mcdanlj · Junho 8, 2020, 2:26pm

s3_bucket is in GlobalSettings which is set from the config/discourse.conf file normally created from environment variables in the app.yml file. SiteSettings are things you change from the Admin Settings in the app.

It looks like when this was first created, you could only change S3 by rebuilding the app, and more recently it became possible to fill in the data in the Admin Settings. I can’t tell what was intended by not just doing a wholesale migration when setting S3 in SiteSettings was added.

[Edit: I inadvertently reversed the two when first posting this response]

pnoeric · Junho 9, 2020, 8:19am

Hmmm @mcdanlj just to confirm, though, you haven’t been able to work out how to make the migrate_from_s3 actually work, correct?

I’m fine with editing whatever settings or low-level files or whatnot that is necessary… I just need to move off S3 ASAP since it’s costing me an arm and a leg.

neounix · Junho 9, 2020, 9:17am

Hi @pnoeric and @mcdanlj,

One debugging approach might be to enter into the rails console and then take a peek at the s2 site settings?

For example, in a plain ole OOTB discourse docker single container standalone app:

root@localhost:~# docker exec -it app rails c
[1] pry(main)> SiteSetting.s3_upload_bucket
=> ""
[2] pry(main)> SiteSetting.enable_s3_uploads
=> false
[3] pry(main)>

comparing the SiteSettings (via the rails console) against the defaults, listed here:

github.com/discourse/discourse

config/site_settings.yml

main

# Available options:
#
# default               - The default value of the setting. For upload site settings, use the id of the upload seeded
#                         in db/fixtures/010_uploads.rb.
# client                - Set to true if the javascript should have access to this setting's value.
# refresh               - Set to true if clients should refresh when the setting is changed.
# min                   - For a string setting, the minimum length. For an integer setting, the minimum value.
# max                   - For a string setting, the maximum length. For an integer setting, the maximum value.
# regex                 - A regex that the value must match.
# validator             - The name of the class that will be use to validate the value of the setting.
# allow_any             - For choice settings allow items not specified in the choice list (default true)
# secret                - Set to true if input type should be password and value needs to be scrubbed from logs (default false).
# enum                  - The setting has a fixed set of allowed values, and only one can be chosen.
#                         Set to the class name that defines the set.
# locale_default        - A hash which overrides according to `SiteSetting.default_locale`.
#                         The key should be as the same as possible value of default_locale.
# mandatory_values      - A list of mandatory values that must be included in the setting, these cannot be changed or removed
#                         in the UI.
# requires_confirmation - A string that indicates if the setting requires confirmation before it can be changed.
#                         Only valid value here is "simple" which will display a confirmation dialog when the setting

This file has been truncated. show original

Perhaps debugging in this manner might be helpful (no idea really, as we don’t use AWS or S3)? Maybe the rails console might help a bit?

itsbhanusharma · Junho 9, 2020, 10:27am

the rake --tasks only shows tasks that have a description. You can view all available tasks through rake -AT

I don’t think it’ll help because I very recently had run these tasks on a test site. Both seem to depend on S3 variables being defined in the env however, it was a couple months ago and migrate_from_s3 didn’t really work for me.

mcdanlj · Junho 9, 2020, 10:51am

Tricky question. I did set s3_bucket in config/discourse.conf as mentioned in the post you linked to, which did resolve this particular error, as I noted there.

This file is inside the container (./launcher enter app). Note that for this to survive ./launcher rebuild app you need also to add DISCOURSE_S3_BUCKET to the env section of your containers/app.yml file.

The fact that I fixed it was why it was a dev post, not a support request; I was asking what developers think is the right solution as I continue to hack at this.

I have about 100GB of files in S3 so I’m moving very carefully. I implemented a limit for posts to look at, and I next need to implement a limit for posts to modify. I’ve been trying one thing at a time. The fact that this appears to be rarely-used code and I’ve seen this error repeatedly makes me concerned about code rot and I don’t want to suddenly deface my entire site due to a bug, and this looks like it could be a good way to make that mistake.

For upload:// (for me, this means non-video) uploads, so far, it appears to be working. I’m doing one at a time and then reviewing the affected post to make sure everything works.
For uploads that don’t use the upload:// syntax (for me, this means video uploads as far as I can tell), where there is a literal reference to the URL in S3, it is mangling the URLs. That’s not a hard bug to fix as soon as I figure out for sure what I’m supposed to change them to but I haven’t done that yet. So that’s likely to be one of the PRs I post soon.

This is a spare time project for me, so no promises on timing.

pnoeric · Junho 9, 2020, 4:18pm

Aha, thank you! I’ll give it a crack.

pnoeric · Junho 16, 2020, 10:16am

Ugh so no luck, @neounix @mcdanlj @vinothkannans. Still failing. But at least there’s a new/different error message…

Here’s what I tried today:

Upgrade to latest Discourse, just to be sure.
Add my s3_bucket in config/discourse.conf
./launcher enter app
Edit containers/app.yml and added DISCOURSE_S3_BUCKET var
Tried rake uploads:migrate_from_s3 and now it fails with a new error message (before it was downcase causing the problem, now it appears to be start_with?):

/var/www/discourse# rake uploads:migrate_from_s3
Migrating uploads from S3 to local storage for 'default'...
rake aborted!
NoMethodError: undefined method `start_with?' for nil:NilClass
/var/www/discourse/app/models/site_setting.rb:161:in `absolute_base_url'
/var/www/discourse/lib/tasks/uploads.rake:138:in `migrate_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:118:in `block in migrate_all_from_s3'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:68:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:78:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:118:in `migrate_all_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:93:in `block in <main>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_from_s3
(See full trace by running task with --trace)

So then I tried ./launcher rebuild app
And again ./launcher enter app, rake uploads:migrate_from_s3

Same problem exactly:

/var/www/discourse# rake uploads:migrate_from_s3
Migrating uploads from S3 to local storage for 'default'...
rake aborted!
NoMethodError: undefined method `start_with?' for nil:NilClass
/var/www/discourse/app/models/site_setting.rb:161:in `absolute_base_url'
/var/www/discourse/lib/tasks/uploads.rake:138:in `migrate_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:118:in `block in migrate_all_from_s3'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:68:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:78:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:118:in `migrate_all_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:93:in `block in <main>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_from_s3
(See full trace by running task with --trace)

Any other ideas?

Doing this process is a real drag btw-- I have to pre-schedule and announce the forum outage days in advance, then on the day of, change the main site so folks can’t get into the forum, and then I have to shut down the forum server on Dig Ocean and take a snapshot before proceeding. That’s ~30 minutes right there. Then I start it up again and then I can try the steps above. I am SO regretting setting up Amazon S3 for media storage! I’ve burned hours trying to undo that choice and still no luck (and still a big $$$ bill from Amazon each month). I’d love to get to the bottom of this. How can I help?

mcdanlj · Junho 16, 2020, 8:15pm

That line is:

        if SiteSetting.Upload.s3_region.start_with?("cn-")

Looks like it wants s3_region also; not clear to my why I didn’t run into that.

I’m not sure I follow your logic; my own migration of ~100GB of content I plan to do live, after a normal site backup. But I’m starting small, which is why I’ve been working on limiting the amount migrated at once. One warning: the code seems wrong for literal URL translations, as I see for video uploads, so if you allowed video uploads you might have a problem there with the code in its current state.

pnoeric · Junho 17, 2020, 11:23am

So maybe I should repeat all the steps I did above, but I’ll put s3_bucket, s3_region, s3_cnd_url, s3_secret_access_key etc. (basically, every variable I have) into the conf and yml files? I’d rather give it more than it (maybe) needs, just so the thing will actually work.

I saw where someone on the Discourse team had suggested to back up the entire local site before starting this transition. Which requires me to take my Digital Ocean server offline.

Right. I’m starting small too… every time I try I am migrating 0 files.

Luckily members are only allowed to upload JPG, GIF and PNG in my forum so it should be ok.

Fingers crossed.

mcdanlj · Junho 17, 2020, 2:53pm

Backup and snapshot are not the same. A snapshot is the crudest form of backup. The Admin console has a backup facility. Make sure that you configure it to back up thumbnails in the configuration first.

Now that you know that you don’t have to take your site down, you should be able to relax. You can use batch_migrate_from_s3 to migrate at most a certain number of uploads. Right now it limits the posts that are considered rather than the migrations done, a bug for me to resolve in a future PR. But I need to also resolve the video upload bug, and I’d like to consider printing feedback because one of the points of the limit is to be able to confirm in affected posts that the migration was successful.

I’m likely to do this all over the next 1-2 months, so if you want to wait on that it might be worth paying a few more months of S3, up to you and I’m not making promises just stating intent.

mcdanlj · Junho 20, 2020, 9:54pm

@pnoeric since you are concerned about site uptime, I thought I’d pass on to you what I’ve learned so far.

I did my migration live, as I mentioned. If I don’t rate-limit the migration, the queues that do things like notify users of each others’ activity get clogged up and the user experience of the site is diminished.

I migrated about 500 posts with videos and about 30K posts with images, which took about two weeks to copmlete.

If you want to try the code I used, it’s currently at

github.com/johnsonm/discourse

lib/tasks/uploads.rake

mkj-fix-more-urls

# frozen_string_literal: true

require "db_helper"
require "digest/sha1"
require "base62"

################################################################################
#                                    gather                                    #
################################################################################

require_dependency "rake_helpers"

task "uploads:gather" => :environment do
  ENV["RAILS_DB"] ? gather_uploads : gather_uploads_for_all_sites
end

def gather_uploads_for_all_sites
  RailsMultisite::ConnectionManagement.each_connection { gather_uploads }
end

This file has been truncated. show original

you can download it and copy it into your app to replace the current contents of lib/tasks/uploads.rake

With this code, you can do something like this:

bin/rake uploads:batch_migrate_from_s3[100,1000]

That will consider only 1000 total posts with uploads, and migrate files from a maximum of 100, before stopping; every time it actually modifies a post after migrating its uploads it will wait until the queue is empty before starting the next one.

If you copy the file in, it will break future site updates until you undo the change. The easiest way to undo it after you are satisfied is just ./launcher rebuild app (although as a developer I use git checkout HEAD lib/tasks/uploads.rake to undo my changes…)

I have noticed that at least with digital ocean spaces, sometimes I have to retry a few times before a migration succeeds. The script as it stands now doesn’t give you any warning when that happens, and you just have to keep running it and waiting to see. I do have a PR waiting for review that prints out errors in that case so that you at least know that something went wrong.

…

I’ve added a simple short retry loop, as well as the error message, and it appears that the retry loop resolves the problem. Also, validation against current rules was being done on past post raw content which could break the migration and silently leave posts that needed to be rebaked; I have also fixed that. You will definitely not want to do a migration without getting at least the validation fix, which is one of the commits in my PR currently up for review.

…

I have finished my migration, to the best of my knowledge. My PR has all the code that I used to complete my migration. It hasn’t been reviewed. I’d suggest following along at Migrate_from_s3 problems if you want.

pnoeric · Julho 1, 2020, 5:20pm

Thank you! I’m going to give this a whirl in the next few days.

mcdanlj · Julho 1, 2020, 9:24pm

I just added a note in that post about there being one bug left that we discovered today that Profile Pictures have gone missing for some users, and I don’t know why. We shrugged and have been asking affected users to restore, with apologies for the problem.

I definitely did frequent backups during this process!

Best of luck!

pnoeric · Julho 3, 2020, 12:02pm

Seriously, could this make me more crazy?

Here’s what I did:

Copied over your new lib/tasks/upload.rake code into my Discourse
Added ALL of my Amazon s3_ variables to config/discourse.conf
Also added them to app.yml (unclear if that does anything, but why not)
Ran this command and got…

root@:/var/www/discourse/config# rake uploads:batch_migrate_from_s3[100,1000]
You must disable S3 uploads before running that task.

And confirmed:

So, ok. I edited the uploads.rake file and just removed that check.

Now I get:

root@:/var/www/discourse/lib/tasks# rake uploads:batch_migrate_from_s3[100,1000]
Migrating uploads from S3 to local storage for 'default'...
Migrating up to 100 of 1000 posts...
... (lots of output here) ...
Modified 91/100: 28795: 28486/1 - https://example.com/t/topic-title-here/28486/1
... (lots of output here) ...

So it appeared to be working! Yay!

After it did that first batch of 100, I checked sidekiq and I saw my test post was queued so I waited for that to finish…

…then went back and checked… and that post is still pulling its image from Amazon S3. I tried “Rebuild HTML” on the post and that didn’t change it.

So then I tried the whole process again, from the rake all the way through, and got the same results-- the same 100 posts were processed, the same things queued in sidekiq, and after letting it run, the image in that test post still coming from S3.

Hmmm, I’m not sure what to try next.

@mcdanlj appreciate any suggestions or advice you might have

mcdanlj · Julho 3, 2020, 1:12pm

That’s exactly what I would expect if you remove that check. I’m not sure why you decided to remove it. It’s on purpose. Turn off uploads to S3 before starting the migration.

pnoeric · Julho 3, 2020, 1:24pm

They were off-- completely off. (The picture of the checkbox in my post is the right setting, correct?) I even turned them on and back off. No go.

Tópico		Respostas	Visualizações
Migrate_from_s3: You must disable S3 uploads before running that task Support	21	1342	19 de Setembro de 2020
Uploads migration to S3 fails Support	26	3141	7 de Maio de 2021
Migrate_to_s3 for Digital Ocean Spaces woes Self-hosting s3	26	3244	27 de Abril de 2019
1 of 16551 uploads are not migrated to S3. S3 migration failed for db 'default' Self-hosting s3 , cdn	22	1752	10 de Agosto de 2023
Rake uploads:migrate_to_s3 error Self-hosting s3	8	827	6 de Janeiro de 2020

Rake uploads:migrate_from_s3 falha

A mensagem de erro

Outras coisas que tentei

Tópicos relacionados