Migrate_to_s3 for Digital Ocean Spaces woes

[pfaffman (Jay Pfaffman)]

Wait. So now I’ve restored the site to a staging server and ran:

rake uploads:migrate_to_s3                                                              
Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Please run these commands in the rails console

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

So does that mean that it’s going to just rebake all the posts and upload them in the background?

OOH! It looks like that’s what’s happening. Maybe I’m finally going to be able to make this transition!

EDIT: This is further confused because I previously had a CDN configured. I guess I need to rebake the posts without the CDN and then do the migrate_to_s3?

Also, I think that there’s a bug that if you have a CDN configured and then enable the s3 CDN, Discourse thinks that assets are on the S3 CDN. No. That’s not it. It looks like having DISCOURSE_S3_CDN_URL defined makes Discourse point to the CDN URL for assets, but the assets aren’t there. I thought that a rake assets:precompile might get it, but no joy. This might need to be another topic

[pfaffman (Jay Pfaffman)]

Well, no, it seems like nothing is happening. Nothing happens when running MigrateUploadScheme and there are no sidekiq jobs queued.

root@shadrack-rbx:/var/www/discourse# rake uploads:migrate_to_s3
Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Please run these commands in the rails console

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
root@shadrack-rbx:/var/www/discourse# rails c
[1] pry(main)> SiteSetting.migrate_to_new_scheme = true
=> true
[2] pry(main)> Jobs::MigrateUploadScheme.new.execute(nil)
=> []

[pfaffman (Jay Pfaffman)]

So, you can hack around this problem by deleting the stuff that’s not in the new scheme, like this:

 bad=Upload.where("url not like '//%' and url not like '/uploads/default/original/_X/%'")
bad.delete_all

This seems like a . – If you rebuild, you’ll need to do it again, so there’s something in db/fixtures that needs to be fixed, I guess. Oh, or maybe just change:

https://github.com/discourse/discourse/blob/master/lib/tasks/uploads.rake#221

to

  if Upload.where("id>0 and url NOT LIKE '//%' AND url NOT LIKE '/uploads/#{db}/original/_X/%'").exists?

I don’t know which is more correct or I might submit a PR.

But when I delete all of those uploads (before I figured out the above), I’m still getting this:

root@shadrack-rbx:/var/www/discourse# DISCOURSE_S3_CDN_URL="https://lc-rbx.ams3.cdn.digitaloceanspaces.com" rake uploads:migrate_to_s3
Migrating uploads to S3 for 'default'...
Please provide the 'DISCOURSE_S3_CDN_URL' environment variable

And this is in discourse.conf as well:

s3_cdn_url = 'https://lc-rbx.ams3.cdn.digitaloceanspaces.com'

I guess that’s because of

https://github.com/discourse/discourse/blob/master/lib/tasks/uploads.rake#L246

I changed it to GlobalSetting.s3_cdn_url.blank? and it’s running. @zogstrip. It looks like this is yours, but it’s 3 months old, so I guess there’s something else here I don’t understand.

I’ve got 165087, so I guess I’ll see what happens when that finished in a day or three.

[pfaffman (Jay Pfaffman)]

Thanks for your help on this @tgxworld. The problem that I’m having now (uploading 73GB) is that at some point I hit rate limits and the job craps out. And it seems that it starts over from the beginning each time, always reporting that there are something like 187123 images to upload.

It’s not your problem, but I’d really like to get this move to spaces complete before I move this site to a new server Real Soon Now. Would you recommend that I just give in an go for AWS S3 or do you think this should work?

[tgxworld (Alan Tan)]

If it is urgent, I’ll recommend going with AWS S3 first since the rake task is used internally by us to migrate uploads to the S3 store. In theory, the rake task should work with Spaces as well but it wouldn’t be our top priority at the moment.

[pfaffman (Jay Pfaffman)]

Indeed. It’s lots faster too. It’s an insane hassle to set up, but I think that spaces isn’t really ready for prime time.