Rake uploads:migrate_to_s3 runs forever

Installation
1

When I run that command, I get:

Migrating uploads to S3 for 'default'...
Uploading files to S3...
 - Listing local files
 => 23 files
 - Listing S3 files
. => 94 files
 - Syncing files to S3
.......................
Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
4 posts were flagged for a rebake
4 posts still require rebaking and will be rebaked during regular job
Done!
Migrating uploads to S3 for 'default'...
Uploading files to S3...
 - Listing local files
 => 23 files
 - Listing S3 files
. => 94 files
 - Syncing files to S3
.......................
Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
4 posts were flagged for a rebake
4 posts still require rebaking and will be rebaked during regular job
Done!
Migrating uploads to S3 for 'default'...
Uploading files to S3...
 - Listing local files
 => 23 files
 - Listing S3 files
. => 94 files
 - Syncing files to S3
.......................
Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
4 posts were flagged for a rebake
4 posts still require rebaking and will be rebaked during regular job
Done!
Migrating uploads to S3 for 'default'...
Uploading files to S3...
 - Listing local files
 => 23 files
 - Listing S3 files
. => 94 files
 - Syncing files to S3
.......................

and it just keeps going.

Ultimately I hit ctrl-C and did rake posts:rebake which worked fine.

Then for kicks I tried the rake uploads:migrate_to_s3 command again, which did exactly the same thing again as above (same output, same endless loop).

Not sure what’s going on, or if this is a problem or not, but wanted to report. Any thoughts?

4 Likes
2

I can confirm this.

I built a new image with DISCOURSE_S3_XXX variables set to enable S3 uploads when the original database was migrated. I would think that this would put all images on S3 from the get-go, but, alas, restoring that backup fails when it tries to migrate to S3.

I then tried to rake uploads:migrate_to_s3 (with all ENV set in the command line before it) and get the infinite loop described in the OP.

I then tried doing a command line restore with all of the DISCOURSE_S3 settings in the command line.

I thought that maybe then I could do a backup and restore, but no. Restore from the web interface fails. Restore from the command line, without passing all of the S3 parameters also fails.

So, it appears that if you want to restore a backup that has uploads in S3, you need to do it from the command line and pass the S3 settings on the command line with the restore.

Not being able to restore a backup that has S3 uploads from the UX seems like a bug.

4 Likes
3

For the next person trying to do a restore :wink: here’s the exact syntax to put the command line options all on the same line with the restore command as @pfaffman recommends)… it’s just all four variables being defined and then the restore command at the very end.

DISCOURSE_S3_BUCKET=my-forum-storage DISCOURSE_S3_REGION=us-east-1 DISCOURSE_S3_ACCESS_KEY_ID=asdasdasdasdasdadsasdas DISCOURSE_S3_SECRET_ACCESS_KEY=aasdasdasdasdasdadsasdasasdasdasdasd discourse restore my-forums-2020-04-11-232922-v20200408121834.tar.gz
2 Likes
4

@pnoeric The endless loop has been fixed in https://github.com/discourse/discourse/commit/baae0e7446baba8a1c74460999904ade57879797

@pfaffman The problem should be fixed in https://github.com/discourse/discourse/commit/c6b411f6c18efc41010afcb83f2a712f642d3a23

6 Likes
5

Oh hooray! Thanks very much. (sometimes a :heart: isn’t enough.)

6 Likes
6

Yes, I’ll echo that too!! Thanks for being so responsive. Much appreciated. E

2 Likes