Uploads migration to S3 fails

Support
1

Command

rake uploads:migrate_to_s3

Output

Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Please run these commands in the rails console

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

I think this is because I have setup volumes in digitalocean and moved all uploads to the volumes disk. What should I do to make the migration work?
Thanks.

1 Like
2

Perhaps @tgxworld can advise?

3

I went one step forward by issuing:

rails c
SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

It is very slowly migrating. It copies one image every 3 seconds
 but it’s doing it I think.

Update: every now and then it goes in deadlock:

PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 13101 waits for ShareLock on transaction 42733666; blocked by process 15775.
Process 15775 waits for ShareLock on transaction 42733664; blocked by process 13101.
HINT:  See server log for query details.
CONTEXT:  while updating tuple (7962,3) in relation "posts"

I reissue the commands and it continues where it left, at least if doesn’t start all over, this is cool!

1 Like
4

Migration has finished, I rebaked all posts, but some images in old posts are missing.

For example in this post:
http://www.rasando.it/t/maggio-2015/391

The first image points to:
http://www.rasando.it/uploads/default/1514/112e9959a3baa8e7.jpg

That jpg is not there.I checked and the /shared/uploads/default/1514 directory exists but it’s empty.

I have searched that file and found it in the tombstone directory!

What am I missing?
How can I move all the tombstoned images in the “uploads” dir?

5

Ok, after moving all tombstone images to the uploads directory, some images reapperared. But many old images haven’t been migrated. I don’t know how the uploads:migrate_to_s3 script works.

Since I want to delete all the uploads directory, I want to migrate all files to S3. Why some files have not been migrated?

6

Did you get this sorted?

7

No RĂ©gis, now I have some images on S3 and other images (usually the oldest) on the digital ocean volume. I wanted to get rid completely of the DO volume, but there are too many images still there.

I think the problem is with the migrate_to_s3 script, it doesn’t pick up those images. Is there a chance to update the script and make it work?
Thank you.

8

What’s the output of rake uploads:migrate_to_s3?

9

After my last update from git it asks for env vars:

Migrating uploads to S3 for 'default'...
Please provide the following environment variables
  - DISCOURSE_S3_BUCKET
  - DISCOURSE_S3_REGION
  - DISCOURSE_S3_ACCESS_KEY_ID
  - DISCOURSE_S3_SECRET_ACCESS_KEY

It didn’t before.

10

Well, have you tried running the rake task by providing these environment variables?

DISCOURSE_S3_BUCKET=blablabla \
DISCOURSE_S3_REGION=blablabla \
DISCOURSE_S3_ACCESS_KEY_ID=blabalbal \
DISCOURSE_S3_SECRET_ACCESS_KEY=blablabla \
rake uploads:migrate_to_s3
1 Like
11

It always asks for the env variables, whatever I do!

I have S3 in my site settings, so I don’t know why this doen’t work:

unless GlobalSetting.use_s3?
    puts <<~TEXT
      Please provide the following environment variables
        - DISCOURSE_S3_BUCKET
        - DISCOURSE_S3_REGION
        - DISCOURSE_S3_ACCESS_KEY_ID
        - DISCOURSE_S3_SECRET_ACCESS_KEY
    TEXT
    exit 2
  end
1 Like
12

If GlobalSetting.use_s3? returns false, it means that S3 is not setup properly.

1 Like
13

But the new uploads are stored on S3, I’ve set it up in the admin panel.

Shoud I put those variables in app.yml? I’m confused.

14

Enter them on the command line before the rake uploads:migrate_to_s3 like in @zogstrip’s post above.

The backslashes permit the entire command to extend over multiple lines. It can also be entered on a single very long line:

DISCOURSE_S3_BUCKET=blablabla DISCOURSE_S3_REGION=blablabla DISCOURSE_S3_ACCESS_KEY_ID=blabalbal DISCOURSE_S3_SECRET_ACCESS_KEY=blablabla rake uploads:migrate_to_s3
4 Likes
15

As written before, the error message appears just the same. Let me recap to be clear:

  • S3 is configured in the site settings and it works. Uploads go to my S3 bucket (and backups too)
  • Some previous uploads have been successfully migrated using the script. The script also rebaked the posts so it’s all good. I did a manual rebake just in case. It worked without asking for the env vars, but it worked partially.
  • Some other uploads did not migrate. I tried running the script multiple times but the uploads were not picked up by the script. Thinking it was a bug in the script I went to github and found a newer version of the script. Launching the newer version it asks the env vars, even if I provide them in the command line or if I set them in the shell. No way.
1 Like
16

@Vulkanino, I think you are going through the same ordeal I had a couple of weeks ago.
I tried to summarise my experience (and some solutions) here.

https://meta.discourse.org/t/migration-of-system-stored-images-to-s3-after-configuration-change/19605/52

5 Likes
17

It’s the Marco’s bug I guess :smiley: Well what’s the proposed solution then?

18

After two months I made it!

Edited the GlobalSetting file and changed the function use_s3? to always return true. It didn’t work in any other way.

Then I edited uploads.rake and commented out this block:

if SiteSetting.Upload.s3_cdn_url.blank?
   puts "Please provide the 'DISCOURSE_S3_CDN_URL' environment variable"
    exit 3
end

The migration still was not happy until I did:
export DISCOURSE_S3_CDN_URL=""
(though I don’t use a CDN).

rake uploads:migrate_to_s3
Migrating uploads to S3 for ‘default’

Uploading files to S3


  • Listing local files
 => 11473 files
  • Listing S3 files
 => 13391 files
  • Syncing files to S3

Yay!

5 Likes
19

What’s the path to this file?

20 
/var/www/discourse# vi lib/tasks/uploads.rake

found using

/var/www/discourse# grep -r 'upload' . | grep rake | grep lib | grep tasks
5 Likes