Uploads migration to S3 fails

סיכום מבוסס בינה מלאכותית

User vulkanino encountered issues migrating Discourse uploads to S3, initially due to local volumes on DigitalOcean and later due to script limitations and environment variable checks. After partial migration and deadlocks during manual execution in the Rails console, vulkanino discovered migrated images were missing, later found in the tombstone directory. Despite S3 being configured via site settings, the rake uploads:migrate_to_s3 task repeatedly demanded environment variables like DISCOURSE_S3_BUCKET, even when provided. zogstrip clarified that GlobalSetting.use_s3? must return true, which wasn’t happening despite S3 working for new uploads. vulkanino eventually succeeded by manually editing GlobalSetting to force use_s3? to return true, commenting out CDN URL validation in lib/tasks/uploads.rake, and setting export DISCOURSE_S3_CDN_URL="" in the container shell. Others, like marcozambi, referenced similar migration bugs. Later users like eextra and Bathinda faced identical issues, with vulkanino confirming the workaround still applies, though the rake file may have changed since their fix two years prior. The core issue appears to be a mismatch between runtime S3 configuration detection and migration script requirements.

Command

rake uploads:migrate_to_s3 

Output

Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Please run these commands in the rails console

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

I think this is because I have setup volumes in digitalocean and moved all uploads to the volumes disk. What should I do to make the migration work?
Thanks.

Perhaps @tgxworld can advise?

I went one step forward by issuing:

rails c
SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

It is very slowly migrating. It copies one image every 3 seconds… but it’s doing it I think.

Update: every now and then it goes in deadlock:

PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 13101 waits for ShareLock on transaction 42733666; blocked by process 15775.
Process 15775 waits for ShareLock on transaction 42733664; blocked by process 13101.
HINT:  See server log for query details.
CONTEXT:  while updating tuple (7962,3) in relation "posts"

I reissue the commands and it continues where it left, at least if doesn’t start all over, this is cool!

Migration has finished, I rebaked all posts, but some images in old posts are missing.

For example in this post:
http://www.rasando.it/t/maggio-2015/391

The first image points to:
http://www.rasando.it/uploads/default/1514/112e9959a3baa8e7.jpg

That jpg is not there.I checked and the /shared/uploads/default/1514 directory exists but it’s empty.

I have searched that file and found it in the tombstone directory!

What am I missing?
How can I move all the tombstoned images in the “uploads” dir?

Ok, after moving all tombstone images to the uploads directory, some images reapperared. But many old images haven’t been migrated. I don’t know how the uploads:migrate_to_s3 script works.

Since I want to delete all the uploads directory, I want to migrate all files to S3. Why some files have not been migrated?

Did you get this sorted?

No Régis, now I have some images on S3 and other images (usually the oldest) on the digital ocean volume. I wanted to get rid completely of the DO volume, but there are too many images still there.

I think the problem is with the migrate_to_s3 script, it doesn’t pick up those images. Is there a chance to update the script and make it work?
Thank you.

What’s the output of rake uploads:migrate_to_s3?

After my last update from git it asks for env vars:

Migrating uploads to S3 for 'default'...
Please provide the following environment variables
  - DISCOURSE_S3_BUCKET
  - DISCOURSE_S3_REGION
  - DISCOURSE_S3_ACCESS_KEY_ID
  - DISCOURSE_S3_SECRET_ACCESS_KEY

It didn’t before.

Well, have you tried running the rake task by providing these environment variables?

DISCOURSE_S3_BUCKET=blablabla \
DISCOURSE_S3_REGION=blablabla \
DISCOURSE_S3_ACCESS_KEY_ID=blabalbal \
DISCOURSE_S3_SECRET_ACCESS_KEY=blablabla \
rake uploads:migrate_to_s3

It always asks for the env variables, whatever I do!

I have S3 in my site settings, so I don’t know why this doen’t work:

unless GlobalSetting.use_s3?
    puts <<~TEXT
      Please provide the following environment variables
        - DISCOURSE_S3_BUCKET
        - DISCOURSE_S3_REGION
        - DISCOURSE_S3_ACCESS_KEY_ID
        - DISCOURSE_S3_SECRET_ACCESS_KEY
    TEXT
    exit 2
  end

If GlobalSetting.use_s3? returns false, it means that S3 is not setup properly.

But the new uploads are stored on S3, I’ve set it up in the admin panel.

Shoud I put those variables in app.yml? I’m confused.

Enter them on the command line before the rake uploads:migrate_to_s3 like in @zogstrip’s post above.

The backslashes permit the entire command to extend over multiple lines. It can also be entered on a single very long line:

DISCOURSE_S3_BUCKET=blablabla DISCOURSE_S3_REGION=blablabla DISCOURSE_S3_ACCESS_KEY_ID=blabalbal DISCOURSE_S3_SECRET_ACCESS_KEY=blablabla rake uploads:migrate_to_s3

As written before, the error message appears just the same. Let me recap to be clear:

  • S3 is configured in the site settings and it works. Uploads go to my S3 bucket (and backups too)
  • Some previous uploads have been successfully migrated using the script. The script also rebaked the posts so it’s all good. I did a manual rebake just in case. It worked without asking for the env vars, but it worked partially.
  • Some other uploads did not migrate. I tried running the script multiple times but the uploads were not picked up by the script. Thinking it was a bug in the script I went to github and found a newer version of the script. Launching the newer version it asks the env vars, even if I provide them in the command line or if I set them in the shell. No way.

@Vulkanino, I think you are going through the same ordeal I had a couple of weeks ago.
I tried to summarise my experience (and some solutions) here.

https://meta.discourse.org/t/migration-of-system-stored-images-to-s3-after-configuration-change/19605/52

It’s the Marco’s bug I guess :smiley: Well what’s the proposed solution then?

After two months I made it!

Edited the GlobalSetting file and changed the function use_s3? to always return true. It didn’t work in any other way.

Then I edited uploads.rake and commented out this block:

if SiteSetting.Upload.s3_cdn_url.blank?
   puts "Please provide the 'DISCOURSE_S3_CDN_URL' environment variable"
    exit 3
end

The migration still was not happy until I did:
export DISCOURSE_S3_CDN_URL=""
(though I don’t use a CDN).

rake uploads:migrate_to_s3
Migrating uploads to S3 for ‘default’…
Uploading files to S3…

  • Listing local files… => 11473 files
  • Listing S3 files… => 13391 files
  • Syncing files to S3

Yay!

What’s the path to this file?

/var/www/discourse# vi lib/tasks/uploads.rake

found using

/var/www/discourse# grep -r 'upload' . | grep rake | grep lib | grep tasks