User vulkanino encountered issues migrating Discourse uploads to S3, initially due to local volumes on DigitalOcean and later due to script limitations and environment variable checks. After partial migration and deadlocks during manual execution in the Rails console, vulkanino discovered migrated images were missing, later found in the tombstone directory. Despite S3 being configured via site settings, the rake uploads:migrate_to_s3 task repeatedly demanded environment variables like DISCOURSE_S3_BUCKET, even when provided. zogstrip clarified that GlobalSetting.use_s3? must return true, which wasn’t happening despite S3 working for new uploads. vulkanino eventually succeeded by manually editing GlobalSetting to force use_s3? to return true, commenting out CDN URL validation in lib/tasks/uploads.rake, and setting export DISCOURSE_S3_CDN_URL="" in the container shell. Others, like marcozambi, referenced similar migration bugs. Later users like eextra and Bathinda faced identical issues, with vulkanino confirming the workaround still applies, though the rake file may have changed since their fix two years prior. The core issue appears to be a mismatch between runtime S3 configuration detection and migration script requirements.
Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Please run these commands in the rails console
SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
I think this is because I have setup volumes in digitalocean and moved all uploads to the volumes disk. What should I do to make the migration work?
Thanks.
rails c
SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
It is very slowly migrating. It copies one image every 3 seconds… but it’s doing it I think.
Update: every now and then it goes in deadlock:
PG::TRDeadlockDetected: ERROR: deadlock detected
DETAIL: Process 13101 waits for ShareLock on transaction 42733666; blocked by process 15775.
Process 15775 waits for ShareLock on transaction 42733664; blocked by process 13101.
HINT: See server log for query details.
CONTEXT: while updating tuple (7962,3) in relation "posts"
I reissue the commands and it continues where it left, at least if doesn’t start all over, this is cool!
Ok, after moving all tombstone images to the uploads directory, some images reapperared. But many old images haven’t been migrated. I don’t know how the uploads:migrate_to_s3 script works.
Since I want to delete all the uploads directory, I want to migrate all files to S3. Why some files have not been migrated?
No Régis, now I have some images on S3 and other images (usually the oldest) on the digital ocean volume. I wanted to get rid completely of the DO volume, but there are too many images still there.
I think the problem is with the migrate_to_s3 script, it doesn’t pick up those images. Is there a chance to update the script and make it work?
Thank you.
After my last update from git it asks for env vars:
Migrating uploads to S3 for 'default'...
Please provide the following environment variables
- DISCOURSE_S3_BUCKET
- DISCOURSE_S3_REGION
- DISCOURSE_S3_ACCESS_KEY_ID
- DISCOURSE_S3_SECRET_ACCESS_KEY
It always asks for the env variables, whatever I do!
I have S3 in my site settings, so I don’t know why this doen’t work:
unless GlobalSetting.use_s3?
puts <<~TEXT
Please provide the following environment variables
- DISCOURSE_S3_BUCKET
- DISCOURSE_S3_REGION
- DISCOURSE_S3_ACCESS_KEY_ID
- DISCOURSE_S3_SECRET_ACCESS_KEY
TEXT
exit 2
end
As written before, the error message appears just the same. Let me recap to be clear:
S3 is configured in the site settings and it works. Uploads go to my S3 bucket (and backups too)
Some previous uploads have been successfully migrated using the script. The script also rebaked the posts so it’s all good. I did a manual rebake just in case. It worked without asking for the env vars, but it worked partially.
Some other uploads did not migrate. I tried running the script multiple times but the uploads were not picked up by the script. Thinking it was a bug in the script I went to github and found a newer version of the script. Launching the newer version it asks the env vars, even if I provide them in the command line or if I set them in the shell. No way.