# Uploads migration to S3 fails
**URL:** https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487
**Category:** Support
**Created:** [January 14, 2019, 12:31pm UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487 "2019-01-14T12:31:38Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@vulkanino](https://meta.discourse.org/u/vulkanino)
#### Post date: [January 14, 2019, 12:31pm UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/1 "2019-01-14T12:31:38Z")
Command
```
rake uploads:migrate_to_s3
```
Output
```
Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Please run these commands in the rails console
SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
```
I think this is because I have setup volumes in digitalocean and moved all uploads to the volumes disk. What should I do to make the migration work?
Thanks.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [January 14, 2019, 9:59pm UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/2 "2019-01-14T21:59:17Z")
Perhaps @tgxworld can advise?
---
### Author:  [@vulkanino](https://meta.discourse.org/u/vulkanino)
#### Post date: [January 14, 2019, 10:15pm UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/3 "2019-01-14T22:15:01Z")
I went one step forward by issuing:
```
rails c
SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
```
It is very slowly migrating. It copies one image every 3 seconds… but it’s doing it I think.
Update: every now and then it goes in deadlock:
```
PG::TRDeadlockDetected: ERROR: deadlock detected
DETAIL: Process 13101 waits for ShareLock on transaction 42733666; blocked by process 15775.
Process 15775 waits for ShareLock on transaction 42733664; blocked by process 13101.
HINT: See server log for query details.
CONTEXT: while updating tuple (7962,3) in relation "posts"
```
I reissue the commands and it continues where it left, at least if doesn’t start all over, this is cool!
---
### Author:  [@vulkanino](https://meta.discourse.org/u/vulkanino)
#### Post date: [January 15, 2019, 9:50am UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/4 "2019-01-15T09:50:04Z")
Migration has finished, I rebaked all posts, but some images in old posts are missing.
For example in this post:
[http://www.rasando.it/t/maggio-2015/391](http://www.rasando.it/t/maggio-2015/391)
The first image points to:
[http://www.rasando.it/uploads/default/1514/112e9959a3baa8e7.jpg](http://www.rasando.it/uploads/default/1514/112e9959a3baa8e7.jpg)
That jpg is not there.I checked and the /shared/uploads/default/1514 directory exists but it’s empty.
I have searched that file and found it in the tombstone directory!
What am I missing?
How can I move all the tombstoned images in the “uploads” dir?
---
### Author:  [@vulkanino](https://meta.discourse.org/u/vulkanino)
#### Post date: [January 15, 2019, 12:37pm UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/5 "2019-01-15T12:37:31Z")
Ok, after moving all tombstone images to the uploads directory, some images reapperared. But many old images haven’t been migrated. I don’t know how the `uploads:migrate_to_s3` script works.
Since I want to delete all the uploads directory, I want to migrate all files to S3. Why some files have not been migrated?
---
### Author:  [@zogstrip](https://meta.discourse.org/u/zogstrip)
#### Post date: [January 25, 2019, 2:49pm UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/6 "2019-01-25T14:49:13Z")
Did you get this sorted?
---
### Author:  [@vulkanino](https://meta.discourse.org/u/vulkanino)
#### Post date: [January 25, 2019, 4:33pm UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/7 "2019-01-25T16:33:01Z")
No Régis, now I have some images on S3 and other images (usually the oldest) on the digital ocean volume. I wanted to get rid completely of the DO volume, but there are too many images still there.
I think the problem is with the `migrate_to_s3` script, it doesn’t pick up those images. Is there a chance to update the script and make it work?
Thank you.
---
### Author:  [@zogstrip](https://meta.discourse.org/u/zogstrip)
#### Post date: [January 25, 2019, 4:33pm UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/8 "2019-01-25T16:33:56Z")
What’s the output of `rake uploads:migrate_to_s3`?
---
### Author:  [@vulkanino](https://meta.discourse.org/u/vulkanino)
#### Post date: [January 25, 2019, 4:42pm UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/9 "2019-01-25T16:42:38Z")
After my last update from git it asks for env vars:
```
Migrating uploads to S3 for 'default'...
Please provide the following environment variables
- DISCOURSE_S3_BUCKET
- DISCOURSE_S3_REGION
- DISCOURSE_S3_ACCESS_KEY_ID
- DISCOURSE_S3_SECRET_ACCESS_KEY
```
It didn’t before.
---
### Author:  [@zogstrip](https://meta.discourse.org/u/zogstrip)
#### Post date: [January 25, 2019, 4:44pm UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/10 "2019-01-25T16:44:21Z")
Well, have you tried running the rake task by providing these environment variables?
```plaintext
DISCOURSE_S3_BUCKET=blablabla \
DISCOURSE_S3_REGION=blablabla \
DISCOURSE_S3_ACCESS_KEY_ID=blabalbal \
DISCOURSE_S3_SECRET_ACCESS_KEY=blablabla \
rake uploads:migrate_to_s3
```
---
### Author:  [@vulkanino](https://meta.discourse.org/u/vulkanino)
#### Post date: [January 25, 2019, 4:48pm UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/11 "2019-01-25T16:48:54Z")
It always asks for the env variables, whatever I do!
I have S3 in my site settings, so I don’t know why this doen’t work:
```
unless GlobalSetting.use_s3?
puts <<~TEXT
Please provide the following environment variables
- DISCOURSE_S3_BUCKET
- DISCOURSE_S3_REGION
- DISCOURSE_S3_ACCESS_KEY_ID
- DISCOURSE_S3_SECRET_ACCESS_KEY
TEXT
exit 2
end
```
---
### Author:  [@zogstrip](https://meta.discourse.org/u/zogstrip)
#### Post date: [January 25, 2019, 4:53pm UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/12 "2019-01-25T16:53:26Z")
If `GlobalSetting.use_s3?` returns `false`, it means that S3 is [not setup properly](https://github.com/discourse/discourse/blob/78ddc82952d8766f516100abd09580f79a5b9c59/app/models/global_setting.rb#L81-L84).
---
### Author:  [@vulkanino](https://meta.discourse.org/u/vulkanino)
#### Post date: [January 25, 2019, 4:54pm UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/13 "2019-01-25T16:54:22Z")
But the new uploads are stored on S3, I’ve set it up in the admin panel.
Shoud I put those variables in app.yml? I’m confused.
---
### Author:  [@schleifer](https://meta.discourse.org/u/schleifer)
#### Post date: [January 25, 2019, 7:20pm UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/14 "2019-01-25T19:20:16Z")
Enter them on the command line before the `rake uploads:migrate_to_s3` like in [@zogstrip’s post above](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/10).
The backslashes permit the entire command to extend over multiple lines. It can also be entered on a single very long line:
```plaintext
DISCOURSE_S3_BUCKET=blablabla DISCOURSE_S3_REGION=blablabla DISCOURSE_S3_ACCESS_KEY_ID=blabalbal DISCOURSE_S3_SECRET_ACCESS_KEY=blablabla rake uploads:migrate_to_s3
```
---
### Author:  [@vulkanino](https://meta.discourse.org/u/vulkanino)
#### Post date: [January 26, 2019, 7:12am UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/15 "2019-01-26T07:12:21Z")
As written before, the error message appears just the same. Let me recap to be clear:
- S3 is configured in the site settings and it works. Uploads go to my S3 bucket (and backups too)
- Some previous uploads have been successfully migrated using the script. The script also rebaked the posts so it’s all good. I did a manual rebake just in case. It worked without asking for the env vars, but it worked partially.
- Some other uploads did not migrate. I tried running the script multiple times but the uploads were not picked up by the script. Thinking it was a bug in the script I went to github and found a newer version of the script. Launching the newer version it asks the env vars, even if I provide them in the command line or if I set them in the shell. No way.
---
### Author:  [@marcozambi](https://meta.discourse.org/u/marcozambi)
#### Post date: [January 26, 2019, 11:09am UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/16 "2019-01-26T11:09:00Z")
@Vulkanino, I think you are going through the same ordeal I had a couple of weeks ago.
I tried to summarise my experience (and some solutions) here.
> [@Migrate\_to\_s3 task fails for placing too many requests](https://meta.discourse.org/t/migrate-to-s3-task-fails-for-placing-too-many-requests/106305):
>
> So, I’ve been fighting for a few days for uploading my locally stored uploads to DigitalOcean Spaces, and after a lot of struggling I managed to run the rake uploads:migrate\_to\_s3, even if so far none of my tries are being successful. A bit of background info: I’m running Discourse 2.2 beta 7 +117 I have successfully set-up Discourse to store uploads and backups to Spaces, and I have them working reliably since days I have defined environmental variables in app.yml as [explained here](https://meta.discourse.org/t/migration-of-system-stored-images-to-s3-after-configuration-change/19605/52) but I had…
[https://meta.discourse.org/t/migration-of-system-stored-images-to-s3-after-configuration-change/19605/52](https://meta.discourse.org/t/migration-of-system-stored-images-to-s3-after-configuration-change/19605/52)
---
### Author:  [@vulkanino](https://meta.discourse.org/u/vulkanino)
#### Post date: [January 26, 2019, 12:57pm UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/17 "2019-01-26T12:57:45Z")
It’s the Marco’s bug I guess 😃 Well what’s the proposed solution then?
---
### Author:  [@vulkanino](https://meta.discourse.org/u/vulkanino)
#### Post date: [March 4, 2019, 10:06am UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/18 "2019-03-04T10:06:55Z")
After two months I made it!
Edited the GlobalSetting file and changed the function use\_s3? to always return true. It didn’t work in any other way.
Then I edited uploads.rake and commented out this block:
```
if SiteSetting.Upload.s3_cdn_url.blank?
puts "Please provide the 'DISCOURSE_S3_CDN_URL' environment variable"
exit 3
end
```
The migration still was not happy until I did:
`export DISCOURSE_S3_CDN_URL=""`
(though I don’t use a CDN).
> rake uploads:migrate\_to\_s3
> Migrating uploads to S3 for ‘default’…
> Uploading files to S3…
>
> - Listing local files… =\> 11473 files
> - Listing S3 files… =\> 13391 files
> - Syncing files to S3
Yay!
---
### Author:  [@csmu](https://meta.discourse.org/u/csmu)
#### Post date: [August 10, 2019, 9:47am UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/19 "2019-08-10T09:47:17Z")
> [@vulkanino](#):
>
> uploads.rake
What’s the path to this file?
---
### Author:  [@csmu](https://meta.discourse.org/u/csmu)
#### Post date: [August 10, 2019, 9:50am UTC](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487/20 "2019-08-10T09:50:30Z")
> [@csmu](#):
>
> What’s the path to this file?
```plaintext
/var/www/discourse# vi lib/tasks/uploads.rake
```
found using
```plaintext
/var/www/discourse# grep -r 'upload' . | grep rake | grep lib | grep tasks
```
[Next page](https://meta.discourse.org/t/uploads-migration-to-s3-fails/106487.md?page=2)