# Changing s3 bucket for uploads
**URL:** https://meta.discourse.org/t/changing-s3-bucket-for-uploads/188553
**Category:** Support
**Created:** [April 29, 2021, 2:12pm UTC](https://meta.discourse.org/t/changing-s3-bucket-for-uploads/188553 "2021-04-29T14:12:22Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@Jite](https://meta.discourse.org/u/Jite)
#### Post date: [April 29, 2021, 2:12pm UTC](https://meta.discourse.org/t/changing-s3-bucket-for-uploads/188553/1 "2021-04-29T14:12:23Z")
Hi there!
We are migrating all our uploads/images between two different s3 compatible services (both are digital ocean spaces if it matters) and I have decided that we are stuck in a quite bad state.
I’ll start with explaining how the migration was done:
1. We cloned/synced the initial bucket to the new bucket with rclone
2. All references in the `Files` page in discourse administration was updated to the new endpoints
3. A re-bake was ran
Sadly, this did not do what we wanted, and now all images are “gone” from the forum. They are still in the s3 bucket (and luckily in the old one still as well) but no post can find their respective image.
The size of the bucket is about 60gb, so it’s (even if not extreme) quite a huge bit of data.
I’ve rebuilt the container, I’ve tried to recover stuff from the tombstone, I’ve done pretty much everything I can think of or find in the support forum or rake tasks.
I’ve as well tried with a database replace (via discourse remap).
Each image looks basically like this in baked content at the moment:
```plaintext
```
Which makes me think that the b64-sha of the link is either broken or the image sha have changed for some reason.
Have anyone done this before? Are all images lost forever? (yes yes, I have a backup and the old images, so I know that there is a way).
---
### Author:  [@Jite](https://meta.discourse.org/u/Jite)
#### Post date: [April 29, 2021, 2:14pm UTC](https://meta.discourse.org/t/changing-s3-bucket-for-uploads/188553/2 "2021-04-29T14:14:41Z")
Might be worth mentioning that I have tried to use the CDN uri provided for the spaces bucket as well (with a rebake).
---
### Author:  [@Jite](https://meta.discourse.org/u/Jite)
#### Post date: [April 29, 2021, 6:45pm UTC](https://meta.discourse.org/t/changing-s3-bucket-for-uploads/188553/3 "2021-04-29T18:45:30Z")
Output from missing uploads:
```plaintext
rake posts:missing_uploads
Looking for missing uploads on: default
Fixing missing uploads:
🚫
17075 post uploads are missing.
16906 uploads are missing.
1 of 16906 are old scheme uploads.
14646 of 139801 posts are affected.
```
post\_uploads have 3448 entries
optimized\_images have 25681 entries
uploads have 5764 entries
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [April 30, 2021, 1:15am UTC](https://meta.discourse.org/t/changing-s3-bucket-for-uploads/188553/4 "2021-04-30T01:15:29Z")
You can see [Moving from one S3 bucket to another](https://meta.discourse.org/t/moving-from-one-s3-bucket-to-another/184779)
I think that i have a draft of howto that I will try to post tomorrow.
---
### Author:  [@Jite](https://meta.discourse.org/u/Jite)
#### Post date: [April 30, 2021, 7:10am UTC](https://meta.discourse.org/t/changing-s3-bucket-for-uploads/188553/5 "2021-04-30T07:10:33Z")
That would be very helpful, tyvm!
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [April 30, 2021, 4:05pm UTC](https://meta.discourse.org/t/changing-s3-bucket-for-uploads/188553/6 "2021-04-30T16:05:22Z")
Hey @Jite !
See if this works for you. If it does, I’ll go about creating a proper #howto
### Old buckets
This assumes that you can install and configure a tool to move your data from your old bucket to a local machine and then again do the same from local to the new bucket. See [aws cli sync](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) (which can be configured for non-AWS buckets) and [gsutil rsync](https://download.huihoo.com/google/gdgdevkit/DVD1/developers.google.com/storage/docs/gsutil/commands/rsync.html) for information. If you have huge amounts of data or are moving between buckets on the same provider, then you might want to investigate methods that move the data directly between buckets.
Get in a directory suitable for a holding space. (e.g., `mkdir temp-bucket; cd temp-bucket`) before doing something like the following. These examples include the `-n` and `--dry-run` switches to show you what will happen. If that looks like what you want, run the command again without that switch.
### Move old data from old bucket to to local
```plaintext
gsutil rsync -r -n gs://=OLD= .
```
or
```plaintext
aws s3 sync s3://=OLD= .
```
### Move data from local to new bucket
```plaintext
gsutil rsync -r -n . gs://=NEW=
```
or
```plaintext
aws s3 sync . s3://=NEW=
```
### Updating the database to use the new bucket
You’’ do these commands at the Rails console, to get there, you’ll do a
```
cd /var/discourse
./launcher enter app
rails c
```
For the new bucket, upload an image with the new configuration and do this:
```plaintext
Upload.last.url
```
You should see something like
```
=> "//discourse-bucket.s3.dualstack.us-east-2.amazonaws.com/`original/2X/7/12345fbea574afc4e02db80107e6682430aede2c.png"
```
You’d then get `discourse-bucket.s3.dualstack.us-east-2.amazonaws.com` for the new bucket. Get the old bucket hostname similarly from the above.
Use this to check that your uploads are where you think they are:
```plaintext
Upload.order(Arel.sql('RANDOM()')).limit(10).pluck(:id, :url)
```
Now, you’ll update the database to use the new bucket rather than the old one. `DbHelper.remap` will replace occurrences in all tables.
```plaintext
DbHelper.remap("//=OLDHOST=/","//=NEWHOST=/")
```
### Moving to AWS might require clearing your `s3_endpoint`.
NOTE: If you have a `s3_endpoint` defined in your SiteSettings in the database and switch to AWS (where no endpoint is needed), then you’ll need to clear that site setting after you build the new container with the updated settings (or after you restore a database that has it set).
### Rebake posts that refer to bucket rather than S3 CDN
If you have posts that link directly to the new s3 bucket (perhaps you didn’t have an `s3_cdn_url` defined before), then here’s how to rebake only the posts that need it.
Get the posts:
```
posts=Post.where("cooked like '%=NEWHOST=%'")
```
See how many:
```
posts.count
```
Rebake those posts:
```
posts.each do |p| p.rebake! end
```
Or, just replace the bucket with the cdn:
```plaintext
posts.each do |p|
p.cooked.gsub!(/=NEWHOST=/,"=CDN=")
p.save!
end
```
---
### Author:  [@Jite](https://meta.discourse.org/u/Jite)
#### Post date: [May 1, 2021, 12:23pm UTC](https://meta.discourse.org/t/changing-s3-bucket-for-uploads/188553/7 "2021-05-01T12:23:48Z")
Thank you for the response.
This is basically what I did the last time, but I tried this again. The issue is that posts.count returns 0. All posts have the transperent.png file in the cooked post and contains a hash in the uncooked.
Is there any way to make it resolve the image correctly during bake?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [May 1, 2021, 1:51pm UTC](https://meta.discourse.org/t/changing-s3-bucket-for-uploads/188553/8 "2021-05-01T13:51:57Z")
Hmm. Right. That cooked change works only to avoid the remake. If a remake does not work then something else is wrong. Maybe the assets are not where discourse thinks?
---
### Author:  [@Jite](https://meta.discourse.org/u/Jite)
#### Post date: [May 1, 2021, 3:17pm UTC](https://meta.discourse.org/t/changing-s3-bucket-for-uploads/188553/9 "2021-05-01T15:17:37Z")
Well, it’s possible, but the “move” was pretty much a 1:1 move of all files in the bucket, hehe…
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [May 1, 2021, 3:30pm UTC](https://meta.discourse.org/t/changing-s3-bucket-for-uploads/188553/10 "2021-05-01T15:30:04Z")
So you can replace the old bucket url with the new and it works?
Do the values in Uploads look right?
---
### Author:  [@Jite](https://meta.discourse.org/u/Jite)
#### Post date: [May 2, 2021, 12:28pm UTC](https://meta.discourse.org/t/changing-s3-bucket-for-uploads/188553/11 "2021-05-02T12:28:44Z")
I’m a bit scared that changing back to old bucket might start of a job to move everything to tombstone due to them being “old” now, but yes, the database seems to point to the correct (new location) images, it’s basically just the old posts that don’t resolve to the correct image (I’m guessing…).
---
### Author:  [@Jite](https://meta.discourse.org/u/Jite)
#### Post date: [May 11, 2021, 8:49pm UTC](https://meta.discourse.org/t/changing-s3-bucket-for-uploads/188553/12 "2021-05-11T20:49:26Z")
After running the toombstone recovery rake task and then the fix\_missing\_uploads rake task, I finally got it to start “fix” the images.
It seems like it’s downloading them and uploading them again, and it takes _a lot of time_ and uses _a lot of resources_, but at the least the users will get their images back!
Thank you for the help @pfaffman 🙂
---
### Author:  [@system](https://meta.discourse.org/u/system)
#### Post date: [June 10, 2021, 8:50pm UTC](https://meta.discourse.org/t/changing-s3-bucket-for-uploads/188553/13 "2021-06-10T20:50:11Z")
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.