# Upload remapping is a bit too enthousiastic

**URL:** https://meta.discourse.org/t/upload-remapping-is-a-bit-too-enthousiastic/118889
**Category:** Bug
**Created:** [May 28, 2019, 3:44pm UTC](https://meta.discourse.org/t/upload-remapping-is-a-bit-too-enthousiastic/118889 "2019-05-28T15:44:56Z")
**Posts on this page:** 14
**Page:** 1

<div class="post-metadata">

### Author: ![RGJ](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/rgj/32/523185_2.png) [@RGJ](https://meta.discourse.org/u/RGJ)
#### Post date: [May 28, 2019, 3:44pm UTC](https://meta.discourse.org/t/upload-remapping-is-a-bit-too-enthousiastic/118889/1 "2019-05-28T15:44:56Z")

</div>

When a backup is restored to a database with a different name, there is a neat little piece of code in `restorer.rb` that attempts to fix this.

> <https://github.com/discourse/discourse/blob/main/lib/backup_restore/restorer.rb#L448>

However, this renames ALL upload filenames, not just the ones that were extracted from the archive. If the database name occurs in an S3 path (which obviously does not change), it changes it as well, causing all images to fail loading.

---

<div class="post-metadata">

### Author: ![codinghorror](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/codinghorror/32/110067_2.png) [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [May 29, 2019, 12:44am UTC](https://meta.discourse.org/t/upload-remapping-is-a-bit-too-enthousiastic/118889/2 "2019-05-29T00:44:11Z")

</div>

What do you think here @tgxworld?

---

<div class="post-metadata">

### Author: ![supermathie](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/supermathie/32/507518_2.png) [@supermathie](https://meta.discourse.org/u/supermathie)
#### Post date: [May 29, 2019, 4:15pm UTC](https://meta.discourse.org/t/upload-remapping-is-a-bit-too-enthousiastic/118889/4 "2019-05-29T16:15:42Z")

</div>

> [@RGJ](#):
>
> DbHelper.remap(“uploads/#{previous\_db\_name}”, “uploads/#{current\_db\_name}”)

Would changing this to include the trailing slash have helped?

```plaintext
DbHelper.remap("uploads/#{previous_db_name}/", "uploads/#{current_db_name}/")

```

That feels more correct in any case.

---

<div class="post-metadata">

### Author: ![RGJ](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/rgj/32/523185_2.png) [@RGJ](https://meta.discourse.org/u/RGJ)
#### Post date: [May 29, 2019, 4:59pm UTC](https://meta.discourse.org/t/upload-remapping-is-a-bit-too-enthousiastic/118889/5 "2019-05-29T16:59:28Z")

</div>

No - I’m talking about filenames like `//discourse-cloud-file-uploads.s3.dualstack.us-west-2.amazonaws.com/standard11/uploads/DBNAME/original/1X/abcd.....jpg`

So I think we’d need to exclude uploads starting with `//` or something like that.

---

<div class="post-metadata">

### Author: ![tgxworld](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tgxworld/32/106117_2.png) [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [May 30, 2019, 1:34am UTC](https://meta.discourse.org/t/upload-remapping-is-a-bit-too-enthousiastic/118889/6 "2019-05-30T01:34:44Z")

</div>

We’re experiencing the pain of this quite abit while migrating local uploads to S3. Unfortunately, no amount of hacky remaps can fix this properly. What we’re currently working on at the moment is to remove/disallow upload links in `Post#raw`. The short upload url, `![test](upload://asdikajsdiasds.png)`, will be used as the public interface for access to uploads in raw.

Once that is done, we will only have to remap the urls in the uploads table and do a rebake of affected posts.

---

<div class="post-metadata">

### Author: ![RGJ](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/rgj/32/523185_2.png) [@RGJ](https://meta.discourse.org/u/RGJ)
#### Post date: [May 30, 2019, 7:30am UTC](https://meta.discourse.org/t/upload-remapping-is-a-bit-too-enthousiastic/118889/7 "2019-05-30T07:30:35Z")

</div>

“only” have to do a rebake… yes, on one hand I think it’s good to get rid of the hacky remaps, on the other hand it seems like the backup/restore and migrate from/to S3 processes are getting more and more complicated and hard.

See also a number of issues in [Migrate\_from\_s3 problems](https://meta.discourse.org/t/migrate-from-s3-problems/119064) which are kind of related but they’re not all covered by your proposed solution.

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [May 30, 2019, 7:52am UTC](https://meta.discourse.org/t/upload-remapping-is-a-bit-too-enthousiastic/118889/8 "2019-05-30T07:52:24Z")

</div>

> [@RGJ](#):
>
> getting more and more complicated and hard.

I do not think so, the moves will get **significantly** simpler. Walk the uploads table and upload all files to new s3 bucket, rebake and you are done.

They do though become more expensive computation wise cause a rebake is required.

This is a cost I think is worth bearing given we get a correct move at the end of the process. But, we have the flexibility of still doing database replace 🎢 as we do today for the `cooked` column and post revisions if that is how we roll, the key change in the process is that we leave `posts.raw` alone and then have 0% of corruption there. `posts.raw` and `posts_uploads` auto generated map will always be stable.

---

<div class="post-metadata">

### Author: ![RGJ](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/rgj/32/523185_2.png) [@RGJ](https://meta.discourse.org/u/RGJ)
#### Post date: [May 30, 2019, 8:00am UTC](https://meta.discourse.org/t/upload-remapping-is-a-bit-too-enthousiastic/118889/9 "2019-05-30T08:00:40Z")

</div>

Ok - good point.

I think a significant improvement would be to walk the uploads table instead of posts (see the topic I linked above) and that would indeed leave the flexibility to replace in `posts.cooked`

---

<div class="post-metadata">

### Author: ![codinghorror](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/codinghorror/32/110067_2.png) [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [November 21, 2019, 4:20pm UTC](https://meta.discourse.org/t/upload-remapping-is-a-bit-too-enthousiastic/118889/10 "2019-11-21T16:20:52Z")

</div>

Is this still an issue @sam?

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [November 21, 2019, 11:02pm UTC](https://meta.discourse.org/t/upload-remapping-is-a-bit-too-enthousiastic/118889/11 "2019-11-21T23:02:48Z")

</div>

`upload://` links in raw markdown help this a lot. I am not sure if @gerhard amended it so we queue a complete rebake post restore. (I think we do)

---

<div class="post-metadata">

### Author: ![gerhard](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/gerhard/32/119479_2.png) [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [November 21, 2019, 11:40pm UTC](https://meta.discourse.org/t/upload-remapping-is-a-bit-too-enthousiastic/118889/12 "2019-11-21T23:40:29Z")

</div>

> [@sam](#):
>
> complete rebake post restore

It depends. We do a rebake of all posts that contain uploads when optimized images aren’t included in backups (which is the default). Otherwise we rely only on the remaps.

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [November 21, 2019, 11:43pm UTC](https://meta.discourse.org/t/upload-remapping-is-a-bit-too-enthousiastic/118889/13 "2019-11-21T23:43:25Z")

</div>

We should probably change it to unconditionally rebake, its safer.

---

<div class="post-metadata">

### Author: ![gerhard](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/gerhard/32/119479_2.png) [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [November 21, 2019, 11:45pm UTC](https://meta.discourse.org/t/upload-remapping-is-a-bit-too-enthousiastic/118889/14 "2019-11-21T23:45:58Z")

</div>

Sure, I just changed it. It will be included in the big refactoring of backups & restores.

---

<div class="post-metadata">

### Author: ![codinghorror](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/codinghorror/32/110067_2.png) [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [November 22, 2019, 6:22am UTC](https://meta.discourse.org/t/upload-remapping-is-a-bit-too-enthousiastic/118889/15 "2019-11-22T06:22:14Z")

</div>


