# Broken Images and Their S3 URLs
**URL:** https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889
**Category:** Support
**Created:** [June 25, 2020, 7:46am UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889 "2020-06-25T07:46:00Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@masterakay](https://meta.discourse.org/u/masterakay)
#### Post date: [June 25, 2020, 7:46am UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/1 "2020-06-25T07:46:00Z")
After the recent update to `2.5.0.beta7` all my images on the forum are broken.
The avatar is also replaced by default one. These are my suggestions on the dashboard:
My Staff Posts for Assets:
One such image: `/original/2X/4/4064b1cc32199de63ad2fa2b5bd2a0c5b6872c6d.png`
However, my S3 has no such image: (why did this happen?)
My favicon and other logos are missing: `/original/1X/b32666bfe0e0eac40aacb178c7430c9f7873b397.png`
However, my S3 has these files only:
Notice how the URL is wrong and there is no `1X` folder inside the `original` folder.
I tried following the tutorial of this post:
> [@Old image uploads become broken images](https://meta.discourse.org/t/old-image-uploads-become-broken-images/47236):
>
> For the past week or so (after the most recent upgrade?), I’ve noticed that some users’ uploaded images are broken. It is clear that they are pasting an image, as the code in their post looks like that, but the files are missing. For example, [this post](https://literatecomputing.com/t/success-with-social-bookmarking/4317) ([raw version](https://literatecomputing.com/raw/4317) has three broken images in it. I cannot imagine what the user could have done to break them. And stranger still, sometimes a post will have some images that work fine, but others that are missing. I’ve done find in the uploads direct…
But it did not fix anything.
---
### Author:  [@Benjamin\_D](https://meta.discourse.org/u/Benjamin_D)
#### Post date: [June 25, 2020, 9:35am UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/2 "2020-06-25T09:35:29Z")
> [@masterakay](#):
>
> One such image: `/original/2X/4/4064b1cc32199de63ad2fa2b5bd2a0c5b6872c6d.png`
may I point out that this image is not in your bucket, it’s 4061…41.png ? 🤔
Is it present locally ?
---
### Author:  [@masterakay](https://meta.discourse.org/u/masterakay)
#### Post date: [June 25, 2020, 10:07am UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/3 "2020-06-25T10:07:03Z")
> [@masterakay](#):
>
> However, my S3 has:
I’m sorry, wrong choice of words. I meant to say I noticed it isn’t in the S3 bucket, none of them are.
How do I check for it locally?
---
### Author:  [@Benjamin\_D](https://meta.discourse.org/u/Benjamin_D)
#### Post date: [June 25, 2020, 12:31pm UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/4 "2020-06-25T12:31:55Z")
Oh, makes indeed more sense 😅
You could check locally with any ssh client that you like, depending on your setup, the files should be in something like /var/discourse/shared/standalone/uploads/default/original/2X/
so you might `ls -R /var/discourse/shared/standalone/uploads/ | grep 4064b` to find the 4064b…d.png
---
### Author:  [@rithvikvibhu](https://meta.discourse.org/u/rithvikvibhu)
#### Post date: [June 26, 2020, 11:47am UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/5 "2020-06-26T11:47:43Z")
Talking about the same Discourse instance as @masterakay here.
We switched over to s3 over 2 years ago, none of the files in the local uploads directory are newer than 2018.
As I understand, images are uploaded in the “original” folder, and processed (cropped, reduced in size, etc.) and put in the “optimized” folder. The URLs used by discourse when uploading images are still from the “/original/1X/” folder. Somehow, with the upgrade, the “1X” folder is gone.
There’s a “default” folder which as a lot of uploads (images, etc.) as well as cropped versions. There are about 10k files here making it ~2.5 GB.
The problem is the image URL in the posts always have the “original” one.
Something we noticed is that the missing files are a subset of the files in the “default” folder.
New uploads work without issues and are stored in “/original/2X/”.
Does anyone have ideas where the “/original/1X/” folder may have gone and if it possible to bring it back?
---
### Author:  [@masterakay](https://meta.discourse.org/u/masterakay)
#### Post date: [June 26, 2020, 12:37pm UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/6 "2020-06-26T12:37:56Z")
@sam @codinghorror I’m so sorry to drag you into this, but it’s quite urgent. We’re students and sort of helpless with the college mounting pressure on us to fix the images.
---
### Author:  [@Ryan\_Nix](https://meta.discourse.org/u/Ryan_Nix)
#### Post date: [June 30, 2020, 12:51pm UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/7 "2020-06-30T12:51:41Z")
I’m seeing this as well, and it just started after a more recent update.
[https://discourse.sesp.northwestern.edu](https://discourse.sesp.northwestern.edu)
---
### Author:  [@schleifer](https://meta.discourse.org/u/schleifer)
#### Post date: [July 2, 2020, 6:02am UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/9 "2020-07-02T06:02:30Z")
How do you have S3 setup? Did you set any values in `app.yml` or just in the admin UI? It looks like something is unexpected for DISCOURSE\_S3\_BUCKET or DISCOURSE\_S3\_BACKUP\_BUCKET.
---
### Author:  [@masterakay](https://meta.discourse.org/u/masterakay)
#### Post date: [July 2, 2020, 7:44am UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/10 "2020-07-02T07:44:19Z")
Yes, like I explained, everything was fine until the recent update. We have followed all the steps.
In particular, we did something like this:
> [@Set up file and image uploads to S3](https://meta.discourse.org/t/setting-up-file-and-image-uploads-to-s3/7229):
>
> So, you want to use S3 to handle image uploads? Here’s the definitive guide, but also see [Configure an S3 compatible object storage provider for uploads](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916) to see how to configure your app.yml. S3 registration Head over to [https://aws.amazon.com/free/](https://aws.amazon.com/free/) and click on [Create a Free Account] During the create account process, make sure you provide payment information, otherwise you won’t be able to use S3. There’s no registration fee, you will only be charged for [what you use](https://aws.amazon.com/s3/pricing/), if you exceed the [AWS …](https://aws.amazon.com/free/)
> [@schleifer](#):
>
> DISCOURSE\_S3\_BUCKET or DISCOURSE\_S3\_BACKUP\_BUCKET
As for these values, I can’t seem to find any variable set in `app.yml`. It wasn’t previously required. Did something change?
---
### Author:  [@schleifer](https://meta.discourse.org/u/schleifer)
#### Post date: [July 2, 2020, 9:35am UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/11 "2020-07-02T09:35:51Z")
> [@masterakay](#):
>
> everything was fine until the recent update
Indeed. Something may not be working right in the code, but we don’t know what nor why. So we need more information.
> [@masterakay](#):
>
> We have followed all the steps.
There are two different ways to setup S3: You can configure the environment in `app.yml` or you can enter the values in the admin UI. The variables are named slightly differently in each.
The topic you linked to describes how to setup in the admin UI. If that is how you setup your site, can you tell us what you have for **`s3_upload_bucket`** , and **`s3_backup_bucket`**?
---
### Author:  [@masterakay](https://meta.discourse.org/u/masterakay)
#### Post date: [July 2, 2020, 12:23pm UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/12 "2020-07-02T12:23:19Z")
> [@schleifer](#):
>
> The topic you linked to describes how to setup in the admin UI. If that is how you setup your site, can you tell us what you have for **`s3_upload_bucket`** , and **`s3_backup_bucket`**?
---
### Author:  [@Ryan\_Nix](https://meta.discourse.org/u/Ryan_Nix)
#### Post date: [July 2, 2020, 1:57pm UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/13 "2020-07-02T13:57:37Z")
@schleifer Same for me. I haven’t changed anything in those settings, either. I also verified my AWS credentials are working.
---
### Author:  [@schleifer](https://meta.discourse.org/u/schleifer)
#### Post date: [July 2, 2020, 3:36pm UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/14 "2020-07-02T15:36:41Z")
Oh, that explains it.
The problem happened because those settings are not supposed to be the same value – search the [setup topic](https://meta.discourse.org/t/-/7229) for the line “Do I really need to use separate buckets for uploads and backups?”
A maintenance job ran on the contents of the backup bucket and affected the uploads because their values overlapped.
@sam we should probably enforce that in code, not just documentation
To fix up your site, there’s two steps:
First, you’ll need to change the backups prefix – adding `/backups` to the end as described in the setup topic, is sufficient.
After, move everything in the S3 bucket back into the correct place. Everything in the top “default” folder should get put back at the top level.
E.g. there’s probably “default/originals” folder there which should be moved up.
You’ll have to use the AWS web console or some other tool to browse the bucket.
---
### Author:  [@masterakay](https://meta.discourse.org/u/masterakay)
#### Post date: [July 2, 2020, 3:41pm UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/15 "2020-07-02T15:41:10Z")
Alright, thank you so much! I shall try the steps out and come back here in case anything ever happens. Any idea why this happened all of a sudden? 😅
---
### Author:  [@rithvikvibhu](https://meta.discourse.org/u/rithvikvibhu)
#### Post date: [July 3, 2020, 7:37am UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/17 "2020-07-03T07:37:21Z")
Hey @schleifer that makes sense and added the backups prefix to the bucket name.
As for the existing uploads, all the uploads are in `/default/` (not in sub-folders). The image urls in posts (and everywhere else) use `/original/*` or `/optimized/*`.
If we move everything in the default folder up one level (to the root), then the images will be in `/*`.
And no, there aren’t any folders within `defaults`, just upload files. It seems to contain files with standard 40-char hash filename as well as some with suffixes like “\_2\_10x10” (which I presume are from optimized).
How do you suggest fixing this? Fixing all posts with new links will take time. Is it possible to somehow group the files in the correct folders based on this filename?
---
### Author:  [@masterakay](https://meta.discourse.org/u/masterakay)
#### Post date: [July 13, 2020, 5:45am UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/18 "2020-07-13T05:45:03Z")
@schleifer any updates on this?
---
### Author:  [@schleifer](https://meta.discourse.org/u/schleifer)
#### Post date: [July 13, 2020, 10:06am UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/19 "2020-07-13T10:06:06Z")
> [@rithvikvibhu](#):
>
> As for the existing uploads, all the uploads are in `/default/` (not in sub-folders).
That’s…unexpected. We’ll have to fix up a bunch of stuff by hand, then.
The most important question would be “are new uploads going to the correct location?”
Assuming that’s true, you can put the old uploads in a known location and adjust their database entry. How many files are there in /default/?
---
### Author:  [@rithvikvibhu](https://meta.discourse.org/u/rithvikvibhu)
#### Post date: [July 13, 2020, 5:03pm UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/20 "2020-07-13T17:03:17Z")
The new uploads are working as expected into sub-folders, fortunately. And the links in posts point to the correct location.
There are \>10k files in `/default/`. Editing each post manually seems like a lot of work. Is there any way to script this? Maybe with a regex replace on all posts?
---
### Author:  [@schleifer](https://meta.discourse.org/u/schleifer)
#### Post date: [July 14, 2020, 12:48am UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/21 "2020-07-14T00:48:02Z")
> [@rithvikvibhu](#):
>
> Is there any way to script this?
That’s the plan, yes. The next thing to do is put all the AWOL files in a known location. In the bucket, what sub-directories are there under `/original/`? There should be `/1X/` and there may be others.
---
### Author:  [@rithvikvibhu](https://meta.discourse.org/u/rithvikvibhu)
#### Post date: [July 14, 2020, 5:10am UTC](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889/22 "2020-07-14T05:10:51Z")
This is the complete folder structure. New uploads (works fine) are stored in `/original/2X/*/.jpg`
```plaintext
/
|---backups
|---default
|---(bunch if backups .tar.gz files)
|---default
|---(10719 uploaded files pdf, images, etc.)
|---inventory
|---1/pesuioforum/{optimized,original}/[DATE]/manifest.{checksum,json}
|---optimized
|---1X
|---(2 files)
|---2X
|---{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f}
|---(uploaded files that start with the folder's name char above) (1055 files in total)
|---original
|---2X
|---(Same as optimized/2X, 520 files)
|---tombstone
|---optimized
|---1X
|---(2 files)
|---original
|---2X
|---(Same as optimized/2X, 1 file)
```
> The next thing to do is put all the AWOL files in a known location.
The `/default/` already seems to have all the images.
[Next page](https://meta.discourse.org/t/broken-images-and-their-s3-urls/155889.md?page=2)