# Images are disappearing off of s3!
**URL:** https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801
**Category:** Bug
**Created:** [September 9, 2018, 7:58pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801 "2018-09-09T19:58:51Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@davidkingham](https://meta.discourse.org/u/davidkingham)
#### Post date: [September 9, 2018, 7:58pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/1 "2018-09-09T19:58:51Z")
I have no idea what is going on but our images are disappearing more and more by the minute. It’s not a CDN issue because I can’t even find the files on s3 anymore. For example: [https://community.naturephotographers.network/t/alpine-bliss/490](https://community.naturephotographers.network/t/alpine-bliss/490)
It started off with one image missing this morning, now at least 10 have disappeared.
I am in panic mode, can anyone help?
---
### Author:  [@davidkingham](https://meta.discourse.org/u/davidkingham)
#### Post date: [September 9, 2018, 8:19pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/2 "2018-09-09T20:19:25Z")
Now up to 17. The only thing I changed this morning was removing the Babble plugin and the question/answer plugin.
---
### Author:  [@Stephen](https://meta.discourse.org/u/Stephen)
#### Post date: [September 9, 2018, 8:20pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/3 "2018-09-09T20:20:29Z")
Were the images originally uploaded to these posts?
Can you share your versions of everything seen in /admin/upgrade?
---
### Author:  [@davidkingham](https://meta.discourse.org/u/davidkingham)
#### Post date: [September 9, 2018, 8:20pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/4 "2018-09-09T20:20:52Z")
Yes, they were displaying fine yesterday.
---
### Author:  [@davidkingham](https://meta.discourse.org/u/davidkingham)
#### Post date: [September 9, 2018, 8:25pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/5 "2018-09-09T20:25:01Z")
I also purged the cache on my CDN and and cloudflare after talking to maxcdn about the first image disappearing, now it seems to be multiplying.
---
### Author:  [@Stephen](https://meta.discourse.org/u/Stephen)
#### Post date: [September 9, 2018, 8:39pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/6 "2018-09-09T20:39:31Z")
All of the posts with missing images are at least two or three days old. Do you have any form of object expiration configured on your bucket?
If you followed the S3 guide it isn’t being used as a cache, it’s the place discourse is offloading originals to. Any expiration settings means the file gets offloaded and then totally separate from discourse it’s being deleted by a process at amazon.
Why are you using a cdn and cloudflare in addition to S3? Is maxcdn providing the S3 bucket?
---
### Author:  [@davidkingham](https://meta.discourse.org/u/davidkingham)
#### Post date: [September 9, 2018, 9:12pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/7 "2018-09-09T21:12:16Z")
I’ve figured out this is related to tombstone, I found the images in a tombstone folder on s3. There was a lifecycle rule in my bucket called purge\_tombstone that was moving the images out of the main folder. I have disabled the rule but I’m wondering if this is a bug causing this? @zogstrip @codinghorror any thoughts?
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [September 9, 2018, 9:16pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/8 "2018-09-09T21:16:05Z")
> [@Stephen](#):
>
> Were the images originally uploaded to these posts?
Additionally, and critically, _were the images in “chat” or “Q&A” topics specific to these plugins you removed?_
---
### Author:  [@davidkingham](https://meta.discourse.org/u/davidkingham)
#### Post date: [September 9, 2018, 9:18pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/9 "2018-09-09T21:18:28Z")
> [@codinghorror](#):
>
> Additionally, and critically, _were the images in “chat” or “Q&A” topics specific to these plugins you removed?_
No, I wasn’t even using Q&A and very few people used the chat. I did delete the groups that Babble left behind…
---
### Author:  [@Stephen](https://meta.discourse.org/u/Stephen)
#### Post date: [September 9, 2018, 9:18pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/10 "2018-09-09T21:18:34Z")
> [@davidkingham](#):
>
> I found the images in a tombstone folder on s3
How was the bucket created and following what process?
---
### Author:  [@davidkingham](https://meta.discourse.org/u/davidkingham)
#### Post date: [September 9, 2018, 9:19pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/11 "2018-09-09T21:19:58Z")
> [@Stephen](#):
>
> How was the bucket created and following what process?
I followed this guide and had no issues for months
> [@Set up file and image uploads to S3](https://meta.discourse.org/t/setting-up-file-and-image-uploads-to-s3/7229):
>
> So, you want to use S3 to handle image uploads? Here’s the definitive guide, but also see [Configure an S3 compatible object storage provider for uploads](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916) to see how to configure your app.yml. S3 registration Head over to [https://aws.amazon.com/free/](https://aws.amazon.com/free/) and click on [Create a Free Account] During the create account process, make sure you provide payment information, otherwise you won’t be able to use S3. There’s no registration fee, you will only be charged for [what you use](https://aws.amazon.com/s3/pricing/), if you exceed the [AWS …](https://aws.amazon.com/free/)
---
### Author:  [@davidkingham](https://meta.discourse.org/u/davidkingham)
#### Post date: [September 9, 2018, 9:43pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/12 "2018-09-09T21:43:52Z")
I was able to move all the images back from the tombstone folder and everything is running smoothly now. The question now becomes why were the images tombstoned in the first place and how do I avoid this in the future?
I found this similar post, but I still can’t put two and two together for what caused this.
> [@Old image uploads become broken images](https://meta.discourse.org/t/old-image-uploads-become-broken-images/47236):
>
> For the past week or so (after the most recent upgrade?), I’ve noticed that some users’ uploaded images are broken. It is clear that they are pasting an image, as the code in their post looks like that, but the files are missing. For example, [this post](https://literatecomputing.com/t/success-with-social-bookmarking/4317) ([raw version](https://literatecomputing.com/raw/4317) has three broken images in it. I cannot imagine what the user could have done to break them. And stranger still, sometimes a post will have some images that work fine, but others that are missing. I’ve done find in the uploads direct…
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [September 9, 2018, 9:46pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/13 "2018-09-09T21:46:26Z")
Are you on latest test passed?
---
### Author:  [@davidkingham](https://meta.discourse.org/u/davidkingham)
#### Post date: [September 9, 2018, 9:46pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/14 "2018-09-09T21:46:54Z")
> [@sam](#):
>
> Are you on latest test passed?
Yes updated this morning
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [September 9, 2018, 9:49pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/15 "2018-09-09T21:49:37Z")
Run a global rebake at least that will stop any new stuff from breaking
Is sidekiq crashing, did you lose a bunch of jobs? What version were you running prior to upgrade?
---
### Author:  [@gege](https://meta.discourse.org/u/gege)
#### Post date: [September 9, 2018, 9:56pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/16 "2018-09-09T21:56:06Z")
hello David,
i met this problem twice before. The latest one happened two days ago. This kind of problem does not happen often on my website.
But i uploaded my images on the hosting server, rather than s3.
I found out that my “bad” writing habits might cause problems.
my “bad” writting habit:
I will **_upload images first_** than **_cut the markdown links_** into the local editor. But i don’t close the discourse editor. Afterwards, i will **_paste them back_**.
Why do i say that?
> “I found out that my “bad” writing habits might cause problems.”
Because the time i set for the following setting just equal the time them disappeared.
`clean orphan uploads grace period hours`
### But I am not sure if it is caused by this reason.🤔
When i totally use the discourse editor ,everything is fine.
What i have done for this:
> I deleted the topics then I re-uploaded the content.
> I rebaked posts.
> I rebuilded the discourse docker instance.
The images are still there. Happy ending 😅
---
### Author:  [@davidkingham](https://meta.discourse.org/u/davidkingham)
#### Post date: [September 9, 2018, 10:25pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/17 "2018-09-09T22:25:40Z")
> [@sam](#):
>
> Run a global rebake at least that will stop any new stuff from breaking
Simple as this?
```
cd /var/discourse
./launcher enter app
rake posts:rebake
```
> [@sam](#):
>
> Is sidekiq crashing, did you lose a bunch of jobs? What version were you running prior to upgrade?
I don’t think it is, I went to /sidekiq and it seems to be running but I am ignorant on this. There are no dead jobs, 73 fails but I don’t know the timeline.
It seems to me that something I did this morning caused tombstone to run, is there any way I can identify that?
---
### Author:  [@Stephen](https://meta.discourse.org/u/Stephen)
#### Post date: [September 9, 2018, 10:29pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/18 "2018-09-09T22:29:08Z")
> [@davidkingham](#):
>
> It seems to me that something I did this morning caused tombstone to run, is there any way I can identify that?
That’s what we’re all trying to get to- something is happening because something _changed_.
But depending on the expiration policy it may not be a change today, it could be n days ago where n is whatever expiration period was set, so it could be 3 hours, 3 days or 3 weeks.
Even personal sites need some kind of schedule of change, I’d recommend a thread in /staff at the minimum. It’s the quickest way to pin down the source of such problems.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [September 9, 2018, 11:42pm UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/19 "2018-09-09T23:42:06Z")
> [@davidkingham](#):
>
> It seems to me that something I did this morning caused tombstone to run, is there any way I can identify that?
@tgxworld and I isolated a condition where uploads were missing associations with posts.
For the last 5 or so years we had a sidekiq deferred job create the association saying that upload X belongs to post Y. This has worked OK but has some pretty bad edge conditions.
If for some reason the job was mega delayed or lost then the associations are not created. If the associations are missing there is a regular job that moves the uploads to tombstone, there is a second policy that eventually deletes from tombstone.
Our new process is to create the association **just as you save** the post. This means this edge case can not happen anymore. I recommended a rebake to make sure if the job missed on a few posts that we would still create the associations.
We fixed this issue late last week, so there is a short transition period in place.
---
### Author:  [@davidkingham](https://meta.discourse.org/u/davidkingham)
#### Post date: [September 10, 2018, 3:32am UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/21 "2018-09-10T03:32:39Z")
Thanks Sam, I ran the rebake and now I seem to be in worse shape. Now the images don’t even display a box where the image should be, example: [https://community.naturephotographers.network/t/the-chamber-of-wonders/495](https://community.naturephotographers.network/t/the-chamber-of-wonders/495)
FYI I manually moved all the files in tombstone to their proper locations rather than doing an rsync, which may have been the fatal flaw and broke the images, what are my options now?
[Next page](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801.md?page=2)