# Images are disappearing off of s3!
**URL:** https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801
**Category:** Bug
**Created:** [09.Сентябрь.2018 19:58:51 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801 "2018-09-09T19:58:51Z")
**Posts on this page:** 18
**Page:** 3
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [17.Сентябрь.2018 16:28:57 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/43 "2018-09-17T16:28:57Z")
The job has been renamed to `rake uploads:recover`
---
### Author:  [@clay](https://meta.discourse.org/u/clay)
#### Post date: [17.Сентябрь.2018 16:41:17 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/44 "2018-09-17T16:41:17Z")
Thanks, Alan. That didn’t fix our issue. I’m not sure what’s happening. After the last update, some older images now are appearing as broken links, though there’s zero chance they were manually deleted.
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [17.Сентябрь.2018 16:44:23 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/45 "2018-09-17T16:44:23Z")
For those posts, can you provide me with the created\_at and baked\_at columns?
---
### Author:  [@clay](https://meta.discourse.org/u/clay)
#### Post date: [17.Сентябрь.2018 17:33:03 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/46 "2018-09-17T17:33:03Z")
I can’t dig into all of them at the moment, but for the one that caused some hand-wringing this morning:
- `created_at`: `Mon, 16 Oct 2017 13:00:38 UTC +00:00`
- `baked_at`: `Mon, 17 Sep 2018 16:14:14 UTC +00:00`
I just rebaked all of our posts to see if it would fix the issue, but it didn’t.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [17.Сентябрь.2018 21:15:06 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/47 "2018-09-17T21:15:06Z")
Try rebaking and then running recover on that post odds are though the image is long gone
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [18.Сентябрь.2018 03:06:23 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/48 "2018-09-18T03:06:23Z")
Try running
```plaintext
.launcher rebuild app
./launcher enter app
DRY_RUN=1 rake uploads:recover
```
to see the number of images affected.
If `rake uploads:recover` didn’t work, you might have to recover the images from a backup. I unfortunately [uncovered another bug](https://github.com/discourse/discourse/commit/ce6a0a5e9e33e3cf7459c180eb566e5f12ec1371) in our tombstone today where uploads in the tombstone may be deleted immediately instead of being retained for `SiteSetting.purge_deleted_uploads_grace_period_days`.
---
### Author:  [@clay](https://meta.discourse.org/u/clay)
#### Post date: [18.Сентябрь.2018 12:54:20 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/49 "2018-09-18T12:54:20Z")
There were 60 or so images listed when I ran `DRY_RUN=1 rake uploads:recover`
Unfortunately, this did not fix the missing images in the affected posts. I’ll consult with my colleague to see if recovering from a backup is feasible.
Before I go wild with restoring, etc… what is the expected behavior of running the `uploads:recover` task?
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [19.Сентябрь.2018 03:22:57 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/50 "2018-09-19T03:22:57Z")
> [@clay](#):
>
> what is the expected behavior of running the `uploads:recover` task?
`uploads:recovery` looks for posts that contain “transparent.png” which is a sign that an upload is missing from the folder.
---
### Author:  [@soke](https://meta.discourse.org/u/soke)
#### Post date: [20.Сентябрь.2018 13:04:33 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/51 "2018-09-20T13:04:33Z")
I’m having the same problem on my forum.
I’m running version v2.2.0.beta2 +5 and some images are disappearing off from s3/original/1X folder.
I’ve enabled CloudTrail to see who is deleting them. I think it can be the purge\_tombstone lifecycle rule that for any reason is failing or something…
Just to say something else, all deleted images seems to be on the s3/tombstone/original/1X folder.
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [20.Сентябрь.2018 15:12:45 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/52 "2018-09-20T15:12:45Z")
hmm how many lines does the following print?
```plaintext
./launcher enter app
rails c
UploadRecovery.new(dry_run: true).recover(Post.where("baked_at >= ?", SiteSetting.purge_deleted_uploads_grace_period_days.days
.ago))
```
---
### Author:  [@clay](https://meta.discourse.org/u/clay)
#### Post date: [20.Сентябрь.2018 16:25:02 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/53 "2018-09-20T16:25:02Z")
> [@tgxworld](#):
>
> UploadRecovery.new(dry\_run: true).recover(Post.where(“baked\_at \>= ?”, SiteSetting.purge\_deleted\_uploads\_grace\_period\_days))
I get this…
```plaintext
ActiveRecord::StatementInvalid: PG::UndefinedFunction: ERROR: operator does not exist: timestamp without time zone >= integer
LINE 1: ...HERE ("posts"."deleted_at" IS NULL) AND (baked_at >= 30) AND...
```
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [20.Сентябрь.2018 16:26:26 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/54 "2018-09-20T16:26:26Z")
Oops I’ve updated the steps again.
> [@clay](#):
>
> Unfortunately, this did not fix the missing images in the affected posts
O wait you are supposed to run it without dry run to fix the posts.
---
### Author:  [@clay](https://meta.discourse.org/u/clay)
#### Post date: [20.Сентябрь.2018 16:29:18 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/55 "2018-09-20T16:29:18Z")
I see 54 images listed – same number as I saw when running `DRY_RUN=1 rake uploads:recover`
---
### Author:  [@clay](https://meta.discourse.org/u/clay)
#### Post date: [20.Сентябрь.2018 16:31:30 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/56 "2018-09-20T16:31:30Z")
> [@tgxworld](#):
>
> O wait you are supposed to run it without dry run to fix the posts.
I did. It did not fix the problem.
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [20.Сентябрь.2018 16:32:14 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/57 "2018-09-20T16:32:14Z")
If that is the case you all have to restore it from a backup. The images have been removed from the tombstone.
---
### Author:  [@clay](https://meta.discourse.org/u/clay)
#### Post date: [20.Сентябрь.2018 16:32:52 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/58 "2018-09-20T16:32:52Z")
Ok, will do. Is the root cause addressed or is there reason to think this will continue to happen?
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [20.Сентябрь.2018 16:34:15 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/59 "2018-09-20T16:34:15Z")
We’ve pushed out the fix and improved our test cases. Currently we’re not expecting images to go missing.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [20.Сентябрь.2018 20:12:41 UTC](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801/60 "2018-09-20T20:12:41Z")
[Предыдущая страница](https://meta.discourse.org/t/images-are-disappearing-off-of-s3/96801.md?page=2)