Images are disappearing off of s3!

Bug
43

The job has been renamed to rake uploads:recover

2 Likes
44

Thanks, Alan. That didn’t fix our issue. I’m not sure what’s happening. After the last update, some older images now are appearing as broken links, though there’s zero chance they were manually deleted.

45

For those posts, can you provide me with the created_at and baked_at columns?

46

I can’t dig into all of them at the moment, but for the one that caused some hand-wringing this morning:

  • created_at: Mon, 16 Oct 2017 13:00:38 UTC +00:00
  • baked_at: Mon, 17 Sep 2018 16:14:14 UTC +00:00

I just rebaked all of our posts to see if it would fix the issue, but it didn’t.

47

Try rebaking and then running recover on that post odds are though the image is long gone

48

Try running

.launcher rebuild app
./launcher enter app
DRY_RUN=1 rake uploads:recover

to see the number of images affected.

If rake uploads:recover didn’t work, you might have to recover the images from a backup. I unfortunately uncovered another bug in our tombstone today where uploads in the tombstone may be deleted immediately instead of being retained for SiteSetting.purge_deleted_uploads_grace_period_days.

6 Likes
49

There were 60 or so images listed when I ran DRY_RUN=1 rake uploads:recover

Unfortunately, this did not fix the missing images in the affected posts. I’ll consult with my colleague to see if recovering from a backup is feasible.

Before I go wild with restoring, etc… what is the expected behavior of running the uploads:recover task?

50

uploads:recovery looks for posts that contain “transparent.png” which is a sign that an upload is missing from the folder.

3 Likes
51

I’m having the same problem on my forum.

I’m running version v2.2.0.beta2 +5 and some images are disappearing off from s3/original/1X folder.

I’ve enabled CloudTrail to see who is deleting them. I think it can be the purge_tombstone lifecycle rule that for any reason is failing or something…

Just to say something else, all deleted images seems to be on the s3/tombstone/original/1X folder.

52

hmm how many lines does the following print?

./launcher enter app
rails c
UploadRecovery.new(dry_run: true).recover(Post.where("baked_at >= ?", SiteSetting.purge_deleted_uploads_grace_period_days.days
.ago))
53

I get this…

ActiveRecord::StatementInvalid: PG::UndefinedFunction: ERROR:  operator does not exist: timestamp without time zone >= integer
LINE 1: ...HERE ("posts"."deleted_at" IS NULL) AND (baked_at >= 30) AND...
54

Oops I’ve updated the steps again.

O wait you are supposed to run it without dry run to fix the posts.

55

I see 54 images listed – same number as I saw when running DRY_RUN=1 rake uploads:recover

56

I did. It did not fix the problem.

57

If that is the case you all have to restore it from a backup. The images have been removed from the tombstone.

58

Ok, will do. Is the root cause addressed or is there reason to think this will continue to happen?

59

We’ve pushed out the fix and improved our test cases. Currently we’re not expecting images to go missing.

7 Likes