Old image uploads become broken images

[karussell (Peter)]

Did this already, see my comment

[peternlewis]

I have had a few reports of lost uploaded images (images show as missing in posts that have not been edited for ages). Usually by the time they are reported, the user has fixed the issue, but in this case I got a report of an issue and managed to look more deeply.

<img src="/uploads/default/original/2X/d/d55cbd4c2d65cac5415340df0a4567ca805efa2d.png" width="575" height="954">

<img src="/uploads/default/original/2X/5/5e93bafe11e13c7af89d91a976e01a919538c486.png" width="498" height="304">

(from this post).

The post was written in Aug 2016, and updated in Feb 2017, and then not touched since then.

I checked in to this and I can see that the files have been removed from the uploads directory sometime in January 2018 (they are in my backups from the end of December 2017, but not in my backups from the end of January).

So somewhere along the lines something went wrong and these files were removed even though they were referenced.

I have checked on the server and the files do not appear to exist in any directory anywhere. What does recover_from_tombstone do and is it likely to help?

Alternatively, how can I restore the individual files from the backups?

[pfaffman (Jay Pfaffman)]

I’ve got an import running now that may have the same problem. I’d try the recover from tombstone task. It could be that there is a bug that tombstones images when they shouldn’t be…

Do you see the missing file in tombstone?

They stay in tombstone 30 days before getting deleted. So if they were moved in January, they were deleted in February.

[peternlewis]

They were definitely gone before Jan 31, so presumably they wont be in tombstone.

@pfaffman

Do you see the missing file in tombstone?

Where do I look for that? Though, as I say, they are presumably long gone from that.

So it seems there is some sort of bug in that code that is deleting images that are referenced.

I still would like to know an easy way to restore them if I have the file, since I always have the file in a backup from some time in the past.

[pfaffman (Jay Pfaffman)]

@peternlewis

They were definitely gone before Jan 31, so presumably they wont be in tombstone.

Right. But I thought you said that you had old backups. They’ll be in that tar file. Or wherever you have the backups you mention.

You can just put them back where they belong. They may get deleted again, but then it’ll be in the tombstone and the recover_from_tombstone will work.

[peternlewis]

@pfaffman

Right. But I thought you said that you had old backups. They’ll be in that tar file. Or wherever you have the backups you mention.

The backups (from Discourse backups) do not appear to include the uploads/tombstone directory.

I have restored the files on the site to their places in the uploads folder, and they have reappeared in the topic post. I will try to monitor the topic and see if they are deleted again.

[pfaffman (Jay Pfaffman)]

Right. You’ll need to go back to backups from before they were tombstoned.

[JMichaelTX]

@pfaffman

It could be that there is a bug that tombstones images when they shouldn’t be…

Is anyone investigating a bug?

It is very time consuming for the site admin to investigate, find, and restore broken images. And the worst part is that images can become broken without anyone being notified.

Any help to find/fix this bug would be greatly appreciated.

[mustachianpost]

Hi there,

I tried to run your commands to fix broken image on my forum: Investing / Brokers FAQ - Wiki - Mustachian Post Community

Before it was showing a broken link. But not it seems to have replaced it by a “transparent” image.
Any clue why/how to fix the issue definitely?

Many thanks for your help!
MP

[codinghorror (Jeff Atwood)]

I believe @tgxworld added a bunch of extra tooling here so maybe there are new things to try?

[qtpngo (Quang Ngo)]

I still got this issue with the newest update from Discourse: 2.2.0.beta2 +45.
Followed the instruction in this reply I solved the issue.

@pfaffman

Though you will probably never need it, the fix went something like this:

# copy the "deleted" images from their to-be-deleted staging area
cd /var/discourse
./launcher enter app
rake uploads:recover_from_tombstone

# rebake to see that it doesn't happen again
rake posts:rebake

Is there any update on this?

[mustachianpost]

@tgxworld @pfaffman so I tried running this:

copy the “deleted” images from their to-be-deleted staging area

cd /var/discourse
./launcher enter app
rake uploads:recover_from_tombstone

rebake to see that it doesn’t happen again

rake posts:rebake

Then, now, all missing images are still transparent.png, and have a data-orig-src with what seems to be the uploaded file…

Anyhow, I still have 748 images that seems problematic…

Do you have any new tooling to definitely fix this? That would help a lot!
Even if it’s manually copy-pasting from tombstone to the uploads folder (I’d just need to know the paths then I could try).

[tgxworld (Alan Tan)]

The instructions in this post are stale. Please follow the steps in

https://meta.discourse.org/t/my-forum-old-images-not-see/97123/22?u=tgxworld

instead.

[mustachianpost]

@tgxworld

./launcher enter app rails c require_dependency “upload_recovery” UploadRecovery.new.recover

Thanks @tgxworld
Just did run it and so far it outputs:

rails.c
require_dependency "upload_recovery"
UploadRecovery.new.recover
[1] pry(main)> require_dependency "upload_recovery"
=> nil
[2] pry(main)> UploadRecovery.new.recover
=> nil
[3] pry(main)>

Should I type exit to end the thing and see if it works? Or should I just wait because many images? (sorry, newbie in Ruby )

[pfaffman (Jay Pfaffman)]

Have you upgraded recently? I’m guessing not.

./launcher rebuild app

[mustachianpost]

@mustachianpost

./launcher enter app

@pfaffman yep I did so last weekend, but re-did it again now.
Still the same output:

rails c
[1] pry(main)> require_dependency "upload_recovery"
=> nil
[2] pry(main)> UploadRecovery.new.recover
=> nil
[3] pry(main)> 

Any clue what else I could do to fix those missing images?

[tgxworld (Alan Tan)]

You can run UploadRecovery.new(dry_run: true).recover to see what images are still missing. For those that are, you’ll have to recover the images from a recent backup.

[mustachianpost]

@tgxworld

UploadRecovery.new(dry_run: true).recover

Thanks for the tip!
Interesting because now it lists only 3 missing images, but that I can’t seem to find in Discourse backups…
Will check if I have more chances with DigitalOcean ones.

[mustachianpost]

OK so in the end everything is fine!
FTR what I did:

• Taking a discourse backup (not the one from Digital Ocean, but the one done by Discourse itself)
• At first I looked at the filenames returned by “UploadRecovery.new(dry_run: true).recover” which I couldn’t find in the “original” upload folder nor in the “optimized” one
• So what I did was to go look for the original filename within the forum where the image was missing, and here I got the “original” filename (as the Ruby request returns the optimized filename it seems)
• Then I went into my backup file (a tar.gz that I downloaded on my laptop) and searched for the (original) filename in the folder “/uploads/default/original/1X”.

HTH.

And thanks again for your support @tgxworld and @pfaffman !

[Brandon_Martus]

I’ve had a similar issue happen after importing my 2M posts from vBulletin.

Currently running through rake uploads:recover_from_tombstone over 46,000 files.

This is taking a long time – 2.6% in the past 8 hours. Is there something I should do to prevent the tombstone files from being removed while this is running?

Is this setting below applicable or is it just for s3?

Thanks!