How to find any missing images?

I know @vinothkannans is working on something that will recover the missing posts to uploads link. He’ll let you know here when it is ready.

4 Likes

Upgrade to the latest version of Discourse and run the task rake posts:missing_uploads again. The task itself will try to recover the missing old scheme uploads from local “uploads” and “tombstone” directories. At the end it will display the count of the uploads which are not found both in database and local storage.

5 Likes

Thanks, seems like a lot of progress

:/var/www/discourse# rake posts:missing_uploads
.............................................................................x.....xx.....................
3 post uploads are missing.

3 uploads are missing.
3 of 2965 posts are affected.

I’d be interested to know which 3, but that’s way better than the 105 which had problems before!

2 Likes

If you run the below command in rails console then you can get the list of missing uploads.

PostCustomField.where(name: Post::MISSING_UPLOADS)

7 Likes

here we go…

[1] pry(main)> PostCustomField.where(name: Post::MISSING_UPLOADS)
=> [#<PostCustomField:0x000055efb0edc790
  id: 337,
  post_id: 2396,
  name: "missing uploads",
  value: "[\"https://SITENAME/uploads/default/original/1X/da27c8c7525666c20b7e04bfd53ccb24647530c6.png\"]",
  created_at: Mon, 29 Apr 2019 20:50:02 UTC +00:00,
  updated_at: Mon, 29 Apr 2019 20:50:02 UTC +00:00>,
 #<PostCustomField:0x000055efb0ee30b8
  id: 338,
  post_id: 2785,
  name: "missing uploads",
  value: "[\"https://SITENAME/uploads/default/original/1X/022705388395c48c415d699d2332bc58d44346c0.png\"]",
  created_at: Mon, 29 Apr 2019 20:50:02 UTC +00:00,
  updated_at: Mon, 29 Apr 2019 20:50:02 UTC +00:00>,
 #<PostCustomField:0x000055efb0ee4238
  id: 339,
  post_id: 2792,
  name: "missing uploads",
  value: "[\"https://SITENAME/uploads/default/original/1X/27bfa704ffb22c4a1afe28c8956dc5ed41731bec.png\"]",
  created_at: Mon, 29 Apr 2019 20:50:02 UTC +00:00,
  updated_at: Mon, 29 Apr 2019 20:50:02 UTC +00:00>]
[3

This is very useful… I can probably work it out from here.

I guess the problem was that the move was never done, and the backup script stopped backing up the old location?

3 Likes

for the record, of the 3 remaining…

1 was really missing
2 had their images… i wonder if what was listed as missing was from previous edits (cropping, etc)

Edited… I had been looking at topics, not posts.

5 Likes

I’m interested to hear the experience of others… I found the image mentioned in the first entry ( da27c8c7525666c20b7e04bfd53ccb24647530c6.png) on my local backup.

It’s an image from another post ID 1604, not 2396. Post 1604 also references that same filename, and is not missing its image.

I don’t have 022705388395c48c415d699d2332bc58d44346c0.png on my local.

I do have the third image 27bfa704ffb22c4a1afe28c8956dc5ed41731bec.png on my local, and while I can’t guess what post it’s from, I don’t think we’d have deleted that post.

@vinothkannans what does post_id: 2396, in the output mean?

The post_id field give you the unique ID of the post. Probably the easiest way to find the post’s topic from the post_id is to enter

Post.find(2396)

in your rails console. That will return the post’s topic_id and post_number. You can use those values to find the post through your site’s UI.

5 Likes

Also, you can navigate to the post using that post_id in below URL

https://discourse.example.com/p/[POST_ID]

7 Likes

Ah, got it… I was confusing post_id with topic_id

If you expect more people will be using rake posts:missing_uploads more… printing the post’s position as site_fqdn/t/topic_id/post_number at the end of the output would help a lot.

PS this edit’s my reply earlier, but still, the remaining damage is quite small

1 Like

Thanks for continuing to work on this, @vinothkannans! :sparkles:

I just tried running the rake posts:missing_uploads task on my site (currently at v2.3.0.beta9 +484), and this is what I got:

# rake posts:missing_uploads
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxx.xxxxxxxx.xxxxxxxxxx..........................x...........................................x.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................x.................................................................................................................................................................................................................
451 post uploads are missing.

255 uploads are missing.
244 of 255 are old scheme uploads.
139 of 8812 posts are affected.

In other words, it looks like I have:

  • 244 missing old scheme uploads
  • 11 missing uploads
  • 196 missing post uploads

Just as an aside…what’s the difference between a missing upload and a missing post upload? (If I’m reading the output correctly, I have 196 more of the latter.)

Anyway, because I appear to have a lot of missing images, I decided to do a bit more spelunking with PostCustomField.where(name: Post::MISSING_UPLOADS). Consider the first result:

#<PostCustomField:0x0000564c2cef9ee8
  id: 161,
  post_id: 43,
  name: "missing uploads",
  value:
   "[\"https://SITENAME/uploads/default/35/4608d96d1b27846f.png\",\"https://SITENAME/uploads/default/35/4608d96d1b27846f.png\"]",
  created_at: Fri, 17 May 2019 05:26:57 UTC +00:00,
  updated_at: Fri, 17 May 2019 05:26:57 UTC +00:00>

I navigated to https://SITENAME/p/43 and, sure enough, upon examining the post’s Markdown source I found a link to an image at /uploads/default/35/4608d96d1b27846f.png. Interestingly enough, though, the image displays fine and even exists at /var/discourse/shared/standalone/uploads/default/35/4608d96d1b27846f.png on the host server.

So…Discourse appears to think that this image is “missing”, but it’s not—it just hasn’t been migrated to the new upload scheme yet. Is this expected behavior and, if so, is there any way to migrate the other (200+) images like this to the new upload scheme?

Thanks!


Update: I think I may have identified the 11 missing uploads mentioned above…and they really do seem to be missing. Here’s what one of them looks like:

#<PostCustomField:0x0000564c2cf0b2d8
  id: 297,
  post_id: 2203,
  name: "missing uploads",
  value:
   "[\"https://SITENAME/uploads/default/original/1X/900eff3dd456ecaf5280568676c4717e27b46c85.jpg\"]",
  created_at: Fri, 17 May 2019 05:26:57 UTC +00:00,
  updated_at: Fri, 17 May 2019 05:26:57 UTC +00:00>

Note that this particular image was stored under /uploads/default/original/1X/ rather than /uploads/default/{SOME_NUMBER}/. It’s from a 2015 post and I can’t find it in any of my backups, so it—and the other 10 images like it—appear to be well and truly gone. :cry:

Still not sure why I have 196 more missing post uploads than uploads, though…

2 Likes

I also have a lot of missing images:

root@xxxxx-app:/var/www/discourse# rake posts:missing_uploads

37 post uploads are missing.

34 uploads are missing.
5 of 34 are old scheme uploads.
33 of 1013268 posts are affected.

but after

root@xxxxx-app:/var/www/discourse# rake uploads:missing
.
.
.

/var/www/discourse/public/uploads/default/original/2X/3/3a9bf205dec2b6bd0b3cc35a3be1f69499960713.JPG
/var/www/discourse/public/uploads/default/original/3X/2/d/2db0ff326859b94824b64c4e0c2b156c562b7a99.jpg
/var/www/discourse/public/uploads/default/original/3X/e/f/ef271ac232c31e747206b47e4de7e0570de9e030.jpg
**10604** of 101083 uploads are missing

/var/www/discourse/public/uploads/default/optimized/2X/3/3d3efaa44fb43b99ec290b75e289080fc448f709_1_657x500.gif
/var/www/discourse/public/uploads/default/optimized/2X/4/4e3b01361d7c30c3df27a9606271175d91edff6d_1_200x200.jpeg
/var/www/discourse/public/uploads/default/optimized/2X/6/68fcd443312e7c500d25a6067c04c98aa1066686_1_200x200.JPG
/var/www/discourse/public/uploads/default/optimized/2X/1/16452ee2749e93e6b47d1388a20e34c1e3832ee1_1_100x100.jpg
/var/www/discourse/public/uploads/default/optimized/2X/0/0e843544885c0ee7a0c350d66bbe852dd4f0a497_1_135x135.jpeg
/var/www/discourse/public/uploads/default/optimized/2X/f/fb3d38e8d1b0e8ae25606156d37b8045f7cbc2b3_1_200x200.jpg
/var/www/discourse/public/uploads/default/optimized/2X/5/54a076f5be7cfabcf0fa34e57b836d85a33a879e_1_200x200.jpg
7 of 247116 optimized_images are missing

I’m having this problem as well on a forum running v2.3.0.beta9. I haven’t restored from backups. It’d be good to know why it’s happening and what we can do to restore the missing uploads. The good news is that the missing uploads seem to be in the tombstone, so hope isn’t lost…

So far, I’ve rebuilt the app and run rake uploads:missing, but that hasn’t helped.

root@forum-app:/var/www/discourse# rake posts:missing_uploads
Looking for missing uploads on: default

146 post uploads are missing.

142 uploads are missing.
81 of 58082 posts are affected.
root@forum-app:/var/www/discourse# rake uploads:missing
Looking for missing uploads on: default

146 post uploads are missing.

142 uploads are missing.
81 of 58082 posts are affected.

The missing uploads are from twoish weeks ago.

Problem solved! rake uploads:recover_from_tombstone did the trick. It’d still be good to understand why this happened…

4 Likes

@vinothkannans, @zogstrip Any updates on this?

Just checking in again. Does anyone know the answers to the following questions from a month ago?

  1. What’s the difference between a missing upload and a missing post upload? (Last I checked, my site appeared to have 196 more of the latter.)

  2. Why does Discourse report images that haven’t been migrated to the new upload scheme as “missing”?

  3. If this is expected behavior, how can I migrate the 200+ images like this on my site to the new upload scheme?

2 Likes

@vinothkannans can advise

  • missing upload - The upload record is found in DB but the file is missing in the file storage.
  • missing post upload - The upload record is not found in DB for an upload URL in the post.

I think those old scheme uploads are not found in the database and/or filesystem too.

You can do it by enabling the setting using the below command. It will migrate those upload if they’re found in DB and filesystem.

SiteSetting.migrate_to_new_scheme = true
4 Likes

Forgive me… where do we load these?