Broken images move to tombstone

support
#1

Hi,

My discourse forum lost 10% images, just show …

image

  1. broken images on /var/www/discourse/public/uploads/tombstone and I have copy tombstone to /var/www/discourse/public/uploads/default.

  2. I have use rake uploads:recover_from_tombstone and rake posts:missing_uploads Does not work.

  3. How to find any missing images? not work.

Have any help?

#2

Any advice @vinothkannans?

#3

Are you sure about the existence of missing uploads in tombstone?

#4

I am sure the image is in the tombstone directory.

Eg:

https://www.example.net/uploads/default/original/4X/1/3/f/13f5a8a1b636146333dacccecf1a98dda9facc71.jpeg

404 Error.

https://cdn.example.net/uploads/default/original/4X/3/4/6/3467394a5f188e390e108a0275335d08bd949d2f.png

200 //have CDN cache Hit.

https://www.example.net/uploads/tombstone/default/original/4X/1/3/f/13f5a8a1b636146333dacccecf1a98dda9facc71.jpeg

200 // in tombstone directory.

#5

rake uploads:recover_from_tombstone method should work here. May I know the post url which have missing uploads?

3 Likes
#6

I pushed the below commit to resolve this problem.

https://github.com/discourse/discourse/commit/0e677daaee3422ac056c3dac1b5bd23a53c4efc2

3 Likes
#7

As I am hunting issue with disappearing thumbnails of Topic List Previews plugin, I have tried if you fix helps for my issue, but it does not.

Why are images moved in tombstone at the first place? Any idea why external images (onebox, youtube preview…) thumbnails would be moved to tombstone?

1 Like
#13

I have use rake posts:rebake

now, all images showing blank, only load transparent.png

I noticed that the all broken images post id > 250,000 (/p/250000)

# rake uploads:recover
W, [2019-05-20T12:31:35.997317 #15886]  WARN -- : Bad date/time value "2019:05:04 24:10:25": min out of range
W, [2019-05-20T12:31:36.876860 #15886]  WARN -- : Bad date/time value "2019:05:04 24:10:25": min out of range

W, [2019-05-20T17:01:52.418659 #15886]  WARN -- : Bad date/time value "2019:05:05 24:08:20": min out of range
W, [2019-05-20T17:01:53.266267 #15886]  WARN -- : Bad date/time value "2019:05:05 24:08:20": min out of range
W, [2019-05-20T17:01:54.631988 #15886]  WARN -- : Bad date/time value "2019:05:05 24:08:48": min out of range
W, [2019-05-20T17:01:55.350312 #15886]  WARN -- : Bad date/time value "2019:05:05 24:08:48": min out of range
W, [2019-05-20T17:01:56.605504 #15886]  WARN -- : Bad date/time value "2019:05:05 24:09:18": min out of range
W, [2019-05-20T17:01:57.055881 #15886]  WARN -- : Bad date/time value "2019:05:05 24:09:18": min out of range
W, [2019-05-20T17:01:58.983880 #15886]  WARN -- : Bad date/time value "2019:05:05 24:09:43": min out of range
W, [2019-05-20T17:01:59.740514 #15886]  WARN -- : Bad date/time value "2019:05:05 24:09:43": min out of range
W, [2019-05-20T17:02:43.100099 #15886]  WARN -- : Bad date/time value "2019:05:05 24:10:11": min out of range
W, [2019-05-20T17:02:43.741339 #15886]  WARN -- : Bad date/time value "2019:05:05 24:10:11": min out of range
W, [2019-05-20T17:02:45.227477 #15886]  WARN -- : Bad date/time value "2019:05:05 24:10:33": min out of range
W, [2019-05-20T17:02:45.813161 #15886]  WARN -- : Bad date/time value "2019:05:05 24:10:33": min out of range
W, [2019-05-20T17:02:47.525792 #15886]  WARN -- : Bad date/time value "2019:05:05 24:11:06": min out of range
W, [2019-05-20T17:02:48.099365 #15886]  WARN -- : Bad date/time value "2019:05:05 24:11:06": min out of range
W, [2019-05-20T17:02:49.861304 #15886]  WARN -- : Bad date/time value "2019:05:05 24:11:32": min out of range
W, [2019-05-20T17:02:50.574582 #15886]  WARN -- : Bad date/time value "2019:05:05 24:11:32": min out of range
W, [2019-05-20T17:02:51.938176 #15886]  WARN -- : Bad date/time value "2019:05:05 24:12:10": min out of range
W, [2019-05-20T17:02:52.450379 #15886]  WARN -- : Bad date/time value "2019:05:05 24:12:10": min out of range
W, [2019-05-20T17:03:28.912249 #15886]  WARN -- : Bad date/time value "2019:05:05 24:12:41": min out of range
W, [2019-05-20T17:03:29.395251 #15886]  WARN -- : Bad date/time value "2019:05:05 24:12:41": min out of range
W, [2019-05-20T17:03:31.213862 #15886]  WARN -- : Bad date/time value "2019:05:05 24:13:10": min out of range
W, [2019-05-20T17:03:31.847879 #15886]  WARN -- : Bad date/time value "2019:05:05 24:13:10": min out of range
W, [2019-05-20T17:03:33.291672 #15886]  WARN -- : Bad date/time value "2019:05:05 24:13:32": min out of range
W, [2019-05-20T17:03:33.888436 #15886]  WARN -- : Bad date/time value "2019:05:05 24:13:32": min out of range
W, [2019-05-20T17:03:35.497848 #15886]  WARN -- : Bad date/time value "2019:05:05 24:13:58": min out of range
W, [2019-05-20T17:03:36.206498 #15886]  WARN -- : Bad date/time value "2019:05:05 24:13:58": min out of range
W, [2019-05-20T17:03:37.458976 #15886]  WARN -- : Bad date/time value "2019:05:05 24:14:28": min out of range
W, [2019-05-20T17:03:37.976304 #15886]  WARN -- : Bad date/time value "2019:05:05 24:14:28": min out of range
W, [2019-05-20T17:04:01.443087 #15886]  WARN -- : Bad date/time value "2019:05:05 24:14:54": min out of range
W, [2019-05-20T17:04:01.908416 #15886]  WARN -- : Bad date/time value "2019:05:05 24:14:54": min out of range
W, [2019-05-20T17:04:03.828456 #15886]  WARN -- : Bad date/time value "2019:05:05 24:15:16": min out of range
W, [2019-05-20T17:04:04.711269 #15886]  WARN -- : Bad date/time value "2019:05:05 24:15:16": min out of range
W, [2019-05-20T17:04:06.086656 #15886]  WARN -- : Bad date/time value "2019:05:05 24:15:52": min out of range
W, [2019-05-20T17:04:06.908895 #15886]  WARN -- : Bad date/time value "2019:05:05 24:15:52": min out of range
W, [2019-05-20T17:04:08.577760 #15886]  WARN -- : Bad date/time value "2019:05:05 24:16:21": min out of range
W, [2019-05-20T17:04:09.094158 #15886]  WARN -- : Bad date/time value "2019:05:05 24:16:21": min out of range
W, [2019-05-20T17:04:10.810286 #15886]  WARN -- : Bad date/time value "2019:05:05 24:16:55": min out of range
W, [2019-05-20T17:04:11.256830 #15886]  WARN -- : Bad date/time value "2019:05:05 24:16:55": min out of range
W, [2019-05-20T17:04:44.181617 #15886]  WARN -- : Bad date/time value "2019:05:05 24:17:23": min out of range
W, [2019-05-20T17:04:44.671349 #15886]  WARN -- : Bad date/time value "2019:05:05 24:17:23": min out of range
W, [2019-05-20T17:04:46.344635 #15886]  WARN -- : Bad date/time value "2019:05:05 24:17:49": min out of range
W, [2019-05-20T17:04:46.878684 #15886]  WARN -- : Bad date/time value "2019:05:05 24:17:49": min out of range
W, [2019-05-20T17:04:48.606346 #15886]  WARN -- : Bad date/time value "2019:05:05 24:18:13": min out of range
W, [2019-05-20T17:04:49.435557 #15886]  WARN -- : Bad date/time value "2019:05:05 24:18:13": min out of range
W, [2019-05-20T17:04:50.683006 #15886]  WARN -- : Bad date/time value "2019:05:05 24:18:56": min out of range
W, [2019-05-20T17:04:51.231679 #15886]  WARN -- : Bad date/time value "2019:05:05 24:18:56": min out of range
W, [2019-05-20T17:04:52.954892 #15886]  WARN -- : Bad date/time value "2019:05:05 24:19:23": min out of range
W, [2019-05-20T17:04:53.696249 #15886]  WARN -- : Bad date/time value "2019:05:05 24:19:23": min out of range
W, [2019-05-20T17:05:05.890965 #15886]  WARN -- : Bad date/time value "2019:05:05 24:19:47": min out of range
W, [2019-05-20T17:05:06.473206 #15886]  WARN -- : Bad date/time value "2019:05:05 24:19:47": min out of range
#14

Looks like topic-previews plugin adds PostUpload record in update_post_image method. But after that link_post_uploads method removes all existing post_uploads before re-generating it. This change in the core triggered this issue. I think the plugin code must handle this to prevent image losses.

@appcomqa do you have any custom plugins installed? And are you in the latest version of Discourse?

5 Likes
Topic List Previews (legacy)
#15

@vinothkannans thanks, will get attention of TLP developers if they can address this properly in plugin.
Cc: @merefield, @angus

2 Likes
#16

Got my attention. Thanks a lot. I’ll try to get a fix out.

2 Likes
#17

no have plugin, and latest version tests-passed

#18

The TLP issue fix is now a plugin PR. It needs a little more testing and review before merging.

My strategy was to create a new custom field on the Post object to temporarily store the id of the thumbnail upload in update_post_image (instead of committing it to the PostUploads table straightaway), then it is added in to the collection of upload id’s in an overridden version of link_post_uploads and committed to the database there as is currently done.

This strategy retains the transactional integrity of the core code.

There is a much lower risk now the thumbnails will be deleted.

https://github.com/angusmcleod/discourse-topic-previews/pull/90