I’ve got a big import that’s got a bunch of tombstoned images. This happens from time to time and I’m not sure why, but that’s not why I’m here. I’m running rake uploads:recover_from_tombstone and it’s taking well over a minute per upload (I think that’s what those 45471 things that it needs to process are). So, I’m looking at a month for it to finish. :-1: And the 25 that it processed appeared not to have done anything (the most recent one did reported that it was restoring an image to its proper place). It’s a standard install on an 8GB Digital Ocean “optimized” droplet. I’ve set db_shared_buffers: "2048MB" db_work_mem: "80MB" UNICORN_WORKERS: 4 At one point I tried increasing num_connections up to 200 (a new trick I learned recently), but that didn’t seem to help either. total used free shared buff/cache available Mem: 7.8G 1.7G 133M 1.0G 6.0G 4.8G Swap: 2.0G 691M 1.3G I’m pretty stumped. EDIT: I think it’s just that the database is not on SSD, but whatever DO calls their extra space and it’s a huge community with a couple million posts.

Thanks, @zogstrip! My quick solution is to rsync the images back to default out of tombstone and then rebake (I think they may need to rebake anyway). This will un-break the images in all posts immediately and the rebake will see that they don’t get tombstoned again, right?

Not sure actually. This will temporarily show the images but the rebake process won’t create the upload records, so they’ll be back in tombstone when the cleanup job runs.

OK, so rsync them over so that the images aren’t broken, and THEN run the rake task to update the database. It’s running much faster on the production server, but it still looks like 20 days to complete 430K (posts?). The site is mostly about images, so broken images is something of a problem. :slight_smile: This happens to me on imports a lot. I’m thinking that there must be something to do to update the database that’s not happening in the importer. I should look at what that rake task is doing…

@zogstrip how safe would the process be that Jay proposes here? Discourse is still moving images from the import into the tombstone, I seem to be at 440k now and I’m concerned that processing will take longer than 30 days - meaning images could get deleted permanently. Also, any speedups in the task would be fantastic!

I have a related question: A little over 1600 entries have been processed now, but not a single file has been moved back from the tombstone into the uploads directory (I verified with a filecount). Does this mean it’s not working? Edit: apparently almost all of them are in the optimized directory, not in original. Digging a little deeper.

The script prints the names of the files or moves. Not all of the posts (?) it’s processing need fixing.

We’ve made some improvements to the uploads:recover_from_tombstone rake task. It even has its own tested class now :wink: thanks to @tgxworld and @vinothkannans lib/upload_recovery.rb 7eea55d56 # frozen_string_literal: true class UploadRecovery def initialize(dry_run: false) @dry_run = dry_run end def recover(posts = Post) posts.have_uploads.find_each do |post| begin analyzer = PostAnalyzer.new(post.raw, post.topic_id) analyzer.cooked_stripped.css("img", "a").each do |media| if media.name == "img" && orig_src = media["data-orig-src"] if dom_class = media["class"] if (Post.white_listed_image_classes & dom_class.split).count > 0 next end end This file has been truncated. show original

Recover_from_tombstone insanely slow

Product Support

zogstrip (Régis HANOL) April 6, 2018, 9:29pm 3

Possible, that job is pretty “dumb”. I’ll have a look if I can find some quick wins.

Topic		Replies	Views
Old image uploads become broken images Product Support	63	9662	May 24, 2021
Uploads:recover_from_tombstone not resurrecting images Bug	4	2052	January 19, 2018
Tombstone directory not getting cleaned up for images Product Support	16	1811	August 22, 2020
What happens if I manually move images out of the tombstone directory? Product Support	8	1499	September 12, 2018
Images tombstoned after import Product Support	2	908	November 28, 2017

Recover_from_tombstone insanely slow

Related topics