Re-adding missing uploads to the database

I’ve got a site where it seems that a bunch of uploads have been removed from the uploads table in the database, but still exist in the filesystem. This leaves links to these broken. I’ve fixed a few by re-uploading files into a new topic to create the record in the DB (at the time I thought that the files were missing too, but @angus graciously pointed out why I wasn’t finding them–One day I’ll ask: why do we have both sha1 and base62 names for all of these assets?) and then re-baking the posts that include those uploads, like this:

def rebake_posts_with_uploads(topic_id, post_number)
  p = Post.find_by(topic_id: topic_id, post_number: post_number)
  exit unless p
  re = /upload:\/\/(.+?)\)/
  shas= p.raw.scan(re)

  shas.each do |sha|
    posts = Post.where("raw like '%#{sha[0]}%'")
    next unless posts
    puts "Found #{posts.count - 1} #{sha[0]}"
    posts.each do |post|
      next if post.id == p.id
      puts "rebake #{post.id}--#{BASE_URL}/t/-/#{post.topic_id}/#{post.post_number}"
      post.rebake!
    end
  end
end

My new plan, since it seems that the files exist in the file system but not in the database, is to do something like this:

for (all files in /shared/uploads/default/original/1x) do |file|
  unless file is in uploads table
     create upload record 
     for each post that includes that upload record
       rebake

Does that seem right? I’m looking at uploads.rake and don’t see anything that seems to do this already. This is sort of the opposite of

but instead of FileUtils.rm(file_path) I’d instead do an Upload.create, I think.

If this seems really stupid or there is a much better solution, I’d love to hear it before I go down this little rabbit hole.

Thanks.

I don’t know how this happened. I was hoping to pin the blame on a custom plugin, but I’m afraid that’s not the case. It may be related to another discussion, in which someone said:

2 Likes

Yeah our history here with uploads is quite spotty, and it is our fault… @sam can you recommend someone to give a quick bit of advice?

2 Likes

This can work, you just have to be careful to test on local … also look at 2x / 3x directories there are uploads everywhere.

3 Likes

I’m trying something like this:

def add_missing_files_to_uploads
  public_directory = Rails.root.join("public").to_s
  db = RailsMultisite::ConnectionManagement.current_db
  uploads_directory = File.join(public_directory, 'uploads', db).to_s
  # uploads and optimized images
  missing = 0
  matched = 0
  Dir.glob("#{uploads_directory}/**/*.pdf").each do |file_path|
    sha1 = Upload.generate_digest(file_path)
    url = file_path.split(public_directory, 2)[1]
    if (Upload.where(sha1: sha1).empty? &&
        Upload.where(url: url).empty?)
      puts "MISSING #{file_path}" if DEBUG
      missing += 1
    else
      matched += 1
    end
  end
  puts "MISS: #{missing}. Match #{matched}"
end

Does that look sort of close? My first test seems to have failed, but I might have screwed something up.