בעיות בהעלאות לאחר שינוי מרכז הנתונים של DO Spaces

תמיכה
8

So, just to give this thread a closure that might be useful to somebody else, this is how we solved the situation.

Essentially, since it was impossible to recover the missing attachments through the various uploads recovery rake tasks, I have put together a Ruby script (apologies in advance, I am definitively NOT a Ryby or Rails developer, so I bet the code is inefficient and ugly, but that’s beside the point :stuck_out_tongue: ) that:

  1. Finds all posts containing the string upload://
  2. Extracts the shortlink of each upload and transforms it in its long form sha1 hash
  3. Queries the Uploads table
  4. If an attachment having the sha1 hash is found in Uploads, that upload is skipped, else the URL of that upload is checked in the old Digital Ocean bucket/spaces.
  5. If the uplink is found in the old bucket/spaces then the shortlink is replaced with the URL to the same upload in the old bucket.
  6. If modified, trigger a rebake of the original post, to let Discourse do the heavy lifting of re-downloading locally the “lost” upload and re-create all it needs in the DB.

To avoid blacklisting and to reduce the load on the server, an interval of 20 seconds is introduced every time a rebake is requested.


def remoteFileExist(url, retries=3)
    puts "Requesting #{url} ..."
    uri = URI(url)
    response = nil
    res = Net::HTTP.get_response(uri)
    puts res['content-type']
    if res.code[0,1] == "2" and res['content-type'].include? 'image'
        return true
    else
        return false
    end
    rescue Net::ReadTimeout => e
        puts "TRY #{retries}/n ERROR: timed out while trying to connect #{e}"
        if retries <= 1
            raise
        end
        remoteFileExist(url, retries - 1)
    end
end


####################################################################


posts=Post.where("raw like '%upload://%' " ).order('topic_id ASC, post_number DESC');
idx = 0;
posts.each do |p|
    idx = idx + 1;
    puts ""

    matches = p.raw.scan(/(!\[(.)*\]\(upload:\/\/([a-zA-Z0-9]+)\.(jpeg|jpg|png|gif|pdf|mp3|mp4|mov)\))/)

    new_raw = p.raw

    matches.each do |m|  
        short_url = m[0];
        short_sha = m[2];
        ext = m[3];
        long_sha = Base62.decode(short_sha).to_s(16).rjust(40,"0")

        upload = Upload.where('sha1 = ?', long_sha)

        puts "#{short_url} -> #{long_sha}\n"

        if upload.all.count == 0
            puts "#{long_sha} not found in DB. Recovering from ams3...\n"

            subdir1 = long_sha[0]
            subdir2 = long_sha[1]

            new_url1 = "https://discourse-data.ams3.digitaloceanspaces.com/original/3X/#{subdir1}/#{subdir2}/#{long_sha}.#{ext}"
            test1 = remoteFileExist(new_url1)
            if test1
                new_raw = new_raw.gsub(short_url, "\n#{new_url1}")
            else
                new_url2 = "https://discourse-data.ams3.digitaloceanspaces.com/original/2X/#{subdir1}/#{long_sha}.#{ext}"
                if remoteFileExist(new_url2)
                    new_raw = new_raw.gsub(short_url, "\n#{new_url2}")
                end
            end
            puts ""
            sleep 5
        end
    end

    if p.raw != new_raw
        puts "OLD\n"
        puts p.raw
        puts "-----------"
        puts "NEW\n"
        puts new_raw
        puts "-----------"
        puts "UPDATING!"
        # goahead = gets
        p.raw = new_raw
        p.cooked = ''
        p.save
        p.rebake!(invalidate_broken_images: true);
        puts "*******************************************"
        sleep 30
    else
        puts "SKIP!"
        puts "*******************************************"
        sleep 1
    end
end