Downloads coming from S3 even with DISCOURSE_S3_CDN_URL set

Support
1

I followed the guide at Using Object Storage for Uploads (S3 & Clones) to set up Backblaze B2 for upload storage and I’m seeing a strange problem.

If I create a thread and upload an image the img src is set to my S3_CDN_URL. If I upload a zip file the URL is at /uploads and redirect directly to my bucket instead of using the S3_CDN_URL. I do not have DISCOURSE_CDN_URL set - the guide indicate it doesn’t affect this issue.

Me relevant settings are:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: "us-west-000"
  DISCOURSE_S3_INSTALL_CORS_RULE: false
  DISCOURSE_S3_CONFIGURE_TOMBSTONE_POLICY: false
  DISCOURSE_S3_ENDPOINT: "https://s3.us-west-000.backblazeb2.com"
  DISCOURSE_S3_ACCESS_KEY_ID: "ID"
  DISCOURSE_S3_SECRET_ACCESS_KEY: "KEY"
  DISCOURSE_S3_CDN_URL: "https://devforum-b2-cdn.freetls.fastly.net"
  DISCOURSE_S3_BUCKET: "bucket-name"
  DISCOURSE_S3_BACKUP_BUCKET: "backup-bucket/devforum"
  DISCOURSE_BACKUP_LOCATION: s3

Why would zip files be handled differently than images?

1 Like
2

This storage thing and its configuration is indeed a big challenge for the many years.

I was storing my images on Aws S3. And then I changed my storage back to local server. Now there are 100s of images that disappeared suddenly.

Though this is not direct reply or comment on your query, I’m just showing empathy towards the similar issues.

3

See site setting: S3 use CDN URL for all uploads

Use CDN URL for all the files uploaded to s3 instead of only for images.

1 Like
4

Thank you.
I’ve assigned the job to some local friend of mine. He is looking into it.
Though on your asking I selected this option just now. Use CDN URL for all the files uploaded to s3 instead of only for images.

But there seems to be some other problem. He has shifted s3 storage to local server. Now many of my old posts, which if had 10 photos, around half are displaying and other half not. (example of one such topic)

He would work on resolving this, coming weekend.

Earlier, before asking him, I naively selected the option to have all images included in my backups, then restored that backup thinking that it’d restore the images to local server, thinking that this would be enough to move all my images from s3 to local.
But this is not so simple (as I’m finding on many related topics on meta).

5

Sorry. I was talking about the other issue.

Yours, I’m pretty sure, is much more complicated.

That’s complicated indeed.

I think that the missing images on that post are because the Upload record doesn’t exist, so you’ll need to do something to get that image back. I could be wrong.

If I’m right, here’s how I solved what I think was a similar problem before. I’m offering no free support beyond this script.

def process_uploads
  begin
    # Read the list of filenames
    filenames = File.readlines('/shared/uploads/allfiles.txt').map(&:strip)
    count = 0

    filenames.each do |filename|
      # Prepend /shared to the filename
      filename.gsub!(/\.\//,"")
      full_path = File.join('/shared/uploads/default/original/', filename)

      begin
        # Check if path exists and is a regular file (not a directory)
        count += 1
        
        if File.exist?(full_path) && File.file?(full_path)
          # Open the file
          File.open(full_path, 'r') do |tempfile|
            # Create upload using the specified parameters
            u = UploadCreator.new(tempfile, 'imported', {}).create_for(-1)
            puts "#{count} -- #{u.id}: #{u.url}"
          end
        else
          puts "Warning: Path not found or is not a regular file: #{full_path}"
        end
      rescue => e
        puts "Error processing file #{full_path}: #{e.message}"
        # Continue with next file even if current one fails
        next
      end
    end
  rescue Errno::ENOENT
    puts "Error: Could not find files.txt"
  rescue => e
    puts "Error reading files.txt: #{e.message}"
  end
end

# Execute the processing
process_uploads;
1 Like