Images didn't load after configuring S3 uploads

Product Support

AI-generated summary

After migrating from Discourse EC2 to Discourse ECS and enabling S3 Uploads, johndom noticed that images didn’t load. They had previously copied images from EC2 to an S3 bucket with public access. Despite running rake commands, the images still didn’t load due to the URLs pointing to the old EC2 uploads.

johndom shared their app.yml configuration and the error messages they received. pfaffman suggested running the migrate to S3 rake task, which johndom attempted but encountered an error.

pfaffman advised that the task should be run on an instance with access to the images. Since the images were manually moved to S3, modifying the uploads table might be necessary. However, if the same database is being used, pfaffman recommended running the EC2 instance again, adding S3 configuration to ENV variables, and then running the migrate-to-s3 task, as described in the Using Object Storage for Uploads (S3 & Clones) documentation.

1

Hello. We just migrated from Discourse EC2 to Discourse ECS (container service). When we were in EC2, S3 Uploads was not set up yet. Before the migration, we copied all images from EC2 to the S3 bucket (with public access). However, after doing the migration with S3 Uploads enabled (no CDN yet), we noticed that images (and avatars) didn’t load at all.

app.yml has:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: us-east-1
  DISCOURSE_S3_ACCESS_KEY_ID: aws_access_key
  DISCOURSE_S3_SECRET_ACCESS_KEY: aws_secret_key
  DISCOURSE_S3_BUCKET: <uploads-bucket>
  DISCOURSE_S3_BACKUP_BUCKET: <backups-bucket>
  DISCOURSE_BACKUP_LOCATION: s3

hooks:
  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets

Running the following in the container didn’t help as well:

rake posts:rebake
rake posts:missing_uploads
rake avatars:refresh

We also noticed the difference between the URLs of images (Discourse still used the “local uploads in EC2" URLs).

This is the link in S3: https://[s3-bucket].s3.amazonaws.com/original/1X/831092e27d6c066c3618a587a5c378e1c794f2c6.png

Link in Discourse (URL still from EC2 uploads): https://[domain]/uploads/default/original/1X/831092e27d6c066c3618a587a5c378e1c794f2c6.png

Error in the browser is: 404 NOT FOUND

How do we configure Discourse to use the correct S3 URLs for the images?

2

Did you run the migrate to S3 rake task?

I would do that on the ec2 before migrating.

3

Hi Jay. Tried executing the command in ECS but encountered an error:

image
image1920×887 136 KB

Just to ask, should this be ran in EC2 only? Is there another way to migrate uploads or update image URLs? Currently, the EC2 instance is stopped and the current ECS setup uses the domain and database/cache already so I’m not sure if we can point the domain and database/cache back to the old EC2 (then rebuild) without downtime. Thank you.

4

You’l want to run the command in an instance that has access to the images. Oh, but if you moved them to S3 yourself by hand, then you might want to do some surgery on the uploads table instead.

I have been working on a howto with instructions for moving from one bucket to another that you might use to solve your problem, though you’ll need to modify those instructions to account for your local-to-bucket rather than bucket-to-bucket changes.

If I were to bill you a flat rate for the job, I’d probably charge 4 hours, though if you were to pay by the hour it might be only one.

Oh! Sorry. I just noticed this. If the EC2 is using the same database, then you might be able to crank it back up (leaving the ECS in place), and run the migrate-to-s3 task there; you’ll need to first put all the S3 stuff in ENV variables as described in Using Object Storage for Uploads (S3 & Clones). This is likely the easiest solution. I’d recommend doing a database-only backup before starting.

5

Hello Jay. Thanks for the response. Got it. I think that will work.