Rake uploads:migrate_from_s3 fails

Tricky question. I did set s3_bucket in config/discourse.conf as mentioned in the post you linked to, which did resolve this particular error, as I noted there.

This file is inside the container (./launcher enter app). Note that for this to survive ./launcher rebuild app you need also to add DISCOURSE_S3_BUCKET to the env section of your containers/app.yml file.

The fact that I fixed it was why it was a dev post, not a support request; I was asking what developers think is the right solution as I continue to hack at this.

I have about 100GB of files in S3 so I’m moving very carefully. I implemented a limit for posts to look at, and I next need to implement a limit for posts to modify. I’ve been trying one thing at a time. The fact that this appears to be rarely-used code and I’ve seen this error repeatedly makes me concerned about code rot and I don’t want to suddenly deface my entire site due to a bug, and this looks like it could be a good way to make that mistake.

  • For upload:// (for me, this means non-video) uploads, so far, it appears to be working. I’m doing one at a time and then reviewing the affected post to make sure everything works.

  • For uploads that don’t use the upload:// syntax (for me, this means video uploads as far as I can tell), where there is a literal reference to the URL in S3, it is mangling the URLs. That’s not a hard bug to fix as soon as I figure out for sure what I’m supposed to change them to but I haven’t done that yet. So that’s likely to be one of the PRs I post soon.

This is a spare time project for me, so no promises on timing.

1 Like