Before I spend too much time writing a migrate_to_s3 rake task, Eric, did you already end up doing this?
We no longer recommend storing images in s3 as it substantially complicates your Discourse install. We do still recommend it for backups.
Ah, interesting. Thanks for letting me know.
Is it possible to execute this in the standard docker installation? I originally installed using s3 and now would like to revert to local. No ruby/rails experience though.
After 1.2 is released @zogstrip can you make sure this task is working as we need a migration path if we are deprecating this.
In case anyone is wondering I did end up writing the
migrate_to_s3 rake task since we migrated meta to S3
Here’s how to do it
Make sure you’ve properly set up S3
SSH into your server and
# ./var/discourse/launcher enter app # rake uploads:migrate_to_s3
And patiently wait for the rake task to finish
Add option to disable backup compression
Thanks for the update.
- Does this remove the files local server?
- I take it the URL’s in the posts are updated via
DbHelper.remap(from, to)so no need to rebuild all the posts?
I want these tasks to be kinda safe, so no, it doesn’t remove files stored locally.
You are right about the remap, it removes the need to rebuild all the posts
@zogstrip After migrating attachments to S3, you dont need to include them in the backups, right? Amazon S3 SLA is pretty high, and my attachments are not 100% mission critical, so to me it sounds sensible to:
- Migrate attachments to S3
- Exclude attachments from backups, thus reducing backup size by 50%.
Well, they aren’t Only local files are backed-up.
I did the migration but discovered some potential issues with it.
I noticed that the subfolder discourse/shared/standalone/uploads/default/optimized was not migrated to S3 bucket. Only the folder original was transfered.
I also noticed that expanding of some large images now produces an error: The image could not be loaded. The reason for this is that the link format is broken:
New large uploads seem to work, so this is an issue with the migration script.
Generating missing optimized images with S3
Setting up file and image uploads to S3
Hm…also some broken images, like this one.
@zogstrip, I would appreciate some best practice suggestions to fix this without risk of data corruption.
Well, it seems that a rebake fixed all the broken images and links. So it is definitely still needed after the S3 migration.
Looks like the
DbHelper.remap call needs a little more info or a 2nd call.
from variable is equal to something like
And the post values contain something like this:
raw: "Example\n<img src='/uploads/default/37/634ed4531d491595.jpg'>\nSome other string", cooked: "<p>Example<br>\n\n<p><img src=\"//example.com/uploads/default/37/634ed4531d491595.jpg\" width=\"100\" height=\"100\"><br>Some other string</p>",
raw value is getting correctly replaced but the
cooked value is not.
To me this would suggest that there are two calls to
- Firstly a new call - where the
fromvalue is in the form
//<domain><optional slash + subfolder><upload.url>.
- Then the current call.
You might need to URL / HTML encode special characters for the “subfolder” part where someone has chosen some crazy subfolder install path, perhaps CJKV characters etc.
Completing in this order with the longer more specific target first to avoid an incorrect partial replacement.
You could also rebake all posts, as you’re changing
Yes, as @ljpp mentioned above a rebake fixed broken images and links.
However my response was focused on solving this for future executions.
When you have over 450,000 posts a rebake is a computationally expensive and lengthy process compared to getting the remap right.
I am still looking for the solution how to migrate the optimized folder:
The migration script succesfully moves the original folder and after a rebake everything is OK, but I have gigabytes of payload in the daily backups due to the optimized not being migrated. Everyone going for S3 will have this issue, and the bigger the forum the bigger the payload.
I am going to do a dirty up! here, as I beleive this issue is significant for everyone who have migrated to S3 after day 1.
Up! Would really love to eliminate those graphics files from my backup.
How do I clear up server space by deleting the old files?