Setting up backup and image uploads to DigitalOcean Spaces

Those steps are indeed correct, I did a few dry runs for these migrations today and even through they run successfully, the images are not rendered in the posts, even after a rebake. I’d recommend holding off on the migration till this is resolved @Woodcock

3 Likes

Whenever I run rake uploads:migrate_from_s3, the urls are correct and images exist at those urls.
The post does not load the images but the composer preview loads the images successfully :thinking:
This is seen before and after rebaking all posts.

Here’s an example url after migrating from s3:
local: ![TaRUkxo|353x500](upload://68CXVBiDpYbpJBTIVN9Gwf8buSE.png)
external: ![TaRUkxo|353x500](/uploads/default/original/1X/2b06bcd73ce121a870637a50bd408e20533808fc.png)

Any ideas why the images would load only in the composer preview? @gerhard or @zogstrip? Do the above urls look right?

2 Likes

I would use:

  1. Put the site in read-only mode
  2. Run the above tutorial
  3. Update site settings to Digital Ocean Spaces
  4. Disable read-only
  5. Remap URLs from S3 to DO Spaces

This will be way faster than the s3 -> local and then local -> s3.

7 Likes

My question is how do you do this step?

Edit: I dug around and found the remap command is probably what I need…

So in my case, the remap should look like this if I’m not mistaken.

rake posts:remap["npn-ndfapda.netdna-ssl.com","npn.sfo2.cdn.digitaloceanspaces.com"]

and then I’m thinking a rebake would be in order after this, or is this unnecessary?

3 Likes

FYI remap does not work because it only affects data in the post which does not contain the url. So, back to my question of how do you remap urls from s3 to spaces? Thanks guys!

1 Like

remap should work, it performs a deep and complete substitution on every single column in the database. As long as you know what from → to is you will be fine. The rebake at the end helps cause then it re-CDNifys your cooked markdown so you can allow for a CDN change as well if needed.

3 Likes

I think that’s the problem though, I need to change the cooked url. I tried to change a single image using this command:

rake posts:remap["https://nature-photographers-network.s3.dualstack.us-west-1.amazonaws.com/optimized/2X/5/5c99d451a64f9f0073a7d0b0d91426adff8becb1_1_666x500.jpeg","https://npn.sfo2.cdn.digitaloceanspaces.com/optimized/2X/5/5c99d451a64f9f0073a7d0b0d91426adff8becb1_1_666x500.jpeg"]

and got: 0 posts remapped!

It seems the cooked url is not stored in the database, am I correct on that assumption?

Not at all, it is definitely stored in the DB… I would try a discourse db:remap which is the more atomic version.

2 Likes

rake posts:remap works on raw, not cooked. I think a rake posts:rebake would have worked, but using discourse db:remap is probably the better idea.

5 Likes

Forgive my ignorance, I tried this and it says Could not find command "db:remap[https://n.....

Is this the correct way to run it?

cd /var/discourse 
./launcher enter app
discourse db:remap["find","replace"]
1 Like

You can see an example of db:remap here

3 Likes

Thanks @sam, I finally got around to this and I’m still having trouble (of course). I ran these two commands as a test:

discourse remap 'https://npn-ndfapda.netdna-ssl.com/optimized/2X/1/1b7d579a96ff36bf113430403a56d8d013a80aec_1_666x500.jpeg' 'https://npn.sfo2.cdn.digitaloceanspaces.com/optimized/2X/1/1b7d579a96ff36bf113430403a56d8d013a80aec_1_666x500.jpeg'

discourse remap 'https://npn-ndfapda.netdna-ssl.com/original/2X/1/1b7d579a96ff36bf113430403a56d8d013a80aec_1_666x500.jpeg' 'https://npn.sfo2.cdn.digitaloceanspaces.com/original/2X/1/1b7d579a96ff36bf113430403a56d8d013a80aec_1_666x500.jpeg'

For each command it did report success:

Remapping posts cooked
1 rows affected!

But, as you can see in this post, the first image is not displaying, which is the url I tried to remap. It’s still the same url despite the remap.

https://community.naturephotographers.network/t/snow-in-the-san-juans/1164

In another post I tried to rebuild the html after doing this and it changes the first part of the url to https://nature-photographers-network.s3.dualstack.us-west-1.amazonaws.com which is my old s3 bucket which is not even in my settings anymore…

I just set up DigitalOcean Spaces on my Discourse following the above instruction. I was able to upload images to Spaces but not to back up. From the backup logs it seems like the backup location has not been changed:

[2018-12-01 06:46:28] EXCEPTION: /var/www/discourse/lib/backup_restore/backuper.rb:250:in `create_archive': Failed to gzip archive.

gzip: /var/www/discourse/public/backups/default/leasehackr-forum-2018-12-01-063452-v20181129094518.tar.gz: No space left on device

My backup setup is the following:

What am I missing? Thank you!

It’s right there in the error message. Your local disk is full.

5 Likes

Yes my local disk is full, that’s why I wanted to move to DO Spaces, which I just created. Is it possible?

On this issue, I think that the use of environment variables of kind DISCOURSE_S3_* in order to make the rake uploads:migrate_to_s3 task work, is bringing some unwanted consequences.
They seem to take precedence and duplicating the exact same settings one can enter in the website settings page, but also changing the link of any other asset (including JS stuff) to the provided CDN URL.
All this just to be able to move uplink files from a local server to S3/DigitalOcean Spaces? Seems a bit of an overkill, or a bug. :confused:
Se the ongoing discussion here.
https://meta.discourse.org/t/migration-of-system-stored-images-to-s3-after-configuration-change/19605/52?u=marcozambi

Are the images uploaded to the digital ocean spaces only? and is it also possible to upload video’s (mp4) files to the spaces?

Maybe a stupid question :slight_smile:

Does this support the new CDN feature of spaces as well?

Has anyone figured out how to successfully rake uploads:migrate_to_s3 from local storage to DO Spaces? I messed around with it for a while, with all the hacks listed in various places, but had no luck. It’s still asking for all of the env vars, which is fine, but it seems to all presume AWS based on the error messages.

2 Likes

Can i edit this file, then make rebuild app without any fear?