# 迁移到 S3 以解决 Digital Ocean Spaces 的问题

**URL:** <https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662>\
**Category:** Self-hosting\
**Tags:** s3\
**Created:** [2019年二月20日 15:54 UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662 "2019-02-20T15:54:27Z")\
**Posts on this page:** 7\
**Page:** 2

<div class="post-metadata">

**Author:** ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)\
**Post date:** [2019年三月15日 18:03 UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/23 "2019-03-15T18:03:47Z")

</div>

Wait. So now I’ve restored the site to a staging server and ran:

```plaintext
rake uploads:migrate_to_s3                                                              
Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Please run these commands in the rails console

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

```

So does that mean that it’s going to just rebake all the posts and upload them in the background?

OOH! It looks like that’s what’s happening. Maybe I’m finally going to be able to make this transition!

EDIT: This is further confused because I previously had a CDN configured. I guess I need to rebake the posts without the CDN and then do the `migrate_to_s3`?

Also, I think that there’s a bug that if you have a CDN configured and then enable the s3 CDN, Discourse thinks that assets are on the S3 CDN. No. That’s not it. It looks like having `DISCOURSE_S3_CDN_URL` defined makes Discourse point to the CDN URL for assets, but the assets aren’t there. I thought that a `rake assets:precompile` might get it, but no joy. This might need to be another topic

---

<div class="post-metadata">

**Author:** ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)\
**Post date:** [2019年三月15日 22:49 UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/24 "2019-03-15T22:49:12Z")

</div>

Well, no, it seems like nothing is happening. Nothing happens when running `MigrateUploadScheme` and there are no sidekiq jobs queued.

```plaintext
root@shadrack-rbx:/var/www/discourse# rake uploads:migrate_to_s3
Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Please run these commands in the rails console

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
root@shadrack-rbx:/var/www/discourse# rails c
[1] pry(main)> SiteSetting.migrate_to_new_scheme = true
=> true
[2] pry(main)> Jobs::MigrateUploadScheme.new.execute(nil)
=> []

```

---

<div class="post-metadata">

**Author:** ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)\
**Post date:** [2019年三月17日 22:45 UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/25 "2019-03-17T22:45:16Z")

</div>

So, you can hack around this problem by deleting the stuff that’s not in the new scheme, like this:

```plaintext
 bad=Upload.where("url not like '//%' and url not like '/uploads/default/original/_X/%'")
bad.delete_all

```

This seems like a 🐛. – If you rebuild, you’ll need to do it again, so there’s something in `db/fixtures` that needs to be fixed, I guess. Oh, or maybe just change:

> <https://github.com/discourse/discourse/blob/main/lib/tasks/uploads.rake#221>

to

```
  if Upload.where("id>0 and url NOT LIKE '//%' AND url NOT LIKE '/uploads/#{db}/original/_X/%'").exists?

```

I don’t know which is more correct or I might submit a PR.

But when I delete all of those uploads (before I figured out the above), I’m still getting this:

```plaintext
root@shadrack-rbx:/var/www/discourse# DISCOURSE_S3_CDN_URL="https://lc-rbx.ams3.cdn.digitaloceanspaces.com" rake uploads:migrate_to_s3
Migrating uploads to S3 for 'default'...
Please provide the 'DISCOURSE_S3_CDN_URL' environment variable

```

And this is in `discourse.conf` as well:

```
s3_cdn_url = 'https://lc-rbx.ams3.cdn.digitaloceanspaces.com'

```

I guess that’s because of

> <https://github.com/discourse/discourse/blob/main/lib/tasks/uploads.rake#L246>

I changed it to `GlobalSetting.s3_cdn_url.blank?` and it’s running. @zogstrip. It looks like this is yours, but it’s 3 months old, so I guess there’s something else here I don’t understand.

I’ve got 165087, so I guess I’ll see what happens when that finished in a day or three.

---

<div class="post-metadata">

**Author:** ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)\
**Post date:** [2019年三月27日 20:20 UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/26 "2019-03-27T20:20:05Z")

</div>

Thanks for your help on this @tgxworld. The problem that I’m having now (uploading 73GB) is that at some point I hit rate limits and the job craps out. And it seems that it starts over from the beginning each time, always reporting that there are something like 187123 images to upload.

It’s not your problem, but I’d really like to get this move to spaces complete before I move this site to a new server Real Soon Now. Would you recommend that I just give in an go for AWS S3 or do you think this should work?

---

<div class="post-metadata">

**Author:** ![tgxworld](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tgxworld/32/106117_2.png) [@tgxworld](https://meta.discourse.org/u/tgxworld)\
**Post date:** [2019年三月28日 03:59 UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/27 "2019-03-28T03:59:52Z")

</div>

If it is urgent, I’ll recommend going with AWS S3 first since the rake task is used internally by us to migrate uploads to the S3 store. In theory, the rake task should work with Spaces as well but it wouldn’t be our top priority at the moment.

---

<div class="post-metadata">

**Author:** ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)\
**Post date:** [2019年三月28日 04:23 UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/28 "2019-03-28T04:23:52Z")

</div>

Indeed. It’s lots faster too. It’s an insane hassle to set up, but I think that spaces isn’t really ready for prime time.

---

<div class="post-metadata">

**Author:** ![system](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/system/32/443519_2.png) [@system](https://meta.discourse.org/u/system)\
**Post date:** [2019年四月27日 04:23 UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/29 "2019-04-27T04:23:58Z")

</div>

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

[上一頁](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662.md?page=1)