### Author:  [@justin](https://meta.discourse.org/u/justin)
#### Post date: [20 בפברואר, 2019, 4:12pm UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/2 "2019-02-20T16:12:16Z")
I ended up getting this working myself by adding those environment variables to app.yml and rebuilding. Feels like a workaround and probably not the best way forward long-term.
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [21 בפברואר, 2019, 1:11am UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/4 "2019-02-21T01:11:50Z")
@rishabh and @zogstrip worked on this task recently so they will probably be able to advise better.
---
### Author:  [@rishabh](https://meta.discourse.org/u/rishabh)
#### Post date: [21 בפברואר, 2019, 10:41am UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/6 "2019-02-21T10:41:11Z")
> [@pfaffman](#):
>
> The latest `uploads.rb` cranks up `Aws::S3` with just a few environment variables (and not those required for Digital Ocean spaces).
I’ve fixed this in:
[https://github.com/discourse/discourse/commit/e69634ec3aa594abd3c2d1057dcf1f65d09c48f5](https://github.com/discourse/discourse/commit/e69634ec3aa594abd3c2d1057dcf1f65d09c48f5)
For the GlobalSetting issue, can you try running the command like this to test if your config is alright?
```plaintext
DISCOURSE_S3_ACCESS_KEY_ID="" \
DISCOURSE_S3_SECRET_ACCESS_KEY="" \
DISCOURSE_S3_REGION="us-east-1" \
DISCOURSE_S3_BUCKET="uploadbucket" \
DISCOURSE_S3_CDN_URL="https://xyz.com" \
DRY_RUN=1 rake uploads:migrate_to_s3
```
_Please make sure that you’ve set the correct DigitalOcean endpoint in `SiteSetting.s3_endpoint` and that you’ve set DRY\_RUN=0 to make the actual migration happen._
---
### Author:  [@hosna](https://meta.discourse.org/u/hosna)
#### Post date: [21 בפברואר, 2019, 11:36am UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/7 "2019-02-21T11:36:04Z")
Any news for [minio](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333)?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [27 בפברואר, 2019, 8:00pm UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/8 "2019-02-27T20:00:24Z")
I still don’t see how ` GlobalSetting.use_s3?` is ever true. I have `GlobalSetting.use_s3`, but not with a `?` and I don’t see how seeting those ENV vars sets `GlobalSetting.use_s3?`. Maybe that’s supposed to happen somewhere that I don’t know about or understand, but I removed the `?` in order to be able to run the script at all.
So I removed the `?` and the dry run looked promising and I ran it for real. It took a couple days (70GB files and a badly configured network adapter) and then it failed like:
```plaintext
Updating the URLs in the database... [14/9360]
rake aborted!
PG::StringDataRightTruncation: ERROR: value too long for type character varying(500)
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.2/lib/patches/db/pg.rb:110:i
n `async_exec'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.2/lib/patches/db/pg.rb:110:i
n `async_exec'
(eval):24:in `async_exec'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_sql-0.2.1/lib/mini_sql/postgres/connection.rb:
118:in `run'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_sql-0.2.1/lib/mini_sql/postgres/connection.rb:
90:in `exec'
/var/www/discourse/lib/db_helper.rb:31:in `block in remap'
/var/www/discourse/lib/db_helper.rb:20:in `each'
/var/www/discourse/lib/db_helper.rb:20:in `remap'
/var/www/discourse/lib/tasks/uploads.rake:364:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:210:in `block in migrate_to_s3_all_sites'
...
```
It’s probably coincidence that it’s the same error as [Trouble with `discourse remap` remapping `topic_links url`](https://meta.discourse.org/t/trouble-with-discourse-remap-remapping-topic-links-url/110052) (it is a different site).
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [27 בפברואר, 2019, 8:50pm UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/9 "2019-02-27T20:50:53Z")
> [@pfaffman](#):
>
> I have `GlobalSetting.use_s3` , but not with a `?`
It’s right here:
>
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [27 בפברואר, 2019, 9:07pm UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/10 "2019-02-27T21:07:28Z")
Ah! I figured it must be in some place I didn’t understand. Thanks for that.
Now that I understand, I think that it’s because I don’t have `GlobalSetting.s3_region`, even if I do pass the ENV variable.
Any guess on what table is causing the `value too long`?
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [27 בפברואר, 2019, 9:31pm UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/11 "2019-02-27T21:31:20Z")
> [@rishabh](#):
>
> ```plaintext
> DISCOURSE_S3_ACCESS_KEY_ID="" \
> DISCOURSE_S3_SECRET_ACCESS_KEY="" \
> DISCOURSE_S3_REGION="us-east-1" \
> DISCOURSE_S3_BUCKET="uploadbucket" \
> DISCOURSE_S3_CDN_URL="https://xyz.com" \
> DRY_RUN=1 rake uploads:migrate_to_s3
>
> ```
That works for me. You are setting a value for for the access keys, right?
> [@pfaffman](#):
>
> I don’t have `GlobalSetting.s3_region` , even if I do pass the ENV variable.
What makes you think that this is the case? You can replace `DRY_RUN=1 rake uploads:migrate_to_s3` with `rails c` in the above command and inspect the values of the settings if you want to see what’s happening…
> [@pfaffman](#):
>
> Any guess on what table is causing the `value too long` ?
It must be the `url` column in the `topic_links` table. It’s the only column with a restriction of 500 chars.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [27 בפברואר, 2019, 9:43pm UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/12 "2019-02-27T21:43:31Z")
> [@gerhard](#):
>
> It must be the `url` column in the `topic_links` table. It’s the only column with a restriction of 500 chars.
Thanks for that. I’ve got 30 on this site (different from the site discussed [here](https://meta.discourse.org/t/trouble-with-discourse-remap-remapping-topic-links-url/110052)) with more than 490 characters. I’m not clear why the remap has this problem. Oh! Is it because this table is getting populated when the posts are rebaked? If that’s it, isn’t it a bug if sticking a URL with \>500 characters in a post causes this error? Should there be some error checking that catches it?
> [@Trouble with \`discourse remap\` remapping \`topic\_links url\`](https://meta.discourse.org/t/trouble-with-discourse-remap-remapping-topic-links-url/110052):
>
> I’m trying to discourse remap community.x.com staging.community.x.com as described [here](https://meta.discourse.org/t/replace-a-string-in-all-posts/48729) (edit: Hmm. Maybe I should have made this a post over there?) and am getting this error: Remapping topic\_links url Error: ERROR: value too long for type character varying(500) The remap has only been partially applied due to the error above. Please re-run the script again.
Tagging @falco since he was involved in the other topic about this.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [27 בפברואר, 2019, 10:05pm UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/13 "2019-02-27T22:05:45Z")
> [@gerhard](#):
>
> You are setting a value for for the access keys, right?
I think the problem was that the S3\_REGION wasn’t set. I just added it to the ENV in `app.yml` and tried again and now.
```
DRY_RUN=1 rake uploads:migrate_to_s3
```
works as expected.
Now I’d like to figure out what to do about the `topic_links` table, as even now that the files are uploaded, it doesn’t look like it’s any (much?) faster and there are \>200K files.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [28 בפברואר, 2019, 3:01pm UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/14 "2019-02-28T15:01:49Z")
I’ll see if I can track down the topic link thing. I guess it’s in `app/models/topic_link`?
The other thing that I don’t understand is how the assets get pushed to S3. I have the S3 cdn defined and now all of the css files are 403 because it’s looking for them in the S3 cnd bucket. I tried bootstrapping with the S3 cdn bucket set, but that didn’t fix it. (and worse, my monitor only looks for ‘discourse’ in the output and didn’t notice they all the assets were broken).
EDIT: Perhaps the URL should be truncated here:
>
I’ve been working on this (off and on) for weeks and would love to get this sorted.
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [28 בפברואר, 2019, 3:58pm UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/15 "2019-02-28T15:58:09Z")
> [@pfaffman](#):
>
> I’ve got 30 on this site (different from the site discussed [here](https://meta.discourse.org/t/trouble-with-discourse-remap-remapping-topic-links-url/110052)) with more than 490 characters. I’m not clear why the remap has this problem.
Take a very good look at those 30 links. Why are they so long? Do the links even make sense? And why are they affected by the remap? Because, no, this isn’t caused by a rebake but by the remapping of upload URLs.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [28 בפברואר, 2019, 4:14pm UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/16 "2019-02-28T16:14:36Z")
> [@gerhard](#):
>
> Do the links even make sense?
No. They are a bunch of stupid links, created by children. The other community where there are 3 such links, one was a create-a-topic link and the other two were something else stupid.
Maybe what to do is just skip URIs that are longer than 500 chars, assuming that they are nonsense?
EDIT: That seems to work, but now I’m getting a non-uniq violation. I’m guessing it’s unrelated?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [28 בפברואר, 2019, 5:17pm UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/17 "2019-02-28T17:17:36Z")
This is my showstopper on this S3 migration right now. (I think I’ll just truncate the topic\_links urls in the DB as suggested [here](https://meta.discourse.org/t/trouble-with-discourse-remap-remapping-topic-links-url/110052/7)).
> [@pfaffman](#):
>
> The other thing that I don’t understand is how the assets get pushed to S3. I have the S3 cdn defined and now all of the css files are 403 because it’s looking for them in the S3 cnd bucket. I tried bootstrapping with the S3 cdn bucket set, but that didn’t fix it. (and worse, my monitor only looks for ‘discourse’ in the output and didn’t notice they all the assets were broken).
How do the assets get pushed to S3?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [1 במרץ, 2019, 1:23am UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/18 "2019-03-01T01:23:31Z")
Well, now it’s even stranger:
```plaintext
ActionController::RoutingError (No route matches [GET] "/t/why-was-the-forum-down-for-so-long/254643/https:/lc-rbx.ams3.cdn.digitaloceanspaces.com/uploads/original/3X/6/1/6191c280e518330ba11197299834f074fe49eb33.gif")
```
See how something is contatenating an image to the route?
I’m also unable to replace any of the site logos.
I can upload to S3 when creating a new post; I can turn off S3 uploads and upload to local storage.
I’m increasingly stumped.
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [11 במרץ, 2019, 6:32am UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/19 "2019-03-11T06:32:47Z")
This was a sneaky one…
[https://github.com/discourse/discourse/commit/d82876896eaf2ccd150ea9202003ed11742d2136](https://github.com/discourse/discourse/commit/d82876896eaf2ccd150ea9202003ed11742d2136)
If you’re on a docker based install, `GlobalSetting` uses the `FileProvider` which reads from `/var/www/discourse/config/discourse.conf`. That file is [only populated from the ENV once when the container is first started](https://github.com/discourse/discourse_docker/blob/024ebc4037b747d921433a45466e9fff45b0710c/templates/web.template.yml#L40-L49). As a result, running `DISCOURSE_S3_REGION rails console` will not have any effect on `GlobalSetting.s3_region` since it has not been added to `config/discourse.conf`.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [11 במרץ, 2019, 10:24am UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/20 "2019-03-11T10:24:07Z")
Thanks, @tgxworld. Does this fix the concatenated path I referenced above?
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [12 במרץ, 2019, 2:37am UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/21 "2019-03-12T02:37:23Z")
> [@pfaffman](#):
>
> I’m also unable to replace any of the site logos.
Do you have more information about the error here?
> [@pfaffman](#):
>
> See how something is contatenating an image to the route?
How are you running into this route?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [13 במרץ, 2019, 4:57pm UTC](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662/22 "2019-03-13T16:57:45Z")
> [@tgxworld](#):
>
> Do you have more information about the error here?
Problem with site logos had to do with the image processing. I changed the `png` to a `gif` and the error went away. I couldn’t figure out what was up with the image that caused the image processing to fail.
> [@tgxworld](#):
>
> How are you running into this route?
Not sure if it’s related, but I think my database is hosed. I got this trying to do a restore:
```plaintext
[2019-03-13 15:44:16] ERROR: invalid input syntax for integer: "{"toplose 6 forums are closing","origi lash meugina93fep93fep9titZ quipaWin _tit,"or aM3:46:1130"
[2019-03-13 15:44:16] CONTEXT: COPY notifications, line 135659, column post_action_id: "{"toplose 6 forums are closing","origi lash meugina93fep93fep9titZ quipaWin _tit,"or aM3:46:1130"
[2019-03-13 15:44:16] EXCEPTION: psql failed
```
[Next page](https://meta.discourse.org/t/migrate-to-s3-for-digital-ocean-spaces-woes/109662.md?page=2)