`rake uploads:migrate_to_s3` needs some love

pfaffman · February 18, 2019, 10:55pm

The uploads:migrate_to_s3 rake task could use some attention.

There is no GlobalSetting.use_s3?, but a GlobalSetting.use_s3 and the ENV variables have changed, as DISCOURSE_S3_BUCKET has been replaced by DISCOURSE_S3_UPLOAD_BUCKET.

https://github.com/discourse/discourse/blob/master/lib/tasks/uploads.rake#L231

I’m only a little bit sure that I know how to fix this, and I think that it’s fresh in @gerhard’s mind.

tgxworld · February 18, 2019, 11:59pm

[1] pry(main)> GlobalSetting.use_s3
NoMethodError: undefined method `use_s3' for GlobalSetting:Class
Did you mean?  use_s3?

github.com

discourse/discourse/blob/c08b2ddecfaed0f0ab3e3b30743b55d5f5230fb2/config/discourse_defaults.conf#L170


      
          secret_key_base =
          
          # fallback path for all assets which are served via the application
          # used by static_controller
          # in multi host setups this allows you to have old unicorn instances serve
          # newly compiled assets
          fallback_assets_path =
          
          # S3 settings used for serving ALL public files
          # be sure to configre a CDN as well per cdn_url
          s3_bucket =
          s3_region =
          s3_access_key_id =
          s3_secret_access_key =
          s3_use_iam_profile = false
          s3_cdn_url =
          s3_endpoint =
          s3_force_path_style =
          
          ### rate limits apply to all sites
          max_user_api_reqs_per_minute = 20

The code in the rake task is still valid from what I can tell. Did you run into any specific problems that made you bring up the issues above?

pfaffman · February 19, 2019, 12:00am

uh. Hmm. I’ll try again.

I think I’m confused because it seems like it should be able to read those variables from SiteSettings and I don’t understand how the backup and upload buckets get defined from just the S3 bucket.

tgxworld · February 19, 2019, 12:17am

For GlobalSettings,

github.com

discourse/discourse/blob/4f5aebc33812a56c34e553bd18c0cfe7ddc4c3f4/config/discourse_defaults.conf#L7-L13


      
          # Discourse supports multiple mechanisms for production config.
          #
          # 1. You can do nothing and get these defaults (not recommended, you should at least set hostname)
          # 2. You can copy this file to config/discourse.conf and amend with your settings
          # 3. You can pass in config from your environment, all the settings below are available.
          #    Append DISCOURSE_ and upper case the setting in ENV. For example:
          #    to pass in db_timeout of 200 you would use DISCOURSE_DB_TIMEOUT=200

Only SiteSettings that are marked shadowed_by_global can be configured via the env. In this case, s3_upload_bucket isn’t.

github.com

discourse/discourse/blob/4f5aebc33812a56c34e553bd18c0cfe7ddc4c3f4/app/models/site_setting.rb#L129-L131


      
          def self.s3_upload_bucket
            SiteSetting.enable_s3_uploads ? SiteSetting.s3_upload_bucket : GlobalSetting.s3_bucket
          end

Closing this as this isn’t a bug.

Topic		Replies	Views
Migrate_from_s3: You must disable S3 uploads before running that task Support	21	1274	September 19, 2020
GlobalSetting.use_s3? - false Installation s3	3	600	June 10, 2020
S3 configuration expectations between global and site settings Dev	0	828	June 4, 2020
Uploads migration to S3 fails Support	26	2949	May 7, 2021
Migrate_to_s3 for Digital Ocean Spaces woes Installation s3	26	3098	April 27, 2019

`rake uploads:migrate_to_s3` needs some love

Related topics