`rake uploads:migrate_to_s3` needs some love

pfaffman · February 18, 2019, 10:55pm

The uploads:migrate_to_s3 rake task could use some attention.

There is no GlobalSetting.use_s3?, but a GlobalSetting.use_s3 and the ENV variables have changed, as DISCOURSE_S3_BUCKET has been replaced by DISCOURSE_S3_UPLOAD_BUCKET.

github.com/discourse/discourse

lib/tasks/uploads.rake

main


      
                sha1 = Upload.generate_digest(file_path)
                url = file_path.split(public_directory, 2)[1]
          
                if Upload.where(sha1: sha1).empty? && Upload.where(url: url).empty? &&
                     OptimizedImage.where(sha1: sha1).empty? && OptimizedImage.where(url: url).empty?
                  FileUtils.rm(file_path)
                  putc "#"
                else
                  putc "."
                end
              end
          
            puts "Removing empty directories..."
            puts `find #{uploads_directory} -type d -empty -exec rmdir {} \\;`
          
            puts "Done!"
          end
          
          ################################################################################
          #                                missing files                                 #
          ################################################################################

I’m only a little bit sure that I know how to fix this, and I think that it’s fresh in @gerhard’s mind.

tgxworld · February 18, 2019, 11:59pm

[1] pry(main)> GlobalSetting.use_s3
NoMethodError: undefined method `use_s3' for GlobalSetting:Class
Did you mean?  use_s3?

github.com/discourse/discourse

config/discourse_defaults.conf

c08b2ddec


      
          secret_key_base =
          
          # fallback path for all assets which are served via the application
          # used by static_controller
          # in multi host setups this allows you to have old unicorn instances serve
          # newly compiled assets
          fallback_assets_path =
          
          # S3 settings used for serving ALL public files
          # be sure to configre a CDN as well per cdn_url
          s3_bucket =
          s3_region =
          s3_access_key_id =
          s3_secret_access_key =
          s3_use_iam_profile = false
          s3_cdn_url =
          s3_endpoint =
          s3_force_path_style =
          
          ### rate limits apply to all sites
          max_user_api_reqs_per_minute = 20

The code in the rake task is still valid from what I can tell. Did you run into any specific problems that made you bring up the issues above?

pfaffman · February 19, 2019, 12:00am

uh. Hmm. I’ll try again.

I think I’m confused because it seems like it should be able to read those variables from SiteSettings and I don’t understand how the backup and upload buckets get defined from just the S3 bucket.

tgxworld · February 19, 2019, 12:17am

For GlobalSettings,

github.com/discourse/discourse

config/discourse_defaults.conf

4f5aebc33


      
          # Discourse supports multiple mechanisms for production config.
          #
          # 1. You can do nothing and get these defaults (not recommended, you should at least set hostname)
          # 2. You can copy this file to config/discourse.conf and amend with your settings
          # 3. You can pass in config from your environment, all the settings below are available.
          #    Append DISCOURSE_ and upper case the setting in ENV. For example:
          #    to pass in db_timeout of 200 you would use DISCOURSE_DB_TIMEOUT=200

Only SiteSettings that are marked shadowed_by_global can be configured via the env. In this case, s3_upload_bucket isn’t.

github.com/discourse/discourse

app/models/site_setting.rb

4f5aebc33


      
          def self.s3_upload_bucket
            SiteSetting.enable_s3_uploads ? SiteSetting.s3_upload_bucket : GlobalSetting.s3_bucket
          end

Closing this as this isn’t a bug.

Topic		Replies	Views
S3 configuration expectations between global and site settings Development	0	910	June 4, 2020
GlobalSetting.use_s3? - false Self-hosting s3	3	715	June 10, 2020
Migrate_from_s3: You must disable S3 uploads before running that task Product Support	21	1592	September 19, 2020
Rake uploads:migrate_from_s3 fails Self-hosting s3	31	3500	April 24, 2021
Uploads migration to S3 fails Product Support	26	3569	May 7, 2021

`rake uploads:migrate_to_s3` needs some love

AI-generated summary

Related topics