Migrate_from_s3: You must disable S3 uploads before running that task

I am trying to migrate all data from S3 back to my server. But, rake uploads:migrate_from_s3 returns error You must disable S3 uploads before running that task. So I unchecked enable s3 uploads option in my discourse setting and also added DISCOURSE_USE_S3: false to my app.yml file. But still no luck, the same error.

[1] pry(main)> SiteSetting.enable_s3_uploads
=> false
[2] pry(main)> GlobalSetting.use_s3
=> false
[3] pry(main)> GlobalSetting.use_s3?
=> true

I am confused. How to disable it?

Does your app.yml contain S3 env variables? You may have to check those to disable s3 uploads.

What is an S3 env variable? In my case, DISCOURSE_USE_S3 is kept under env.

env:
  LANG: en_US.UTF-8
  # DISCOURSE_DEFAULT_LOCALE: en

  UNICORN_WORKERS: 2

  ## TODO: The domain name this Discourse instance will respond to
  DISCOURSE_HOSTNAME: engineersasylum.com

  ## Uncomment if you want the container to be started with the same
  ## hostname (-h option) as specified above (default "$hostname-$config")
  #DOCKER_USE_HOSTNAME: true

  ## TODO: List of comma delimited emails that will be made admin and developer
  ## on initial signup example 'user1@example.com,user2@example.com'
  DISCOURSE_DEVELOPER_EMAILS: 'praveen369gen@gmail.com'

  #S3 Configuration
  DISCOURSE_USE_S3: false
  DISCOURSE_S3_REGION: ap-south-1
  DISCOURSE_S3_ACCESS_KEY_ID: AKxxxxxxxxxxxxxxxxxxxxxDQ
  DISCOURSE_S3_SECRET_ACCESS_KEY: QWVxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxLigGs
  DISCOURSE_S3_BUCKET: enginxxxxxxxxxxket
  DISCOURSE_S3_BACKUP_BUCKET: enginxxxxxxxxxxxxcket
  DISCOURSE_BACKUP_LOCATION: s3
1 Like

… and don’t forgot @Pravi that you must rebuild your container (app) after you change most of the vars in your yml file; some might change by stopping and starting your container; but when it doubt, rebuild the container (app) after you make a change to your yml.

I rebuild the container after every change in the app.yml file. Still, its turned off.

Try changing backup location to local?
I’m not sure if it would help but it’ll be my best bet.

I added local to backup location, then rebuild the container and tried to migrate, but still no luck. Its giving me You must disable S3 uploads before running that task

I’m sorry, I don’t know what may be causing this.

After you rebuild with the DISCOURSE_USE_S3=false in the yml, you ran the rake task and it still says “You must disable S3 uploads”? Did you go to your admin panel and disable S3 there also?

At the moment, what’s the output of:

SiteSetting.Upload.enable_s3_uploads
SiteSetting.enable_s3_uploads
GlobalSetting.use_s3?

The logic for GlobalSetting.use_s3? doesn’t use the environment in the same way - it uses:

  def self.use_s3?
    (@use_s3 ||=
      begin
        s3_bucket &&
        s3_region && (
          s3_use_iam_profile || (s3_access_key_id && s3_secret_access_key)
        ) ? :true : :false
      end) == :true
  end

I can see how this might be confusing - I’ll have someone look at it.

For now, unsetting DISCOURSE_USE_S3 in the environment and DISCOURSE_S3_BUCKET should let you migrate from S3.

2 Likes

This is what I get,

[1] pry(main)> SiteSetting.Upload.enable_s3_uploads
=> true
[2] pry(main)> SiteSetting.enable_s3_uploads
=> false
[3] pry(main)> GlobalSetting.use_s3?
=> true
[4] pry(main)>

Yes, I add DISCOURSE_USE_S3=false to the app.yml, then I rebuild the container, then rake task but still, it’s giving me same output. Yes, I disabled upload in the admin page.

As said here, you should also unset the DISCOURSE_S3_BUCKET In the yml. Probably just put a # before it.

Right now, I am rebuilding the container. I will report the outcome in couple of minutes.

I don’t know how the last one can be true if you set the env variable to false and rebuild. The SiteSetting ones are from your admin panel and the GlobalSettings is through the app.yml.

The SiteSetting in my admin panel is disabled.

s3

S3 Configuration

  #DISCOURSE_USE_S3: false
  DISCOURSE_S3_REGION: ap-south-1
  DISCOURSE_S3_ACCESS_KEY_ID: AKXXXXXXXXXXXXXXXDQ
  DISCOURSE_S3_SECRET_ACCESS_KEY: AvsvsxxxxxxxxxxxxxxxxvcccccccLigGs
  #DISCOURSE_S3_BUCKET: engineers-leakybucket
  DISCOURSE_S3_BACKUP_BUCKET: engineers-backupbucket
  DISCOURSE_BACKUP_LOCATION: s3
  #DISCOURSE_S3_CDN_URL: https://engineers-leakybucket.s3.dualstack.ap-south-1.amazonaws.com

Unsetting DISCOURSE_USE_S3 and DISCOURSE_S3_BUCKET gives me this error,

root@ip-172-31-7-247-app:/var/www/discourse# rake uploads:migrate_from_s3
Migrating uploads from S3 to local storage for 'default'...
rake aborted!
NoMethodError: undefined method `downcase' for nil:NilClass
/var/www/discourse/app/models/global_setting.rb:107:in `s3_bucket_name'
/var/www/discourse/app/models/site_setting.rb:157:in `absolute_base_url'
/var/www/discourse/lib/tasks/uploads.rake:138:in `migrate_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:118:in `block in migrate_all_from_s3'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:68:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:78:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:118:in `migrate_all_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:93:in `block in <main>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_from_s3
(See full trace by running task with --trace)

Update:

Before Unsetting DISCOURSE_USE_S3 and DISCOURSE_S3_BUCKET,

[1] pry(main)> SiteSetting.Upload.enable_s3_uploads
=> true
[2] pry(main)> SiteSetting.enable_s3_uploads
=> false
[3] pry(main)> GlobalSetting.use_s3?
=> true
[4] pry(main)> 

After Unsetting DISCOURSE_USE_S3 and DISCOURSE_S3_BUCKET,

[1] pry(main)> SiteSetting.Upload.enable_s3_uploads
=> false
[2] pry(main)> SiteSetting.enable_s3_uploads
=> false
[3] pry(main)> GlobalSetting.use_s3?
=> false
[4] pry(main)>
1 Like

Hi @Pravi

Yes, I was going to suggest just now, you comment out all these ENVs, and try this:

DISCOURSE_USE_S3: false
#DISCOURSE_S3_REGION: ap-south-1
#DISCOURSE_S3_ACCESS_KEY_ID: AKXXXXXXXXXXXXXXXDQ
#DISCOURSE_S3_SECRET_ACCESS_KEY: AvsvsxxxxxxxxxxxxxxxxvcccccccLigGs
##DISCOURSE_S3_BUCKET: engineers-leakybucket
#DISCOURSE_S3_BACKUP_BUCKET: engineers-backupbucket
#DISCOURSE_BACKUP_LOCATION: s3
#DISCOURSE_S3_CDN_URL: https://engineers-leakybucket.s3.dualstack.ap-south-1.amazonaws.com 

But then I read your latest update (thanks for the details) and it seems you are making progress.

Congrats on making some progress. Good job!

I tried that too. Commented out everything except DISCOURSE_USE_S3: false and the same error as previous pops up.

root@ip-172-31-7-247-app:/var/www/discourse# rake uploads:migrate_from_s3
Migrating uploads from S3 to local storage for 'default'...
rake aborted!
NoMethodError: undefined method `downcase' for nil:NilClass
/var/www/discourse/app/models/global_setting.rb:107:in `s3_bucket_name'
/var/www/discourse/app/models/site_setting.rb:157:in `absolute_base_url'
/var/www/discourse/lib/tasks/uploads.rake:138:in `migrate_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:118:in `block in migrate_all_from_s3'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:68:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:78:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:118:in `migrate_all_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:93:in `block in <main>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_from_s3
(See full trace by running task with --trace)

The SiteSetting and GlobalSettings are disabled.

root@ip-172-31-7-247-app:/var/www/discourse# rails c
[1] pry(main)> SiteSetting.Upload.enable_s3_uploads
=> false
[2] pry(main)> SiteSetting.enable_s3_uploads
=> false
[3] pry(main)> GlobalSetting.use_s3?
=> false
[4] pry(main)> 

Thanks, but I am not sure I have made any progress. Now I have to deal with this new error.

Anyone know how to fix this problem?