Migrate_from_s3: You must disable S3 uploads before running that task

Pravi · August 12, 2020, 2:29pm

I am trying to migrate all data from S3 back to my server. But, rake uploads:migrate_from_s3 returns error You must disable S3 uploads before running that task. So I unchecked enable s3 uploads option in my discourse setting and also added DISCOURSE_USE_S3: false to my app.yml file. But still no luck, the same error.

[1] pry(main)> SiteSetting.enable_s3_uploads
=> false
[2] pry(main)> GlobalSetting.use_s3
=> false
[3] pry(main)> GlobalSetting.use_s3?
=> true

I am confused. How to disable it?

itsbhanusharma · August 12, 2020, 2:39pm

Does your app.yml contain S3 env variables? You may have to check those to disable s3 uploads.

Pravi · August 12, 2020, 2:47pm

What is an S3 env variable? In my case, DISCOURSE_USE_S3 is kept under env.

env:
  LANG: en_US.UTF-8
  # DISCOURSE_DEFAULT_LOCALE: en

  UNICORN_WORKERS: 2

  ## TODO: The domain name this Discourse instance will respond to
  DISCOURSE_HOSTNAME: engineersasylum.com

  ## Uncomment if you want the container to be started with the same
  ## hostname (-h option) as specified above (default "$hostname-$config")
  #DOCKER_USE_HOSTNAME: true

  ## TODO: List of comma delimited emails that will be made admin and developer
  ## on initial signup example 'user1@example.com,user2@example.com'
  DISCOURSE_DEVELOPER_EMAILS: 'praveen369gen@gmail.com'

  #S3 Configuration
  DISCOURSE_USE_S3: false
  DISCOURSE_S3_REGION: ap-south-1
  DISCOURSE_S3_ACCESS_KEY_ID: AKxxxxxxxxxxxxxxxxxxxxxDQ
  DISCOURSE_S3_SECRET_ACCESS_KEY: QWVxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxLigGs
  DISCOURSE_S3_BUCKET: enginxxxxxxxxxxket
  DISCOURSE_S3_BACKUP_BUCKET: enginxxxxxxxxxxxxcket
  DISCOURSE_BACKUP_LOCATION: s3

neounix · August 12, 2020, 2:50pm

… and don’t forgot @Pravi that you must rebuild your container (app) after you change most of the vars in your yml file; some might change by stopping and starting your container; but when it doubt, rebuild the container (app) after you make a change to your yml.

Pravi · August 12, 2020, 2:53pm

I rebuild the container after every change in the app.yml file. Still, its turned off.

itsbhanusharma · August 12, 2020, 3:05pm

Try changing backup location to local?
I’m not sure if it would help but it’ll be my best bet.

Pravi · August 12, 2020, 3:20pm

I added local to backup location, then rebuild the container and tried to migrate, but still no luck. Its giving me You must disable S3 uploads before running that task

itsbhanusharma · August 12, 2020, 3:31pm

I’m sorry, I don’t know what may be causing this.

AntiMetaman · August 12, 2020, 5:03pm

After you rebuild with the DISCOURSE_USE_S3=false in the yml, you ran the rake task and it still says “You must disable S3 uploads”? Did you go to your admin panel and disable S3 there also?

supermathie · August 12, 2020, 5:56pm

At the moment, what’s the output of:

SiteSetting.Upload.enable_s3_uploads
SiteSetting.enable_s3_uploads
GlobalSetting.use_s3?

The logic for GlobalSetting.use_s3? doesn’t use the environment in the same way - it uses:

  def self.use_s3?
    (@use_s3 ||=
      begin
        s3_bucket &&
        s3_region && (
          s3_use_iam_profile || (s3_access_key_id && s3_secret_access_key)
        ) ? :true : :false
      end) == :true
  end

I can see how this might be confusing - I’ll have someone look at it.

For now, unsetting DISCOURSE_USE_S3 in the environment and DISCOURSE_S3_BUCKET should let you migrate from S3.

Pravi · August 12, 2020, 6:49pm

This is what I get,

[1] pry(main)> SiteSetting.Upload.enable_s3_uploads
=> true
[2] pry(main)> SiteSetting.enable_s3_uploads
=> false
[3] pry(main)> GlobalSetting.use_s3?
=> true
[4] pry(main)>

Pravi · August 12, 2020, 6:52pm

Yes, I add DISCOURSE_USE_S3=false to the app.yml, then I rebuild the container, then rake task but still, it’s giving me same output. Yes, I disabled upload in the admin page.

AntiMetaman · August 12, 2020, 6:56pm

As said here, you should also unset the DISCOURSE_S3_BUCKET In the yml. Probably just put a # before it.

Pravi · August 12, 2020, 6:57pm

Right now, I am rebuilding the container. I will report the outcome in couple of minutes.

AntiMetaman · August 12, 2020, 6:59pm

I don’t know how the last one can be true if you set the env variable to false and rebuild. The SiteSetting ones are from your admin panel and the GlobalSettings is through the app.yml.

Pravi · August 12, 2020, 7:04pm

The SiteSetting in my admin panel is disabled.

Pravi · August 12, 2020, 7:06pm

S3 Configuration

  #DISCOURSE_USE_S3: false
  DISCOURSE_S3_REGION: ap-south-1
  DISCOURSE_S3_ACCESS_KEY_ID: AKXXXXXXXXXXXXXXXDQ
  DISCOURSE_S3_SECRET_ACCESS_KEY: AvsvsxxxxxxxxxxxxxxxxvcccccccLigGs
  #DISCOURSE_S3_BUCKET: engineers-leakybucket
  DISCOURSE_S3_BACKUP_BUCKET: engineers-backupbucket
  DISCOURSE_BACKUP_LOCATION: s3
  #DISCOURSE_S3_CDN_URL: https://engineers-leakybucket.s3.dualstack.ap-south-1.amazonaws.com

Unsetting DISCOURSE_USE_S3 and DISCOURSE_S3_BUCKET gives me this error,

root@ip-172-31-7-247-app:/var/www/discourse# rake uploads:migrate_from_s3
Migrating uploads from S3 to local storage for 'default'...
rake aborted!
NoMethodError: undefined method `downcase' for nil:NilClass
/var/www/discourse/app/models/global_setting.rb:107:in `s3_bucket_name'
/var/www/discourse/app/models/site_setting.rb:157:in `absolute_base_url'
/var/www/discourse/lib/tasks/uploads.rake:138:in `migrate_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:118:in `block in migrate_all_from_s3'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:68:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:78:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:118:in `migrate_all_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:93:in `block in <main>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_from_s3
(See full trace by running task with --trace)

Update:

Before Unsetting DISCOURSE_USE_S3 and DISCOURSE_S3_BUCKET,

[1] pry(main)> SiteSetting.Upload.enable_s3_uploads
=> true
[2] pry(main)> SiteSetting.enable_s3_uploads
=> false
[3] pry(main)> GlobalSetting.use_s3?
=> true
[4] pry(main)>

After Unsetting DISCOURSE_USE_S3 and DISCOURSE_S3_BUCKET,

[1] pry(main)> SiteSetting.Upload.enable_s3_uploads
=> false
[2] pry(main)> SiteSetting.enable_s3_uploads
=> false
[3] pry(main)> GlobalSetting.use_s3?
=> false
[4] pry(main)>

neounix · August 13, 2020, 1:34am

Hi @Pravi

Yes, I was going to suggest just now, you comment out all these ENVs, and try this:

DISCOURSE_USE_S3: false
#DISCOURSE_S3_REGION: ap-south-1
#DISCOURSE_S3_ACCESS_KEY_ID: AKXXXXXXXXXXXXXXXDQ
#DISCOURSE_S3_SECRET_ACCESS_KEY: AvsvsxxxxxxxxxxxxxxxxvcccccccLigGs
##DISCOURSE_S3_BUCKET: engineers-leakybucket
#DISCOURSE_S3_BACKUP_BUCKET: engineers-backupbucket
#DISCOURSE_BACKUP_LOCATION: s3
#DISCOURSE_S3_CDN_URL: https://engineers-leakybucket.s3.dualstack.ap-south-1.amazonaws.com

But then I read your latest update (thanks for the details) and it seems you are making progress.

Congrats on making some progress. Good job!

Pravi · August 13, 2020, 4:08am

I tried that too. Commented out everything except DISCOURSE_USE_S3: false and the same error as previous pops up.

root@ip-172-31-7-247-app:/var/www/discourse# rake uploads:migrate_from_s3
Migrating uploads from S3 to local storage for 'default'...
rake aborted!
NoMethodError: undefined method `downcase' for nil:NilClass
/var/www/discourse/app/models/global_setting.rb:107:in `s3_bucket_name'
/var/www/discourse/app/models/site_setting.rb:157:in `absolute_base_url'
/var/www/discourse/lib/tasks/uploads.rake:138:in `migrate_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:118:in `block in migrate_all_from_s3'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:68:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:78:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:118:in `migrate_all_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:93:in `block in <main>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_from_s3
(See full trace by running task with --trace)

The SiteSetting and GlobalSettings are disabled.

root@ip-172-31-7-247-app:/var/www/discourse# rails c
[1] pry(main)> SiteSetting.Upload.enable_s3_uploads
=> false
[2] pry(main)> SiteSetting.enable_s3_uploads
=> false
[3] pry(main)> GlobalSetting.use_s3?
=> false
[4] pry(main)>

Thanks, but I am not sure I have made any progress. Now I have to deal with this new error.

Pravi · August 14, 2020, 4:16am

Anyone know how to fix this problem?

Topic		Replies	Views
How to turn off S3 storage? installation	28	2004	August 12, 2020
Uploads migration to S3 fails support	26	2803	May 7, 2021
Migrate_to_s3 for Digital Ocean Spaces woes installation s3	26	2920	April 27, 2019
Defining DISCOURSE_S3_CDN_URL links to assets in S3 CDN URL support	43	5409	April 22, 2020
Rake uploads:migrate_from_s3 fails installation s3	31	2632	April 24, 2021

Migrate_from_s3: You must disable S3 uploads before running that task

Related Topics