Hello guys,
How can I completely turn off AWS S3 storage and bring back all the files and backup in the S3 storage back to my EC2 server?
1 Like
I believe this will do the trick:
./launcher enter app
rake uploads:migrate_from_s3
rake posts:rebake
This migrates uploads FROM S3 to your web server. Then you can go back to your app.yml and change DISCOURSE_USE_S3: false. Then rebuild.
9 Likes
I would also go in the admin area of discourse and uncheck all the boxes that activate S3 mode.
2 Likes
I did as mentioned by @AntiMetaman but it gives me an error rake aborted. First, it asks me to turn off S3 upload. I turned of s3 upload and tried rake again, then this error pops up.
root@ip-172-31-7-247-app:/var/www/discourse# rake uploads:migrate_from_s3 --trace
** Invoke uploads:migrate_from_s3 (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute uploads:migrate_from_s3
Migrating uploads from S3 to local storage for 'default'...
rake aborted!
NoMethodError: undefined method `downcase' for nil:NilClass
/var/www/discourse/app/models/global_setting.rb:107:in `s3_bucket_name'
/var/www/discourse/app/models/site_setting.rb:157:in `absolute_base_url'
/var/www/discourse/lib/tasks/uploads.rake:138:in `migrate_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:118:in `block in migrate_all_from_s3'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:68:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:78:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:118:in `migrate_all_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:93:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:281:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:281:in `execute'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/usr/local/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:80:in `run'
bin/rake:13:in `<top (required)>'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `load'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `kernel_load'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:28:in `run'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:476:in `exec'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:30:in `dispatch'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:24:in `start'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_from_s3This one, right?
I came across this topic today. Might be worth holding off if you can.
If you turn off S3 only from the admin panel, that only disables it at site level. You want to globally disable it which is why you have to change the value from true to false in the app.yml and then rebuild.
It should work if you setup your S3. I just tried a migrate_from_s3 and migrate_to_s3. It works for me. I haven’t gotten any errors. I don’t know what this error means so a Discourse dev at this point should help:
NoMethodError: undefined method downcase’ for nil:NilClass`
Strange my reply below was deleted; so I am posting it again.
That error occurs when a string method is called on an object which is nil; and generally happens in Ruby when a string object is expected, but it is nil.
That is why that error occurs.
1 Like
Added DISCOURSE_USE_S3: false to app.yml but still the same error. Which version of discourse are you using? I am using the latest version.
I am also on latest. Did you get any errors on rebuild? If so, can you post those?
No, Not got any kind of error.
If the error is only after your rake command, then can you post exactly what you typed? When you first got S3 configured, didn’t you have to migrate files to S3?
Here is my full app.yml
## this is the all-in-one, standalone Discourse Docker container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild app
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed
templates:
- "templates/postgres.template.yml"
- "templates/redis.template.yml"
- "templates/web.template.yml"
- "templates/web.ratelimited.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
- "templates/web.ssl.template.yml"
- "templates/web.letsencrypt.ssl.template.yml"
## - "templates/cloudflare.template.yml"
## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see https://meta.discourse.org/t/17247 for details
expose:
- "80:80" # http
- "443:443" # https
params:
db_default_text_search_config: "pg_catalog.english"
## Set db_shared_buffers to a max of 25% of the total memory.
## will be set automatically by bootstrap based on detected RAM, or you can override
db_shared_buffers: "128MB"
## can improve sorting performance, but adds memory usage per-connection
#db_work_mem: "40MB"
## Which Git revision should this container use? (default: tests-passed)
#version: tests-passed
env:
LANG: en_US.UTF-8
# DISCOURSE_DEFAULT_LOCALE: en
## How many concurrent web requests are supported? Depends on memory and CPU cores.
## will be set automatically by bootstrap based on detected CPUs, or you can override
UNICORN_WORKERS: 2
## TODO: The domain name this Discourse instance will respond to
DISCOURSE_HOSTNAME: engineersasylum.com
## Uncomment if you want the container to be started with the same
## hostname (-h option) as specified above (default "$hostname-$config")
#DOCKER_USE_HOSTNAME: true
## TODO: List of comma delimited emails that will be made admin and developer
## on initial signup example 'user1@example.com,user2@example.com'
DISCOURSE_DEVELOPER_EMAILS: 'praveen369gen@gmail.com'
## TODO: The SMTP mail server used to validate new accounts and send notifications
DISCOURSE_SMTP_ADDRESS: email-smtp.us-east-1.amazonaws.com
DISCOURSE_SMTP_PORT: 587
DISCOURSE_SMTP_USER_NAME: AXXXXXXXXXXXXXXX
DISCOURSE_SMTP_PASSWORD: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
#DISCOURSE_SMTP_ENABLE_START_TLS: true # (optional, default true)
## If you added the Lets Encrypt template, uncomment below to get a free SSL certificate
LETSENCRYPT_ACCOUNT_EMAIL: praxxxxxx@gmail.com
## The CDN address for this Discourse instance (configured to pull)
## see https://meta.discourse.org/t/14857 for details
#DISCOURSE_CDN_URL: //discourse-cdn.example.com
DISCOURSE_USE_S3: false
## The Docker container is stateless; all data is stored in /shared
volumes:
- volume:
host: /var/discourse/shared/standalone
guest: /shared
- volume:
host: /var/discourse/shared/standalone/log/var-log
guest: /var/log
## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
after_code:
- exec:
cd: $home/plugins
cmd:
- git clone https://github.com/discourse/docker_manager.git
- git clone https://github.com/discourse/discourse-solved.git
- git clone https://github.com/discourse/discourse-adplugin.git
- git clone https://github.com/discoursehosting/discourse-sitemap.git
- git clone https://github.com/discourse/discourse-whos-online.git
- git clone https://github.com/discourse/discourse-push-notifications.git
- git clone https://github.com/discourse/discourse-chat-integration.git
- git clone https://github.com/davidtaylorhq/discourse-telegram-notifications.git
- git clone https://github.com/discourse/discourse-knowledge-explorer.git
- git clone https://github.com/discourse/discourse-math.git
## Any custom commands to run after building
run:
- exec: echo "Beginning of custom commands"
## If you want to set the 'From' email address for your first registration, uncomment and change:
## After getting the first signup email, re-comment the line. It only needs to run once.
#- exec: rails r "SiteSetting.notification_email='info@unconfigured.discourse.org'"
- exec: echo "End of custom commands"
I don’t remember exactly, but I think I have migrated to S3 using rake uploads:migrate_to_s3
I have no clue but here’s a similar thread that has the exact same error you did:
You may want to check with a team member.
Hi @Pravi
Sidebar question:
In your post of your yml you “say”:
DISCOURSE_SMTP_PASSWORD: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
Is this correct? Is the password in double quotes in your yml?
Yes, it is in double quotes. I have not changed it since I added it two years ago.
I have removed the double quotes, then rebuild the app again. Then tried rake uploads:migrate_from_s3 again, but still no luck. Same error
Hi @Pravi
To be honest, am an anti-fan of AWS and S3 so I do not use them.
Have you considered looking at the Rake task and reverse engineering that script and simply do the task manually, step-by-step?
I don’t have that much idea of discourse internal script and I am little scared to edit the script. I don’t want to ruin everything.
Can you tell me the location of discourse.config file? I can’t find it anywhere on my server. The find command returns nothing.
root@ip-172-31-7-247:/var/discourse# find . -name "*.config"
root@ip-172-31-7-247:/var/discourse# find . -name "*.conf"
./shared/standalone/letsencrypt/engineersasylum.com/engineersasylum.com.conf
./shared/standalone/letsencrypt/engineersasylum.com/engineersasylum.com.csr.conf
./shared/standalone/letsencrypt/engineersasylum.com_ecc/engineersasylum.com.conf
./shared/standalone/letsencrypt/engineersasylum.com_ecc/engineersasylum.com.csr.conf
./shared/standalone/letsencrypt/ca/acme-v01.api.letsencrypt.org/ca.conf
./shared/standalone/letsencrypt/ca/acme-v02.api.letsencrypt.org/ca.conf
./shared/standalone/letsencrypt/account.conf
./shared/standalone/postgres_data/pg_hba.conf
./shared/standalone/postgres_data/postgresql.auto.conf
./shared/standalone/postgres_data/pg_ident.conf
./shared/standalone/postgres_data/postgresql.conf
root@ip-172-31-7-247:/var/discourse#Hi @Pravi
Sorry that you misunderstood me.
I did not ask you to edit that file or any file.
To “Reverse engineer” a script, in my mind, means to study it and see how it works and has nothing to do with editing the file. It means to read it and understand it.
Normally if we read and understand any script or file giving us an error, we can normally get an understanding of how to fix the problem.
You are posting error messages from a file which you have not read, or so it seems.
Hope this helps.
I get your point. I will try to do it.
Can you suggest the location of the file ‘discourse.config’? Where are the configuration for the S3, stored?