How to turn off S3 storage?

I am also on latest. Did you get any errors on rebuild? If so, can you post those?

No, Not got any kind of error.

If the error is only after your rake command, then can you post exactly what you typed? When you first got S3 configured, didn’t you have to migrate files to S3?

Here is my full app.yml

## this is the all-in-one, standalone Discourse Docker container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild app
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
  - "templates/web.ssl.template.yml"
  - "templates/web.letsencrypt.ssl.template.yml"
##  - "templates/cloudflare.template.yml"

## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see https://meta.discourse.org/t/17247 for details
expose:
  - "80:80"   # http
  - "443:443" # https

params:
  db_default_text_search_config: "pg_catalog.english"

  ## Set db_shared_buffers to a max of 25% of the total memory.
  ## will be set automatically by bootstrap based on detected RAM, or you can override
  db_shared_buffers: "128MB"

  ## can improve sorting performance, but adds memory usage per-connection
  #db_work_mem: "40MB"

  ## Which Git revision should this container use? (default: tests-passed)
  #version: tests-passed

env:
  LANG: en_US.UTF-8
  # DISCOURSE_DEFAULT_LOCALE: en

  ## How many concurrent web requests are supported? Depends on memory and CPU cores.
  ## will be set automatically by bootstrap based on detected CPUs, or you can override
  UNICORN_WORKERS: 2

  ## TODO: The domain name this Discourse instance will respond to
  DISCOURSE_HOSTNAME: engineersasylum.com

  ## Uncomment if you want the container to be started with the same
  ## hostname (-h option) as specified above (default "$hostname-$config")
  #DOCKER_USE_HOSTNAME: true

  ## TODO: List of comma delimited emails that will be made admin and developer
  ## on initial signup example 'user1@example.com,user2@example.com'
  DISCOURSE_DEVELOPER_EMAILS: 'praveen369gen@gmail.com'

  ## TODO: The SMTP mail server used to validate new accounts and send notifications
  DISCOURSE_SMTP_ADDRESS: email-smtp.us-east-1.amazonaws.com
  DISCOURSE_SMTP_PORT: 587
  DISCOURSE_SMTP_USER_NAME: AXXXXXXXXXXXXXXX
  DISCOURSE_SMTP_PASSWORD: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  #DISCOURSE_SMTP_ENABLE_START_TLS: true           # (optional, default true)

  ## If you added the Lets Encrypt template, uncomment below to get a free SSL certificate
  LETSENCRYPT_ACCOUNT_EMAIL: praxxxxxx@gmail.com

  ## The CDN address for this Discourse instance (configured to pull)
  ## see https://meta.discourse.org/t/14857 for details
  #DISCOURSE_CDN_URL: //discourse-cdn.example.com
  DISCOURSE_USE_S3: false

## The Docker container is stateless; all data is stored in /shared
volumes:
  - volume:
      host: /var/discourse/shared/standalone
      guest: /shared
  - volume:
      host: /var/discourse/shared/standalone/log/var-log
      guest: /var/log

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/discourse-solved.git
          - git clone https://github.com/discourse/discourse-adplugin.git
          - git clone https://github.com/discoursehosting/discourse-sitemap.git
          - git clone https://github.com/discourse/discourse-whos-online.git
          - git clone https://github.com/discourse/discourse-push-notifications.git
          - git clone https://github.com/discourse/discourse-chat-integration.git
          - git clone https://github.com/davidtaylorhq/discourse-telegram-notifications.git
          - git clone https://github.com/discourse/discourse-knowledge-explorer.git
          - git clone https://github.com/discourse/discourse-math.git

## Any custom commands to run after building
run:
  - exec: echo "Beginning of custom commands"
  ## If you want to set the 'From' email address for your first registration, uncomment and change:
  ## After getting the first signup email, re-comment the line. It only needs to run once.
  #- exec: rails r "SiteSetting.notification_email='info@unconfigured.discourse.org'"
  - exec: echo "End of custom commands"

I don’t remember exactly, but I think I have migrated to S3 using rake uploads:migrate_to_s3

I have no clue but here’s a similar thread that has the exact same error you did:

You may want to check with a team member.

Hi @Pravi

Sidebar question:

In your post of your yml you “say”:

DISCOURSE_SMTP_PASSWORD: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

Is this correct? Is the password in double quotes in your yml?

Yes, it is in double quotes. I have not changed it since I added it two years ago.

I have removed the double quotes, then rebuild the app again. Then tried rake uploads:migrate_from_s3 again, but still no luck. Same error

Hi @Pravi

To be honest, am an anti-fan of AWS and S3 so I do not use them.

Have you considered looking at the Rake task and reverse engineering that script and simply do the task manually, step-by-step?

I don’t have that much idea of discourse internal script and I am little scared to edit the script. I don’t want to ruin everything.

Can you tell me the location of discourse.config file? I can’t find it anywhere on my server. The find command returns nothing.

root@ip-172-31-7-247:/var/discourse# find . -name "*.config"
root@ip-172-31-7-247:/var/discourse# find . -name "*.conf"
./shared/standalone/letsencrypt/engineersasylum.com/engineersasylum.com.conf
./shared/standalone/letsencrypt/engineersasylum.com/engineersasylum.com.csr.conf
./shared/standalone/letsencrypt/engineersasylum.com_ecc/engineersasylum.com.conf
./shared/standalone/letsencrypt/engineersasylum.com_ecc/engineersasylum.com.csr.conf
./shared/standalone/letsencrypt/ca/acme-v01.api.letsencrypt.org/ca.conf
./shared/standalone/letsencrypt/ca/acme-v02.api.letsencrypt.org/ca.conf
./shared/standalone/letsencrypt/account.conf
./shared/standalone/postgres_data/pg_hba.conf
./shared/standalone/postgres_data/postgresql.auto.conf
./shared/standalone/postgres_data/pg_ident.conf
./shared/standalone/postgres_data/postgresql.conf
root@ip-172-31-7-247:/var/discourse#

Hi @Pravi

Sorry that you misunderstood me.

I did not ask you to edit that file or any file.

To “Reverse engineer” a script, in my mind, means to study it and see how it works and has nothing to do with editing the file. It means to read it and understand it.

Normally if we read and understand any script or file giving us an error, we can normally get an understanding of how to fix the problem.

You are posting error messages from a file which you have not read, or so it seems.

Hope this helps.

I get your point. I will try to do it.
Can you suggest the location of the file ‘discourse.config’? Where are the configuration for the S3, stored?

Hi @Pravi

Here ya go, just for you…

That file is in the container.

You will find that file and other config files here:

# docker exec -it app bash
# cd /var/www/discourse/config

I absolutely recommend you do not exit any of those files unless you really know what are doing.

However, its definitely cool to read them and to understand them.

1 Like

But I can’t find discourse.config file in the container. I have the only app.yml in the container folder.

The container folder you are talking about is NOT the container.

The container is a Docker construct build and launched to run as a Docker container.

BTW, do not take this the wrong way, but do you understand what a Docker container is? It is perfectly OK if you don’t, BTW. Many people use Discourse daily and have not yet learned Docker container basics. It’s never too late to learn :slight_smile:

Reference:

PS: @Pravi. I provided you the code to enter the container and change to the directory to where the config file are. Did you consider following that code I posted?

PS2: The rake task you are having trouble with (migrate_from_s3) can be found here:

1 Like

I have not much idea about discourse.

Yes, I find those files.

How can I troubleshoot the problem from the rake code?

Hi @Pravi,

One approach would be to look at your log file entries immediately before your error message, and see if you can locate the line (or lines) in the rake task file; and try to isolate what is causing your issue, exactly.

Solutions are normally the easy part. The hard part is discovering what the problem is.

Could you be more specific. You mean I have to look for log file in my website before executing the rake command. I didnt get it…

Hey @Pravi

This is becoming a lesson in basic software error troubleshooting… which is OK.

You already have your logging info in your post #4 in this topic:

There is a lot of info there.

You also a the full Discourse Ruby code for this rake task, above.

My apologies, but I have a lot of tasks to do as well, so I’m just trying to show you, at a high level, how to debug and the solve your own problems the same way most of us would go about it.

Hope this helps.

I am sorry for taking too much time of yours. One last time, I tried to migrate to S3 instead of migrate from, which gives me following error even though I have provided all the details of s3 storage in my admin panel. Please take a look if you are familiar with this error.

root@ip-172-31-7-247:/var/discourse# ./launcher enter app
root@ip-172-31-7-247-app:/var/www/discourse# rake uploads:migrate_to_s3
rake aborted!
FileStore::ToS3MigrationError: Please provide the following environment variables:
  - DISCOURSE_S3_BUCKET
  - DISCOURSE_S3_REGION
  and either
  - DISCOURSE_S3_ACCESS_KEY_ID
  - DISCOURSE_S3_SECRET_ACCESS_KEY
  or
  - DISCOURSE_S3_USE_IAM_PROFILE
/var/www/discourse/lib/file_store/to_s3_migration.rb:38:in `s3_options_from_env'
/var/www/discourse/lib/tasks/uploads.rake:237:in `create_migration'
/var/www/discourse/lib/tasks/uploads.rake:245:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:224:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:68:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:78:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:222:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:218:in `block in <main>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_to_s3
(See full trace by running task with --trace)

Update:

I have added the all these DISCOURSE_S3_BUCKET, DISCOURSE_S3_REGION, DISCOURSE_S3_ACCESS_KEY_ID, DISCOURSE_S3_SECRET_ACCESS_KEY to app.yml and the above problem is solved. Now I am able to migrate to S3, but I cant migrate from S3. Its asking me to turn off S3 upload even though I have added DISCOURSE_USE_S3: false in my app.yml

How to confirm that S3 upload is turned off?

[5] pry(main)> SiteSetting.enable_s3_uploads
=> false
[6] pry(main)> GlobalSetting.use_s3
=> false
[7] pry(main)> GlobalSetting.use_s3?
=> true

Hey @Pravi

I am passing the baton to the great discussion about this, in this meta team topic:

Lots of very details discussion there.

All the best… and hope this helps.

1 Like