Discourse+Docker+AWS - "bootstrap" talks to DB fine, "run" doesn't

Installation
1

Howdy!

I’m running Discourse using discourse docker on EC2 using RDS as my database. My app.yml contains all of the proper DISCOURSE_DB_* settings, and sure enough, when I ran launcher bootstrap app, everything worked just fine. However, when I run “launcher run app”, I was getting 502’s hitting port 80. So I connected to the container using docker exec -ti app bash - env shows all of the proper environment variables being set, but the unicorn.stderr.log is filled with:

I, [2015-04-11T23:11:12.197216 #3216]  INFO -- : Refreshing Gem list
URGENT: FATAL:  password authentication failed for user "discourse"
FATAL:  password authentication failed for user "discourse"
 Failed to initialize site default
I, [2015-04-11T23:11:18.399669 #3216]  INFO -- : listening on addr=0.0.0.0:3000 fd=9
E, [2015-04-11T23:11:18.498142 #3216] ERROR -- : FATAL:  password authentication failed for user "discourse"
FATAL:  password authentication failed for user "discourse"
 (PG::ConnectionBad)
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/postgresql_adapter.rb:888:in `initialize'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/postgresql_adapter.rb:888:in `new'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/postgresql_adapter.rb:888:in `connect'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/postgresql_adapter.rb:568:in `initialize'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/postgresql_adapter.rb:41:in `new'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/postgresql_adapter.rb:41:in `postgresql_connection'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:435:in `new_connection'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:445:in `checkout_new_connection'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `acquire_connection'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:351:in `block in checkout'
/usr/local/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:350:in `checkout'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:265:in `block in connection'
/usr/local/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:264:in `connection'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:541:in `retrieve_connection'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.9/lib/active_record/connection_handling.rb:113:in `retrieve_connection'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.9/lib/active_record/connection_handling.rb:87:in `connection'
config/unicorn.conf.rb:47:in `block in reload'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:519:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:519:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:140:in `start'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'

To be 100% positive, I ran psql from inside the container with $DISCOURSE_DB_HOST, $DISCOURSE_DB_USERNAME, and $DISCOURSE_DB_NAME as arguments, pasted the contented of $DISCOURSE_DB_PASSWORD into the prompt, and connected without issue.

Any thoughts or assistance would be invaluable. Thanks!

-jag

2

Solved - though uncomfortably.

My DB password ended with a # character. I can see it in the environment variable. It doesn’t show in Rails.application.config.database_configuration.

Obscure, right?

1 Like
3

You might as well win the lottery or get struck by lightening. :wat:

Glad you got it fixed!

4

I know, right? Weird enough to write a blog post about: Head slamming against wall kind of bug | Joshua "jag" Ginsberg

2 Likes
5

I wonder, tho, shouldn’t this be considered a legitimate bug?

It looks like the config values aren’t being quoted at some point while being handed into the container so that the # gets interpreted as starting a comment…

6

That’s a YAML comment, the fix is to do this:

  key: "value#"
4 Likes
7

The value goes from a YAML file from discourse_docker (which is in my control and is quoted and therefore correctly goes) to an environment variable inside the container. The environment shows the correct value. From that point it goes to Ruby (not in my control), and judging by the comments in config/application.yml about the production rails environment, probably not via YAML again. Regardless, I did my end right - this is a bug, and clearly fixable as evidenced by the fact that rake db:migrate was still able to work. :smile:

8

For what it’s worth, I think I was bit by this issue as well.

I had the following in app.yml

DISCOURSE_DB_PASSWORD: ‘pn4&?.%d5=6#^4*f=;’

After changing the password to something more reasonable I was able to complete the seeding step and launch the app.