Failing after running "./launcher rebuild app" on Ubuntu 14.04

(Pedro Netto) #1

Guys, I have a perfectly vanilla Discourse installation that’s been working just fine forever, but with Mandrill’s change to paid accounts I updated my app.yml file to use MailGun instead and when I ran ./launcher rebuild app there was an error (The whole log can be found here: Discourse Bootstrap Error - From what I could understand it has to do with the postgres database, which is somehow on the way (below).

Can you help me figure this out? Right now the installation is up because I ran ./launcher start app, but this of course uses the old version (v1.5.0.beta2 +13) and not the new one (1.6.0.beta2), which is probably coming with the rebuilt app.

I’d be glad to send more information if needed! Thank you in advance.

Postgres error:

2016-04-21 17:56:42 UTC [79-1] postgres@postgres ERROR:  database "discourse" already exists
2016-04-21 17:56:42 UTC [79-2] postgres@postgres STATEMENT:  CREATE DATABASE discourse;
createdb: database creation failed: ERROR:  database "discourse" already exists
I, [2016-04-21T17:56:42.655805 #38]  INFO -- : 
I, [2016-04-21T17:56:42.656457 #38]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2016-04-21 17:56:42 UTC [90-1] postgres@discourse ERROR:  role "discourse" already exists
2016-04-21 17:56:42 UTC [90-2] postgres@discourse STATEMENT:  create user discourse;
ERROR:  role "discourse" already exists

The final Fail message:


RuntimeError: cd /var/www/discourse && su discourse -c ‘bundle exec rake db:migrate’ failed with return #<Process::Status: pid 372 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:105:in `spawn’
exec failed with the params {“cd”=>"$home", “hook”=>“bundle_exec”, “cmd”=>[“su discourse -c ‘bundle install --deployment --verbose --without test --without development’”, “su discourse -c ‘bundle exec rake db:migrate’”, “su discourse -c ‘bundle exec rake assets:precompile’”]}
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

Docker information:

ubuntu@host:/var/discourse$ docker --version
Docker version 1.8.1, build d12ea79

ubuntu@host:/var/discourse$ docker info
Containers: 1
Images: 14
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 16
Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-85-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 2
Total Memory: 3.859 GiB
Name: ip-172-31-47-61
WARNING: No swap limit support

(cpradio) #2

I’m getting the same error when trying to bootstrap a new Discourse instance. Been getting it since yesterday. I’ve got a stock app.yml, no plugins, not trying to use letsencrypt either.

(Pedro Netto) #3

Thanks for this @cpradio, it’s positive to know I’m not the only one having this error. Now let’s see if someone from the project has an opinion about this. Cheers!

(cpradio) #4

Here is my log (or at least the relevant part)

I, [2016-04-21T21:58:32.906620 #1472]  INFO -- : Writing /var/www/discourse/public/assets/admin/templates/version-checks-17ae58d766a13c49d99e3c732a07ede231fa8e78cfb66415c25ed70c43fbc714.hbs
rake aborted!
Tilt::ES6ModuleTranspilerTemplate::JavaScriptError: Script Timed Out
/var/www/discourse/lib/es6_module_transpiler/tilt/es6_module_transpiler_template.rb:111:in `block in evaluate'
/var/www/discourse/lib/es6_module_transpiler/tilt/es6_module_transpiler_template.rb:65:in `block in protect'
/var/www/discourse/lib/es6_module_transpiler/tilt/es6_module_transpiler_template.rb:63:in `synchronize'
/var/www/discourse/lib/es6_module_transpiler/tilt/es6_module_transpiler_template.rb:63:in `protect'
/var/www/discourse/lib/es6_module_transpiler/tilt/es6_module_transpiler_template.rb:109:in `evaluate'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/tilt-2.0.2/lib/tilt/template.rb:99:in `render'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/legacy_tilt_processor.rb:25:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/processor_utils.rb:75:in `call_processor'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/processor_utils.rb:57:in `block in call_processors'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/processor_utils.rb:56:in `reverse_each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/processor_utils.rb:56:in `call_processors'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/loader.rb:134:in `load_from_unloaded'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/loader.rb:60:in `block in load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/loader.rb:317:in `fetch_asset_from_dependency_cache'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/loader.rb:44:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/cached_environment.rb:20:in `block in initialize'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/cached_environment.rb:47:in `yield'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/cached_environment.rb:47:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/bundle.rb:23:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/utils.rb:183:in `dfs'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/bundle.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/processor_utils.rb:75:in `call_processor'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/processor_utils.rb:57:in `block in call_processors'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/processor_utils.rb:56:in `reverse_each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/processor_utils.rb:56:in `call_processors'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/loader.rb:134:in `load_from_unloaded'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/loader.rb:60:in `block in load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/loader.rb:317:in `fetch_asset_from_dependency_cache'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/loader.rb:44:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/cached_environment.rb:20:in `block in initialize'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/cached_environment.rb:47:in `yield'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/cached_environment.rb:47:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/base.rb:66:in `find_asset'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/base.rb:73:in `find_all_linked_assets'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/manifest.rb:142:in `block in find'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/legacy.rb:114:in `block (2 levels) in logical_paths'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/path_utils.rb:223:in `block in stat_tree'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/path_utils.rb:207:in `block in stat_directory'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/path_utils.rb:204:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/path_utils.rb:204:in `stat_directory'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/path_utils.rb:222:in `stat_tree'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/legacy.rb:105:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/legacy.rb:105:in `block in logical_paths'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/legacy.rb:104:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/legacy.rb:104:in `logical_paths'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/manifest.rb:140:in `find'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/sprockets/manifest.rb:185:in `compile'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-rails-3.0.4/lib/sprockets/rails/task.rb:68:in `block (3 levels) in define'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-3.6.0/lib/rake/sprocketstask.rb:147:in `with_logger'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sprockets-rails-3.0.4/lib/sprockets/rails/task.rb:67:in `block (2 levels) in define'
Tasks: TOP => assets:precompile
(See full trace by running task with --trace)
I, [2016-04-21T22:01:26.863875 #38]  INFO -- : Purging temp files
Bundling assets

I, [2016-04-21T22:01:27.141540 #38]  INFO -- : Terminating async processes
I, [2016-04-21T22:01:27.143985 #38]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/9.3/bin/postmaster -D /etc/postgresql/9.3/main pid: 108
I, [2016-04-21T22:01:27.161559 #38]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 237
2016-04-21 22:01:27 UTC [108-2] LOG:  received fast shutdown request
2016-04-21 22:01:27 UTC [108-3] LOG:  aborting any active transactions
2016-04-21 22:01:27 UTC [115-2] LOG:  autovacuum launcher shutting down
237:signal-handler (1461276087) Received SIGTERM scheduling shutdown...
237:M 21 Apr 22:01:27.349 # User requested shutdown...
237:M 21 Apr 22:01:27.352 * Saving the final RDB snapshot before exiting.
2016-04-21 22:01:27 UTC [112-1] LOG:  shutting down
237:M 21 Apr 22:01:29.125 * DB saved on disk
237:M 21 Apr 22:01:29.128 # Redis is now ready to exit, bye bye...
2016-04-21 22:01:30 UTC [112-2] LOG:  database system is shut down

RuntimeError: cd /var/www/discourse && su discourse -c 'bundle exec rake assets:precompile' failed with return #<Process::Status: pid 1470 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:105:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle install --deployment --verbose --without test --without development'", "su discourse -c 'bundle exec rake db:migrate'", "su discourse -c 'bundle exec rake assets:precompile'"]}
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

(cpradio) #5

@pedronetto, yours may be a bad site customization. Looking at your logs a bit closer. Mine definitely isn’t related to that though. :confused:

(Jeff Atwood) #6

Weird, I just built a $99 install yesterday and had no issues whatsoever. Let me try again now, as I have another one.

Aaaand… both rebuilt just fine, but they are 2GB RAM with 2GB swap.

There was a big Sprockets update. Could that be an issue @tgxworld?

(cpradio) #7

I just added 2 GB of SWAP to my VM which is also a 2 GB RAM, hoping that solves it. But the install doc said SWAP was optional for that condition… so I originally didn’t do that. I’ll let you know if it completes.

I’ve been trying many different combinations over the past 24 hours. SWAP is my latest alteration to see if it pans out. What is super strange is this server was running Discourse just fine last week, but I wanted to redo it from scratch again so I could play with the Lets Encrypt templates (and I haven’t been able to get far enough to try that yet).

(Jeff Atwood) #8

Swap isn’t your issue – you are running out of CPU time. That’s why the transpiler is timing out.

We saw this on Docker upgrades to 1.10 where Docker changed their image format and had to re-checksum all images, which put a big load on CPU (probably 50% of a single CPU) and this would make the transpiler time out if you rebuilt while all that was going on.

(Jeff Atwood) #9

I checked and the timeout raise still looks valid to me, I will increase it a bit

(Blake Erickson) #10

I set up a site yesterday on DO with 1GB and 1GB of swap and it worked fine.

(cpradio) #11

So the increase timeout worked, but the install is still doing something painful. I’m going to tear the server apart and rebuild it, as something isn’t quite right. My other build works flawless, so something obviously went off the hinges with this one.

(Pedro Netto) #12

Why do you think that? Something specific you noticed in my log?

(cpradio) #13

I’m not 100% sure. But for 1), I can’t get SSL to work on the new server at all. It just doesn’t honor the request at all. 2) rebuilds, entering the app, doing anything was painfully slow. Not sure what happened, probably is due to the service I’m using to host said VPS, but the other server is also twice as powerful, with twice as many cores and twice the memory.

Nonetheless, this is a hail-mary attempt now as I’ve got nothing else to try at this point.

(cpradio) #14

Oh, wait… so my prior post is really a response to what you quoted. What I saw in your log was the following:

I, [2016-04-21T17:57:48.969507 #38]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
rake aborted!
NoMethodError: undefined method `head_tag_baked=' for #<SiteCustomization:0x007fc682a1e528>

Which makes me think a site customization is blocking yours. Which means our two issues were not as related as initially thought.

You can disable site customizations using

(Pedro Netto) #15

@cpradio Oh that’s interesting, thanks for noticing! I took a look at the customizations and removed one of the two Styles I had and rebulit the app. Now I get this and after the server is stopped (and not updated, apparently). Any ideas?

ubuntu@host:/var/discourse$ sudo ./launcher rebuild app
Ensuring discourse docker is up to date
Fetching origin
Discourse Docker is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 10 app
done JP

(cpradio) #16

Others are getting that too

They said just running ./launcher start app got their instance up and running again.

(cpradio) #17

Just updating, rebuilding my VPS has solved all of the CPU issues I was experiencing. Makes me wonder if the server it got built on was over taxed.

@codinghorror thanks for helping me identify the root cause.

(Pedro Netto) #19

@codinghorror do you have any ideas on this? Thanks!

(cpradio) #20

That issue was fixed. It was related to

(Jay Pfaffman) #21

That fix for liked in on Saturday. Do a git pull and try it again.