Excon / OpenSSL problem on upgrade


(Thomas VAILLIER) #1

I have an issue when updating discourse.
My server is behind a corporate proxy which messes with SSL certificates.

Here is an extract of the log
...
I, [2017-06-07T10:05:29.997416 #14]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
2017-06-07 10:05:45.583 UTC [12635] discourse@discourse LOG:  duration: 162.566 ms  statement: set client_encoding to 'UTF8'
156:M 07 Jun 10:05:46.143 * 10 changes in 300 seconds. Saving...
156:M 07 Jun 10:05:46.146 * Background saving started by pid 12636
12636:C 07 Jun 10:05:46.617 * DB saved on disk
12636:C 07 Jun 10:05:46.619 * RDB: 18 MB of memory used by copy-on-write
156:M 07 Jun 10:05:46.648 * Background saving terminated with success
rake aborted!
Excon::Error::Certificate: SSL_connect returned=1 errno=0 state=error: certificate verify failed (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:

            `Excon.defaults[:ssl_ca_path] = path_to_certs`
            `ENV['SSL_CERT_DIR'] = path_to_certs`
            `Excon.defaults[:ssl_ca_file] = path_to_file`
            `ENV['SSL_CERT_FILE'] = path_to_file'
            `Excon.defaults[:ssl_verify_callback] = callback`
                (see OpenSSL::SSL::SSLContext#verify_callback)
or:
            `Excon.defaults[:ssl_verify_peer] = false` (less secure).
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:46:in `eval'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/ssl_socket.rb:116:in `initialize'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/connection.rb:403:in `new'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/connection.rb:403:in `socket'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/connection.rb:100:in `request_call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/middlewares/mock.rb:48:in `request_call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/middlewares/instrumentor.rb:26:in `request_call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/middlewares/base.rb:16:in `request_call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/middlewares/base.rb:16:in `request_call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/middlewares/base.rb:16:in `request_call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/connection.rb:249:in `request'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon.rb:239:in `head'
/var/www/discourse/lib/final_destination.rb:68:in `resolve'
/var/www/discourse/lib/file_helper.rb:24:in `download'
/var/www/discourse/app/models/user_avatar.rb:71:in `import_url_for_user'
(eval):21:in `block (2 levels) in run_file'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:46:in `eval'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:46:in `block (2 levels) in run_file'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:58:in `block in open'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:57:in `open'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:57:in `open'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:36:in `block in run_file'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.8/lib/active_record/transactions.rb:220:in `transaction'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:35:in `run_file'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:26:in `block in run'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:25:in `each'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:25:in `run'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu.rb:29:in `seed'
/var/www/discourse/lib/tasks/db.rake:8:in `block in <top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:22:in `load'
/usr/local/bin/bundle:22:in `<main>'
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:46:in `eval'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/ssl_socket.rb:116:in `initialize'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/connection.rb:403:in `new'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/connection.rb:403:in `socket'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/connection.rb:100:in `request_call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/middlewares/mock.rb:48:in `request_call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/middlewares/instrumentor.rb:26:in `request_call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/middlewares/base.rb:16:in `request_call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/middlewares/base.rb:16:in `request_call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/middlewares/base.rb:16:in `request_call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon/connection.rb:249:in `request'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/excon-0.55.0/lib/excon.rb:239:in `head'
/var/www/discourse/lib/final_destination.rb:68:in `resolve'
/var/www/discourse/lib/file_helper.rb:24:in `download'
/var/www/discourse/app/models/user_avatar.rb:71:in `import_url_for_user'
(eval):21:in `block (2 levels) in run_file'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:46:in `eval'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:46:in `block (2 levels) in run_file'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:58:in `block in open'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:57:in `open'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:57:in `open'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:36:in `block in run_file'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.8/lib/active_record/transactions.rb:220:in `transaction'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:35:in `run_file'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:26:in `block in run'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:25:in `each'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu/runner.rb:25:in `run'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/seed-fu-2.3.5/lib/seed-fu.rb:29:in `seed'
/var/www/discourse/lib/tasks/db.rake:8:in `block in <top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:22:in `load'
/usr/local/bin/bundle:22:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
I, [2017-06-07T10:05:56.069432 #14]  INFO -- :
== Seed from /var/www/discourse/db/fixtures/001_categories.rb

== Seed from /var/www/discourse/db/fixtures/002_groups.rb

== Seed from /var/www/discourse/db/fixtures/003_post_action_types.rb
....

Is there any way to get more details about the faulting URL, or to disable certificate validation (yeah, I know :frowning:)

Thanks!

EDIT : The same error is reported here


(Jay Pfaffman) #2

I think, then, that you can’t use Let’s Encrypt. Is that how you’re doing your certs?

If not, another possibility would be to turn off SSL in Discourse and reverse proxy to it from an external Nginx.


(Thomas VAILLIER) #3

My discourse installation is only accessible with http, no ssl involved (it’s an intranet installation)


(Rafael dos Santos Silva) #4

I had an install just like that, and you need to include the root certificate of your internal certificate authority inside the image, using one of the hooks.


(Matt Palmer) #5

You are, in a word, boned. As @Falco says, you can wedge the extra trust anchor into the install using build-time hooks, and if you’re really lucky he might even remember how he did it and save you having to figure it out. In general, though, SSL middleboxes are absolutely terrible, and you’re going to have a bad time forever and ever, Amen.


(Kevin McKinney) #6

@Falco - any chance you could walk a linux noob through that process?

The last time an issue like this came up for us, my IT department whitelisted the rubygems.org URL, and that resolved the problem. In this case it’s not clear what URL it’s even trying to resolve a certificate for.

I super agree with this. I’ve hated this ever since my company started doing it.

Story of my life.


(Thomas VAILLIER) #7

I asked the guy responsible for the proxy to whitelist cdn.discourse.org and it now works.
It still sucks, but now it’s up and running again.


Upgrade Failure - Unable to verify certificate
(Kevin McKinney) #8

We’re putting in a ticket for IT to whitelist that site, hopefully that fixes it for us too.

Out of curiosity, nothing in the log indicates anything about cdn.discourse.org so I’m just wondering how you knew to whitelist that particular URL. Just asking for future reference.


(Sam Saffron) #9

@eviltrout , I think we should add protection here in our db seeds. Just because final destination is not able to download the avatar for the user is no reason to have the seeds explode.

(also perhaps we should seed the asset from disk instead of hitting discourse)


(Matt Palmer) #11

Sherlock Holmes-grade detective work. :deerstalker: The last seed mentioned is the discobot one (plugins/discourse-narrative-bot/db/fixtures/001_discobot.rb), and it’s exploding in the import_url_for_user call, which gets passed a cdn.discourse.org URL.


(Robin Ward) #12

This should fix it until we can properly upload the avatar from a local disk. I tried that quickly but it wasn’t straightforward.


(Allen - Watchman Monitoring) #13

Would that also fix this:

?


(Jeff Atwood) #14

Might be related – @eviltrout?


(Robin Ward) #15

I can’t see that specific error in your logs, but it’s quite possible it will fix it.