Backup is failing due to bad HTTPS

Support
1

The backup is failing again and again.

Here’s the log:

SSL_connect returned=1 errno=0 state=error: certificate verify failed (ok)
/usr/local/lib/ruby/2.5.0/net/protocol.rb:44:in `connect_nonblock'
/usr/local/lib/ruby/2.5.0/net/protocol.rb:44:in `ssl_socket_connect'
/usr/local/lib/ruby/2.5.0/net/http.rb:981:in `connect'
/usr/local/lib/ruby/2.5.0/net/http.rb:920:in `do_start'
/usr/local/lib/ruby/2.5.0/net/http.rb:915:in `start'
/usr/local/lib/ruby/2.5.0/delegate.rb:83:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/net_http/connection_pool.rb:297:in `start_session'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/net_http/connection_pool.rb:96:in `session_for'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/net_http/handler.rb:121:in `session'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/net_http/handler.rb:73:in `transmit'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/net_http/handler.rb:47:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/plugins/content_length.rb:12:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/s3_signer.rb:109:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/s3_signer.rb:57:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/s3_host_id.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/xml/error_handler.rb:8:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/s3_signer.rb:87:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/redirects.rb:18:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:171:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:202:in `retry_request'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:185:in `retry_if_possible'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:173:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:202:in `retry_request'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:185:in `retry_if_possible'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:173:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:202:in `retry_request'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:185:in `retry_if_possible'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:173:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/dualstack.rb:34:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/accelerate.rb:50:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/md5s.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/bucket_name_restrictions.rb:13:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/expect_100_continue.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/bucket_dns.rb:33:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/rest/handler.rb:8:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/plugins/endpoint.rb:45:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/param_validator.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/dualstack.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/accelerate.rb:35:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/plugins/response_target.rb:23:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/request.rb:70:in `send_request'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/poller.rb:63:in `send_request'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/poller.rb:49:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/waiter.rb:105:in `block in poll'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/waiter.rb:102:in `loop'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/waiter.rb:102:in `poll'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/waiter.rb:92:in `block (2 levels) in wait'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/waiter.rb:91:in `catch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/waiter.rb:91:in `block in wait'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/waiter.rb:90:in `catch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/waiter.rb:90:in `wait'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/waiters.rb:57:in `wait'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/bucket.rb:93:in `wait_until_exists'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/bucket.rb:74:in `exists?'
/var/www/discourse/lib/s3_helper.rb:252:in `s3_bucket'
/var/www/discourse/lib/s3_helper.rb:171:in `list'
/var/www/discourse/lib/backup_restore/s3_backup_store.rb:62:in `unsorted_files'
/var/www/discourse/lib/backup_restore/backup_store.rb:21:in `files'
/var/www/discourse/lib/backup_restore/backup_store.rb:26:in `latest_file'
/var/www/discourse/app/jobs/scheduled/schedule_backup.rb:11:in `execute'
/var/www/discourse/app/jobs/base.rb:230:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.6/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/app/jobs/base.rb:219:in `block in perform'
/var/www/discourse/app/jobs/base.rb:215:in `each'
/var/www/discourse/app/jobs/base.rb:215:in `perform'
/var/www/discourse/app/jobs/base.rb:277:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_scheduler-0.9.1/lib/mini_scheduler/manager.rb:82:in `process_queue'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_scheduler-0.9.1/lib/mini_scheduler/manager.rb:30:in `block in initialize'
1 Like
2

Something is wrong with your HTTPS.

1 Like
3

I’m using lets encrypt for https and it seems its working fine.

btw my backups is integrated with a minio server. The minio server is also https and seems to be fine too. Maybe a recent change to backup to minio script is breaking it? :thinking: Cause I have two instances and the one that is not yet updated to latest is working correctly.

4

The certificate presented by the Minio server isn’t trusted by the Discourse server. This is usually a missing intermediate certificate or something like that. I suggest testing the certificate of the Minio server in a service like SSL Server Test: meta.discourse.org (Powered by Qualys SSL Labs)

4 Likes
5

I cant find anything wrong with the backup server.

https://www.ssllabs.com/ssltest/analyze.html?d=m2.res.zabanshenas.com

As I told my second instance is correctly uploading the backup to the same minio server. Any idea?

6

Are both Discourses running the same version?

8

This stack trace makes it looks like the connection to AWS is failing with a cert error.

Do you have a decrypting corporate proxy on your network in between this server and AWS?

EDIT: oh, I see, Minio is “AWS”.

If you run awscli commands from inside the container against Minio, do they work or fail with a cert error?

10

can you tell me how can I do this?

11

Try rebuilding your Discourse container. It seems to be quite old.

git pull
./launcher rebuild app

You can also check if the connection to the server works by running the following in the container:

openssl s_client -connect m2.res.zabanshenas.com:443

But please note that we do not test Discourse with MinIO and therefore it’s an unsupported alternative to S3.

2 Likes
12

It seems to me the output is ok

cd /var/discourse
[root@test discourse]# ./launcher enter app
which: no docker.io in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
root@test-app:/var/www/discourse# openssl s_client -connect m2.res.zabanshenas.com:443
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = m2.res.zabanshenas.com
verify return:1
---
Certificate chain
 0 s:/CN=m2.res.zabanshenas.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFZTCCBE2gAwIBAgISA6f68HmW83yWzGISoVtqmNvuMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xOTAyMjYwODU3MjVaFw0x
OTA1MjcwODU3MjVaMCExHzAdBgNVBAMTFm0yLnJlcy56YWJhbnNoZW5hcy5jb20w
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC9RP7NN+JRYH2MzEYry90j
Luhw3n9oPw/0xx2oyd6PIoRzQH7JeXynYb26cNl9SJJu1p2UPBFmd/nIZ0duscpj
53StrQD7Rts2y6SJRIm0lvlK+HeMWy4DDkGOuGOUjgnqArIkbRNaViv/HFM/rMao
IiJT9KuERLJeLxpeoRHVAvX1ByqRcQA2NTercUBt8J3bUd8Pi+lrNI4GcZKe9esw
/N+XrucoQNvUQIRml+4pGjV1F8c74JWx1hp4HF65ingxLkCOlNBYt+4+/ogpWT9C
9Y6r4bnZS86ZMgYWN3WhS9baf1Vgg94meSSIIqf8riYOb+ZQVg8zk1k/oOg8oiFr
AgMBAAGjggJsMIICaDAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUH
AwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFByal0Ah23Q+IB25
3BqU6eB/izasMB8GA1UdIwQYMBaAFKhKamMEfd265tE5t6ZFZe/zqOyhMG8GCCsG
AQUFBwEBBGMwYTAuBggrBgEFBQcwAYYiaHR0cDovL29jc3AuaW50LXgzLmxldHNl
bmNyeXB0Lm9yZzAvBggrBgEFBQcwAoYjaHR0cDovL2NlcnQuaW50LXgzLmxldHNl
bmNyeXB0Lm9yZy8wIQYDVR0RBBowGIIWbTIucmVzLnphYmFuc2hlbmFzLmNvbTBM
BgNVHSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIB
FhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQUGCisGAQQB1nkCBAIEgfYE
gfMA8QB2AHR+2oMxrTMQkSGcziVPQnDCv/1eQiAIxjc1eeYQe8xWAAABaSk8z74A
AAQDAEcwRQIgW4mc5hkOoIdRUgiYQ/kQ2Pka9WyCEumVZwTMNvV9wTQCIQC/nQr8
KndX2zOOPAO0P0ShbpMBpyeVtvZHAbWK+j6laQB3AGPy283oO8wszwtyhCdXazOk
jWF3j711pjixx2hUS9iNAAABaSk80BoAAAQDAEgwRgIhAM2YhJPtHrT2uoNiUzsB
Z4C3+qz8cMkl8xa6WMVnvT4dAiEAhcjHUQM6jhVCiCElTafpMfBTQvYrlJ7b8L+d
I+KD6N8wDQYJKoZIhvcNAQELBQADggEBAJrh3u/xRcHJnjedaIt4QDZ6f2+2Dyfv
fNjyDzbo3jX+Cd1Me/JKSgJxwfrcd7UBMa76RHXH0PjmhBiNbqm2sXdvyNygogJ9
ETdPnKzPoGM0UwIcJqXKkOcX4PzFnRNX+6CtvU+Crlruiwmc0mp8H858L0wcAlx2
Pn43kXcd8TbBlqp1Pbq/3hXsR0vrAZvB863M2sZ3G8ihzZ8veb5Vivf44viFNwYo
u3T1/7B3kw0HCsgFWJfewVjSFz/L+NUVoo77QwiCF1+hbAHG/f5jOb+d30MUISlQ
tE/nDrGEHM/1RkRq8iBk/x7JenCPlOnifbDir40HvQ6pfrFTk2sWAfs=
-----END CERTIFICATE-----
subject=/CN=m2.res.zabanshenas.com
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3237 bytes and written 431 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 7E2165B99BEEDFB010844224B0E207E7CE734E680A5080FFDB0F98525846A922
    Session-ID-ctx: 
    Master-Key: 7958C8F9A8A6C984A16D301C2C9A26DB7C4BD2AD60893EE17C24EB9290CC4156E89AA372F121CA41D9BBA86196827CC2
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    0000 - e9 af 4f 02 bd b0 20 d0-35 3c 58 46 0d c8 95 2d   ..O... .5<XF...-
    0010 - 8e 2f 81 08 f4 38 d5 54-ed 10 ed 79 ce 61 d1 56   ./...8.T...y.a.V
    0020 - 0f cf 4d 17 be 1a 9e 81-50 4f 60 fb c1 9e 76 86   ..M.....PO`...v.
    0030 - c2 b8 b9 0f 1a 75 99 a4-d2 90 e3 f0 1f f0 54 7a   .....u........Tz
    0040 - 88 d2 d5 a2 9d 09 f4 c8-bd 04 bb 50 57 5d a8 57   ...........PW].W
    0050 - 0c 25 28 ba 99 91 bf fc-1c 4a 4a 67 84 89 13 97   .%(......JJg....
    0060 - d1 a8 0c e6 2f 19 76 5d-4f ea e6 bb 02 88 e9 bc   ..../.v]O.......
    0070 - 25 56 f2 ad e6 0e 28 d6-47 b6 89 77 ac 59 3a af   %V....(.G..w.Y:.
    0080 - 6c fc 39 0f f5 6a a2 08-ad 1f 11 44 3f 6f d1 0b   l.9..j.....D?o..
    0090 - 92 c3 f5 30 e9 be 47 63-d0 cd d5 b6 38 71 35 a0   ...0..Gc....8q5.
    00a0 - b6 26 a0 4c d6 92 47 c5-81 9b f6 cc c7 bd 1f b3   .&.L..G.........

    Start Time: 1554117160
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
closed
13

The error that you’re seeing doesn’t look like it but
https://github.com/discourse/discourse/commit/ad6ad3f67972e74495cb424b95577e497c811b42#diff-8f55b413f98b526e5dbb98a08945de89

Removed a site setting required for minio to work. My minio backups failed too and I moved them to aws S3.

@rishabh, this might be worth a look.

3 Likes
14

@pfaffman That commit basically broke my minio discourse s3 backup today after upgrading to the latest (2.3.0-beta6). The following resolved it after spending 2 hours fixing it:

  • Add a DNS entry for your backup-bucket-name.backup.example.com.
  • Add a MINIO_DOMAIN environment variable to your /etc/default/minio file.
    MINIO_DOMAIN="backup.example.com,backup-bucket-name.backup.example.com"
  • In case you are using Let’s Encrypt for minio SSL certificates:
    • Expand your SSL certificate by adding another -m backup-bucket-name.backup.example.com certbot command argument.
    • Restart minio service after copying / symlinking your SSL certificates.
      systemctl restart minio
5 Likes
15

can we also use the same aproach for using minio for upload files?

16

@hosna Not really sure. What is your Discourse version?

For my configuration, I only upload my discourse backups to another minio s3 server. When I upgraded to 2.3.0.beta6, Admin / Backups started giving an “Internal Server Error” message box. It turned out that it was not actually resolving the bucket-name.backup.example.com domain (DNS-style instead of path-based Bucket naming). It is now working as expected for my minio s3 configuration. :sweat: :smile:

References:

2 Likes
17

my discourse is already upgraded to v2.3.0.beta6 +22

18

@hosna What about your minio version? Is it also the latest or an older one?

1 Like
19

the minio server version is 2018-09-01T00:38:25Z

20

That is a fairly an old release and is relatively insecure (see Minio releases - critical releases). Please update it after taking a backup of your minio configuration / data folders. Updating it can be as easy as:

minio update
systemctl restart minio
2 Likes
21

Thank you very much for your help. So I suppose after update, I should be able to use minio for my upload files too and fix my problem

3 Likes
22

No problem. Hopefully it will work.

To be honest, I am surprised that you did not hit the bug that was described earlier along with the fix. Maybe it is proxy-related as was mentioned earlier.

  • Please make sure to do a backup of everything you need just in case. Hope for the best, plan for the worst :smile:.
  • Another thing I noticed Ruby 2.5.0 in the logs attached earlier and Discourse 2.3.0.beta6 is using now 2.5.3 if I remember correctly :thinking: Please correct me if I am wrong on this one.