The backup is failing again and again. Here’s the log: SSL_connect returned=1 errno=0 state=error: certificate verify failed (ok) /usr/local/lib/ruby/2.5.0/net/protocol.rb:44:in `connect_nonblock' /usr/local/lib/ruby/2.5.0/net/protocol.rb:44:in `ssl_socket_connect' /usr/local/lib/ruby/2.5.0/net/http.rb:981:in `connect' /usr/local/lib/ruby/2.5.0/net/http.rb:920:in `do_start' /usr/local/lib/ruby/2.5.0/net/http.rb:915:in `start' /usr/local/lib/ruby/2.5.0/delegate.rb:83:in `method_missing' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/net_http/connection_pool.rb:297:in `start_session' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/net_http/connection_pool.rb:96:in `session_for' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/net_http/handler.rb:121:in `session' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/net_http/handler.rb:73:in `transmit' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/net_http/handler.rb:47:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/plugins/content_length.rb:12:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/s3_signer.rb:109:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/s3_signer.rb:57:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/s3_host_id.rb:15:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/xml/error_handler.rb:8:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/s3_signer.rb:87:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/redirects.rb:18:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:171:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:202:in `retry_request' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:185:in `retry_if_possible' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:173:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:202:in `retry_request' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:185:in `retry_if_possible' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:173:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:202:in `retry_request' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:185:in `retry_if_possible' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/retry_errors.rb:173:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/dualstack.rb:34:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/accelerate.rb:50:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/md5s.rb:31:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/bucket_name_restrictions.rb:13:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/expect_100_continue.rb:22:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/bucket_dns.rb:33:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/rest/handler.rb:8:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/plugins/endpoint.rb:45:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/param_validator.rb:24:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/dualstack.rb:26:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/plugins/accelerate.rb:35:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/plugins/response_target.rb:23:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/request.rb:70:in `send_request' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/poller.rb:63:in `send_request' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/poller.rb:49:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/waiter.rb:105:in `block in poll' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/waiter.rb:102:in `loop' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/waiter.rb:102:in `poll' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/waiter.rb:92:in `block (2 levels) in wait' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/waiter.rb:91:in `catch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/waiter.rb:91:in `block in wait' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/waiter.rb:90:in `catch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/aws-sdk-core/waiters/waiter.rb:90:in `wait' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/waiters.rb:57:in `wait' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/bucket.rb:93:in `wait_until_exists' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.30.1/lib/aws-sdk-s3/bucket.rb:74:in `exists?' /var/www/discourse/lib/s3_helper.rb:252:in `s3_bucket' /var/www/discourse/lib/s3_helper.rb:171:in `list' /var/www/discourse/lib/backup_restore/s3_backup_store.rb:62:in `unsorted_files' /var/www/discourse/lib/backup_restore/backup_store.rb:21:in `files' /var/www/discourse/lib/backup_restore/backup_store.rb:26:in `latest_file' /var/www/discourse/app/jobs/scheduled/schedule_backup.rb:11:in `execute' /var/www/discourse/app/jobs/base.rb:230:in `block (2 levels) in perform' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.6/lib/rails_multisite/connection_management.rb:63:in `with_connection' /var/www/discourse/app/jobs/base.rb:219:in `block in perform' /var/www/discourse/app/jobs/base.rb:215:in `each' /var/www/discourse/app/jobs/base.rb:215:in `perform' /var/www/discourse/app/jobs/base.rb:277:in `perform' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_scheduler-0.9.1/lib/mini_scheduler/manager.rb:82:in `process_queue' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_scheduler-0.9.1/lib/mini_scheduler/manager.rb:30:in `block in initialize'

@pfaffman That commit basically broke my minio discourse s3 backup today after upgrading to the latest (2.3.0-beta6). The following resolved it after spending 2 hours fixing it: Add a DNS entry for your backup-bucket-name.backup.example.com. Add a MINIO_DOMAIN environment variable to your /etc/default/minio file.MINIO_DOMAIN="backup.example.com,backup-bucket-name.backup.example.com" In case you are using Let’s Encrypt for minio SSL certificates: Expand your SSL certificate by adding another -m backup-bucket-name.backup.example.com certbot command argument. Restart minio service after copying / symlinking your SSL certificates.systemctl restart minio

[image] hosna: SSL_connect returned=1 errno=0 state=error: certificate verify failed (ok) Something is wrong with your HTTPS.

I’m using lets encrypt for https and it seems its working fine. btw my backups is integrated with a minio server. The minio server is also https and seems to be fine too. Maybe a recent change to backup to minio script is breaking it? :thinking: Cause I have two instances and the one that is not yet updated to latest is working correctly.

The certificate presented by the Minio server isn’t trusted by the Discourse server. This is usually a missing intermediate certificate or something like that. I suggest testing the certificate of the Minio server in a service like SSL Server Test: meta.discourse.org (Powered by Qualys SSL Labs)

I cant find anything wrong with the backup server. SSL Server Test: m2.res.zabanshenas.com (Powered by Qualys SSL Labs) As I told my second instance is correctly uploading the backup to the same minio server. Any idea?

Are both Discourses running the same version?

[image] hosna: SSL_connect returned=1 errno=0 state=error: certificate verify failed (ok) /usr/local/lib/ruby/2.5.0/net/protocol.rb:44:in `connect_nonblock' /usr/local/lib/ruby/2.5.0/net/protocol.rb:44:in `ssl_socket_connect' /usr/local/lib/ruby/2.5.0/net/http.rb:981:in `connect' /usr/local/lib/ruby/2.5.0/net/http.rb:920:in `do_start' /usr/local/lib/ruby/2.5.0/net/http.rb:915:in `start' /usr/local/lib/ruby/2.5.0/delegate.rb:83:in `method_missing' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/net_http/connection_pool.rb:297:in `start_session' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.46.1/lib/seahorse/client/net_http/connection_pool.rb:96:in `session_for' This stack trace makes it looks like the connection to AWS is failing with a cert error. Do you have a decrypting corporate proxy on your network in between this server and AWS? EDIT: oh, I see, Minio is “AWS”. If you run awscli commands from inside the container against Minio, do they work or fail with a cert error?

[image] supermathie: If you run awscli commands from inside the container against Minio, do they work or fail with a cert error? can you tell me how can I do this?

Try rebuilding your Discourse container. It seems to be quite old. git pull ./launcher rebuild app You can also check if the connection to the server works by running the following in the container: openssl s_client -connect m2.res.zabanshenas.com:443 But please note that we do not test Discourse with MinIO and therefore it’s an unsupported alternative to S3.

It seems to me the output is ok cd /var/discourse [root@test discourse]# ./launcher enter app which: no docker.io in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin) root@test-app:/var/www/discourse# openssl s_client -connect m2.res.zabanshenas.com:443 CONNECTED(00000003) depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3 verify return:1 depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 verify return:1 depth=0 CN = m2.res.zabanshenas.com verify return:1 --- Certificate chain 0 s:/CN=m2.res.zabanshenas.com i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 i:/O=Digital Signature Trust Co./CN=DST Root CA X3 --- Server certificate -----BEGIN CERTIFICATE----- MIIFZTCCBE2gAwIBAgISA6f68HmW83yWzGISoVtqmNvuMA0GCSqGSIb3DQEBCwUA MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xOTAyMjYwODU3MjVaFw0x OTA1MjcwODU3MjVaMCExHzAdBgNVBAMTFm0yLnJlcy56YWJhbnNoZW5hcy5jb20w ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC9RP7NN+JRYH2MzEYry90j Luhw3n9oPw/0xx2oyd6PIoRzQH7JeXynYb26cNl9SJJu1p2UPBFmd/nIZ0duscpj 53StrQD7Rts2y6SJRIm0lvlK+HeMWy4DDkGOuGOUjgnqArIkbRNaViv/HFM/rMao IiJT9KuERLJeLxpeoRHVAvX1ByqRcQA2NTercUBt8J3bUd8Pi+lrNI4GcZKe9esw /N+XrucoQNvUQIRml+4pGjV1F8c74JWx1hp4HF65ingxLkCOlNBYt+4+/ogpWT9C 9Y6r4bnZS86ZMgYWN3WhS9baf1Vgg94meSSIIqf8riYOb+ZQVg8zk1k/oOg8oiFr AgMBAAGjggJsMIICaDAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUH AwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFByal0Ah23Q+IB25 3BqU6eB/izasMB8GA1UdIwQYMBaAFKhKamMEfd265tE5t6ZFZe/zqOyhMG8GCCsG AQUFBwEBBGMwYTAuBggrBgEFBQcwAYYiaHR0cDovL29jc3AuaW50LXgzLmxldHNl bmNyeXB0Lm9yZzAvBggrBgEFBQcwAoYjaHR0cDovL2NlcnQuaW50LXgzLmxldHNl bmNyeXB0Lm9yZy8wIQYDVR0RBBowGIIWbTIucmVzLnphYmFuc2hlbmFzLmNvbTBM BgNVHSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIB FhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQUGCisGAQQB1nkCBAIEgfYE gfMA8QB2AHR+2oMxrTMQkSGcziVPQnDCv/1eQiAIxjc1eeYQe8xWAAABaSk8z74A AAQDAEcwRQIgW4mc5hkOoIdRUgiYQ/kQ2Pka9WyCEumVZwTMNvV9wTQCIQC/nQr8 KndX2zOOPAO0P0ShbpMBpyeVtvZHAbWK+j6laQB3AGPy283oO8wszwtyhCdXazOk jWF3j711pjixx2hUS9iNAAABaSk80BoAAAQDAEgwRgIhAM2YhJPtHrT2uoNiUzsB Z4C3+qz8cMkl8xa6WMVnvT4dAiEAhcjHUQM6jhVCiCElTafpMfBTQvYrlJ7b8L+d I+KD6N8wDQYJKoZIhvcNAQELBQADggEBAJrh3u/xRcHJnjedaIt4QDZ6f2+2Dyfv fNjyDzbo3jX+Cd1Me/JKSgJxwfrcd7UBMa76RHXH0PjmhBiNbqm2sXdvyNygogJ9 ETdPnKzPoGM0UwIcJqXKkOcX4PzFnRNX+6CtvU+Crlruiwmc0mp8H858L0wcAlx2 Pn43kXcd8TbBlqp1Pbq/3hXsR0vrAZvB863M2sZ3G8ihzZ8veb5Vivf44viFNwYo u3T1/7B3kw0HCsgFWJfewVjSFz/L+NUVoo77QwiCF1+hbAHG/f5jOb+d30MUISlQ tE/nDrGEHM/1RkRq8iBk/x7JenCPlOnifbDir40HvQ6pfrFTk2sWAfs= -----END CERTIFICATE----- subject=/CN=m2.res.zabanshenas.com issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 --- No client certificate CA names sent Peer signing digest: SHA512 Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 3237 bytes and written 431 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES128-GCM-SHA256 Session-ID: 7E2165B99BEEDFB010844224B0E207E7CE734E680A5080FFDB0F98525846A922 Session-ID-ctx: Master-Key: 7958C8F9A8A6C984A16D301C2C9A26DB7C4BD2AD60893EE17C24EB9290CC4156E89AA372F121CA41D9BBA86196827CC2 Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 86400 (seconds) TLS session ticket: 0000 - e9 af 4f 02 bd b0 20 d0-35 3c 58 46 0d c8 95 2d ..O... .5<XF...- 0010 - 8e 2f 81 08 f4 38 d5 54-ed 10 ed 79 ce 61 d1 56 ./...8.T...y.a.V 0020 - 0f cf 4d 17 be 1a 9e 81-50 4f 60 fb c1 9e 76 86 ..M.....PO`...v. 0030 - c2 b8 b9 0f 1a 75 99 a4-d2 90 e3 f0 1f f0 54 7a .....u........Tz 0040 - 88 d2 d5 a2 9d 09 f4 c8-bd 04 bb 50 57 5d a8 57 ...........PW].W 0050 - 0c 25 28 ba 99 91 bf fc-1c 4a 4a 67 84 89 13 97 .%(......JJg.... 0060 - d1 a8 0c e6 2f 19 76 5d-4f ea e6 bb 02 88 e9 bc ..../.v]O....... 0070 - 25 56 f2 ad e6 0e 28 d6-47 b6 89 77 ac 59 3a af %V....(.G..w.Y:. 0080 - 6c fc 39 0f f5 6a a2 08-ad 1f 11 44 3f 6f d1 0b l.9..j.....D?o.. 0090 - 92 c3 f5 30 e9 be 47 63-d0 cd d5 b6 38 71 35 a0 ...0..Gc....8q5. 00a0 - b6 26 a0 4c d6 92 47 c5-81 9b f6 cc c7 bd 1f b3 .&.L..G......... Start Time: 1554117160 Timeout : 300 (sec) Verify return code: 0 (ok) --- closed

The error that you’re seeing doesn’t look like it but https://github.com/discourse/discourse/commit/ad6ad3f67972e74495cb424b95577e497c811b42#diff-8f55b413f98b526e5dbb98a08945de89 Removed a site setting required for minio to work. My minio backups failed too and I moved them to aws S3. @rishabh, this might be worth a look.

Backup is failing due to bad HTTPS

Product Support

azawawi (Ahmad M. Zawawi) April 1, 2019, 4:34pm 18

@hosna What about your minio version? Is it also the latest or an older one?

Topic		Replies	Views
Unable to backup to custom s3 endpoint due to https certificate error Self-hosting backups , s3	7	1286	July 30, 2020
Has anyone succeed in use minio for upload files? Product Support	7	1409	November 12, 2019
Forum rendered unusable Self-hosting s3	10	1081	September 4, 2023
Error in rebuilding using minio as object store Product Support	8	4050	September 4, 2022
Help restoring - system hung at midnight Self-hosting	26	2959	June 13, 2022

Backup is failing due to bad HTTPS

Related topics