Unable to backup or navigate to backups

merefield · November 11, 2024, 7:51am

Getting

The backup has failed.

Here’s the log:

undefined method `start_with?' for nil /var/www/discourse/app/models/site_setting.rb:172:in `use_dualstack_endpoint’
/var/www/discourse/lib/s3_helper.rb:269:in `s3_options' /var/www/discourse/lib/backup_restore/s3_backup_store.rb:14:in `initialize’
/var/www/discourse/lib/backup_restore/backup_store.rb:17:in `new'`

after a very recent rebuild to latest.

In my case S3 is hosted with a vanilla AWS bucket.

Could this be related?

hosna · November 11, 2024, 8:17am

I use minio object storage for my s3 and get this error:

rake aborted!
Aws::S3::Errors::Http504Error: Aws::S3::Errors::Http504Error (Aws::S3::Errors::Http504Error)
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/dualstack.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/accelerate.rb:43:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/response_target.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/request.rb:72:in `send_request'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/client.rb:11285:in `list_objects_v2'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/bucket.rb:1304:in `block (2 levels) in objects'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/user_agent.rb:28:in `feature'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/bucket.rb:1303:in `block in objects'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:101:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:101:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:101:in `block in non_empty_batches'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:52:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:52:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:52:in `block in each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:58:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:58:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:58:in `each'
/var/www/discourse/lib/tasks/s3.rake:14:in `map'
/var/www/discourse/lib/tasks/s3.rake:14:in `existing_assets'
/var/www/discourse/lib/tasks/s3.rake:210:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => s3:expire_missing_assets
(See full trace by running task with --trace)
I, [2024-11-11T08:02:51.942337 #1]  INFO -- : Checking for stale S3 assets...

I, [2024-11-11T08:02:51.944197 #1]  INFO -- : Terminating async processes
I, [2024-11-11T08:02:51.944334 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 39
I, [2024-11-11T08:02:51.944432 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 107
2024-11-11 08:02:51.944 UTC [39] LOG:  received fast shutdown request
107:signal-handler (1731312171) Received SIGTERM scheduling shutdown...
2024-11-11 08:02:51.945 UTC [39] LOG:  aborting any active transactions
2024-11-11 08:02:51.949 UTC [39] LOG:  background worker "logical replication launcher" (PID 54) exited with exit code 1
2024-11-11 08:02:51.950 UTC [49] LOG:  shutting down
107:M 11 Nov 2024 08:02:51.960 # User requested shutdown...
107:M 11 Nov 2024 08:02:51.960 * Saving the final RDB snapshot before exiting.
2024-11-11 08:02:52.047 UTC [39] LOG:  database system is shut down
107:M 11 Nov 2024 08:02:52.207 * DB saved on disk
107:M 11 Nov 2024 08:02:52.208 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:expire_missing_assets failed with return #<Process::Status: pid 3550 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"cd"=>"$home", "cmd"=>["sudo -E -u discourse bundle exec rake s3:upload_assets", "sudo -E -u discourse bundle exec rake s3:expire_missing_assets"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
4f601885ffad64bcd29f6dbd06df1f0f86ad301341d40e752df7447809a32eff

@martin

merefield · November 11, 2024, 11:16pm

Another issue is you can’t navigate to backups anymore:

Error while computing report `storage_stats`: undefined method `start_with?' for nil
/var/www/discourse/app/models/site_setting.rb:172:in `use_dualstack_endpoint'
/var/www/discourse/lib/s3_helper.rb:269:in `s3_options'

merefield · November 11, 2024, 11:24pm

Could this be the reason?

[7] pry(main)> SiteSetting.Upload.s3_region
=> nil
[8] pry(main)> SiteSetting.s3_region
=> "eu-west-2"
[9] pry(main)>

SiteSetting.Upload.s3_region is blank on my instance, so starts_with? is throwing an error.

github.com

discourse/discourse/blob/4816e8e0ecebe7048a0b2ea65bf52d66eabe4515/app/models/site_setting.rb#L172


      
          
          def self.enable_s3_transfer_acceleration
            if SiteSetting.enable_s3_uploads
              SiteSetting.enable_s3_transfer_acceleration
            else
              GlobalSetting.enable_s3_transfer_acceleration
            end
          end
          
          def self.use_dualstack_endpoint
            SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region.start_with?("cn-")
          end
          
          def self.enable_s3_uploads
            SiteSetting.enable_s3_uploads || GlobalSetting.use_s3?
          end
          
          def self.s3_base_url
            path = self.s3_upload_bucket.split("/", 2)[1]
            "#{self.absolute_base_url}#{path ? "/" + path : ""}"
          end

merefield · November 11, 2024, 11:30pm

In this PR:

github.com/discourse/discourse

FIX: S3 custom endpoint incompatible with dualstack

discourse:main ← discourse:issue/fix-s3-endpoint-dualstack

opened 12:41AM - 08 Nov 24 UTC

martin-brennan

+22 -5

Followup 0568d36133081e52f25f05585c1a568c3b828d79 S3 itself and other S3-comp…atible providers do not allow using an S3 custom endpoint and dualstack at the same time, so this commit fixes that by not using dualstack when the endpoint is present. c.f. https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902

The spec uses SiteSetting.s3_region, but the code uses SiteSetting.Upload.s3_region

martin · November 12, 2024, 5:56am

@hosna and @merefield , is SiteSetting.enable_s3_uploads true for both of you? If it is we should just be using SiteSetting.s3_region:

github.com

discourse/discourse/blob/f9d92fa8776134836d4fd717f11c937f6609337d/app/models/site_setting.rb#L151-L153


      
          def self.s3_region
            SiteSetting.enable_s3_uploads ? SiteSetting.s3_region : GlobalSetting.s3_region
          end

What about SiteSetting::Upload.enable_s3_uploads, what does that give you?

I could do a fix like this, using safe navigation for s3_region:

def self.use_dualstack_endpoint
  SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region&.start_with?("cn-")
end

But I want to understand why SiteSetting.Upload.s3_region is blank for you.

merefield · November 12, 2024, 6:40am

OK this gets stranger, there are two settings?

I’m using S3 for backups but not uploads (Which is a valid choice and has been the case for years? )
Enabling them both is not enforced by the interface.
I’ve not changed any settings for years.

Thus SiteSetting::Upload.s3_region is blank as I’m not using it?

Although as side note uploads should be bundled with the backups regardless, but SiteSetting.s3_region should be sufficient for that?

My backups are still failing

hosna · November 12, 2024, 11:42am

if you mean this settings, no its not enabled for me

I configured s3 in my app.yml file like this

which has been mentioned here

@martin

omarfilip · November 13, 2024, 3:03am

Just to report the same problem as mentioned above. No access to /admin/backups (error 500), backup failed.

S3 backups are enabled, S3 uploads are not - it’s been like this and successfully running for years.

Error log

### Message (14 copies reported)

Error while computing report `storage_stats`: undefined method `start_with?' for nil /var/www/discourse/app/models/site_setting.rb:172:in `use_dualstack_endpoint' /var/www/discourse/lib/s3_helper.rb:269:in `s3_options' /var/www/discourse/lib/backup_restore/s3_backup_store.rb:14:in `initialize' /var/www/discourse/lib/backup_restore/backup_store.rb:17:in `new' /var/www/discourse/lib/backup_restore/backup_store.rb:17:in `create' /var/www/discourse/app/models/concerns/reports/storage_stats.rb:10:in `report_storage_stats' /var/www/discourse/app/models/report.rb:269:in `public_send' /var/www/discourse/app/models/report.rb:269:in `block in find' /var/www/discourse/app/models/report.rb:165:in `block in wrap_slow_query' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.1.5/lib/active_record/connection_adapters/abstract/transaction.rb:535:in `block in within_new_transaction' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.1.5/lib/active_support/concurrency/null_lock.rb:9:in `synchronize' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.1.5/lib/active_record/connection_adapters/abstract/transaction.rb:532:in `within_new_transaction' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:344:in `transaction' /var/www/discourse/app/models/report.rb:160:in `wrap_slow_query' /var/www/discourse/app/models/report.rb:267:in `find' /var/www/discourse/app/controllers/admin/reports_controller.rb:97:in `block (2 levels) in bulk' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/actionpack-7.1.5/lib/action_controller/metal/strong_parameters.rb:400:in `block in each_pair' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/actionpack-7.1.5/lib/action_controller/metal/strong_parameters.rb:399:in `each_pair' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/actionpack-7.1.5/lib/action_controller/metal/strong_parameters.rb:399:in `each_pair' /var/www/discourse/app/contro...

### Backtrace

activesupport-7.1.5/lib/active_support/broadcast_logger.rb:134:in `block in error'

activesupport-7.1.5/lib/active_support/broadcast_logger.rb:231:in `block in dispatch'

activesupport-7.1.5/lib/active_support/broadcast_logger.rb:231:in `each'

activesupport-7.1.5/lib/active_support/broadcast_logger.rb:231:in `dispatch'

activesupport-7.1.5/lib/active_support/broadcast_logger.rb:134:in `error'

/var/www/discourse/app/models/report.rb:295:in `rescue in find'

/var/www/discourse/app/models/report.rb:262:in `find'

/var/www/discourse/app/controllers/admin/reports_controller.rb:97:in `block (2 levels) in bulk'

actionpack-7.1.5/lib/action_controller/metal/strong_parameters.rb:400:in `block in each_pair'

actionpack-7.1.5/lib/action_controller/metal/strong_parameters.rb:399:in `each_pair'

actionpack-7.1.5/lib/action_controller/metal/strong_parameters.rb:399:in `each_pair'

/var/www/discourse/app/controllers/admin/reports_controller.rb:76:in `block in bulk'

/var/www/discourse/lib/hijack.rb:64:in `instance_eval'

/var/www/discourse/lib/hijack.rb:64:in `block in hijack'

concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/promises.rb:911:in `callback_on_resolution'

concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/promises.rb:797:in `call_callback'

concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/promises.rb:803:in `call_callbacks'

concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/promises.rb:692:in `resolve_with'

concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/promises.rb:1325:in `resolve'

/var/www/discourse/lib/scheduler/defer.rb:115:in `block in do_work'

rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'

rails_multisite-6.1.0/lib/rails_multisite/connection_management.rb:21:in `with_connection'

/var/www/discourse/lib/scheduler/defer.rb:109:in `do_work'

/var/www/discourse/lib/scheduler/defer.rb:97:in `block (2 levels) in start_thread'

### Env

1/14

|hostname|forum-app|
| --- | --- |
|process_id|849|
|application_version|234133bd3b0750b2675a1c2c6745616c899df990|
|HTTP_HOST|forum.glasair-owners.com|
|REQUEST_METHOD|GET|
|HTTP_USER_AGENT|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36|
|HTTP_ACCEPT|application/json, text/javascript, */*; q=0.01|
|HTTP_REFERER|https://forum.glasair-owners.com/admin|
|HTTP_X_FORWARDED_FOR|xxx|
|HTTP_X_REAL_IP|xxx|
|username|xxx|
|time|6:47 pm|
||params null|

Chet_McDoniel · November 13, 2024, 3:30am

Same problem here…

pfaffman · November 14, 2024, 5:07pm

No. That’s the issue, I’m pretty sure. They are using S3 for backups, but not for uploads, which is a pretty common setup for self-hosters. It’s easy to configure S3 backups, and doesn’t require a CDN or changing the YML to upload assets to precompile.

It’s blank because they aren’t using S3 for uploads.

      SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region.start_with?("cn-")

from the github link above, should be

      GlobalSettings.use_s3 && SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region.start_with?("cn-")

or maybe

      SiteSetting.enable_s3_uploads && SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region.start_with?("cn-")

I’m not quite sure how use_s3 and enable_s3_uploads are connected.

pfaffman · November 14, 2024, 5:18pm

And I think that

github.com

discourse/discourse/blob/4816e8e0ecebe7048a0b2ea65bf52d66eabe4515/app/models/site_setting.rb#L175


      
              SiteSetting.enable_s3_transfer_acceleration
            else
              GlobalSetting.enable_s3_transfer_acceleration
            end
          end
          
          def self.use_dualstack_endpoint
            SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region.start_with?("cn-")
          end
          
          def self.enable_s3_uploads
            SiteSetting.enable_s3_uploads || GlobalSetting.use_s3?
          end
          
          def self.s3_base_url
            path = self.s3_upload_bucket.split("/", 2)[1]
            "#{self.absolute_base_url}#{path ? "/" + path : ""}"
          end
          
          def self.absolute_base_url
            url_basename = SiteSetting.s3_endpoint.split("/")[-1]

should really be def self.enable_s3_uploads? (since’s it’s reporting whether s3_uploads are enabled rather than enabling s3 uploads) and then

  SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region.start_with?("cn-")

should be

  use_s3_uploads? && SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region.start_with?("cn-")

OTOH, that’s what Sam named it 7 years ago, and it seems like it’s been just fine.

so, as the function is named now

  use_s3_uploads && SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region.start_with?("cn-")

merefield · November 14, 2024, 6:50pm

shouldn’t that be?:

use_s3_uploads? && !SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region.start_with?("cn-")

ie the s3 upload endpoint is not blank?

hosna · November 14, 2024, 7:04pm

but I also use s3 for uploads. I configured it in my app.yml.

MarcP · November 14, 2024, 7:29pm

I have the same… logged into my storage provider to check if I paid the bill, and I did.

Why would such a breaking change be pushed without any announcement?

Are we meant to enter all details in app.yml now?

Why are all settings removed from the admin backend? How can we retrieve those settings? I’ve set this up once years ago and have no clue what my key or secret is and I’m not even sure if I can retrieve this without resetting the key (which would affect other services I use the same s3 endpoints for…)

merefield · November 14, 2024, 8:12pm

I believe the settings are still there in the UI (back-up location, frequency, s3 bucket etc.), it’s just not functioning as before.

I don’t believe it’s an intended functional change, it’s just broken?

It should surely still be possible to backup to S3 without having to store uploads in S3 - these are two separate if somewhat aligned use cases …

Al_Ni · November 14, 2024, 9:36pm

Same problem
Was using S3 for backup not for uploading

pfaffman · November 14, 2024, 11:02pm

Because it wasn’t supposed to break anything, but the s3 code has to support a whole bunch of edge cases, and unless you wrote that code or paid attention to its development over the years it’s pretty hard to jump in and see just how big the problem really is.

martin · November 15, 2024, 2:06am

Sorry all, I was off sick for a couple of days. This should fix the problem:

github.com/discourse/discourse

FIX: Admin backups erroring because of S3 dualstack

discourse:main ← discourse:issue/fix-s3-backup-dualstack

opened 01:53AM - 15 Nov 24 UTC

martin-brennan

+30 -10

Followup 0568d36133081e52f25f05585c1a568c3b828d79 Followup 97cf069a061e350edfc5…310266e80e58532a39a4 Due to the S3 dualstack endpoint change, sites with S3 backups configured but _not_ S3 uploads were erroring, with admins unable to access the backups page. This commit fixes the error by not enabling S3 dualstack endpoints if S3 uploads have not been enabled, backups don't need to use them. c.f. https://meta.discourse.org/t/unable-to-backup-or-navigate-to-backups/335899

I tried this locally with S3 uploads disabled and S3 backups enabled before and after the fix, I was able to replicate the same error and then resolve it.

omarfilip · November 15, 2024, 3:42am

Rebuilt and backed up successfully.

Thanks @martin .

Topic		Replies	Views
Intended path to migrate S3 to local Installation uploads , s3	5	746	May 13, 2023
Rebuild issue: [Cannot set dual-stack in combination with a custom endpoint.] Bug s3 , fixed	15	190	November 18, 2024
Help restoring - system hung at midnight Installation	26	2203	June 13, 2022
Set up file and image uploads to S3 Self-Hosting configuring , how-to	23	116688	November 4, 2024
Forum rendered unusable Installation s3	10	779	September 4, 2023

Unable to backup or navigate to backups

Related topics