无法备份或导航到备份

merefield · 2024 年11 月 11 日 07:51

发生以下情况：

备份失败。

这是日志：

undefined method `start_with?' for nil /var/www/discourse/app/models/site_setting.rb:172:in `use_dualstack_endpoint’
/var/www/discourse/lib/s3_helper.rb:269:in `s3_options' /var/www/discourse/lib/backup_restore/s3_backup_store.rb:14:in `initialize’
/var/www/discourse/lib/backup_restore/backup_store.rb:17:in `new'`

在最近一次重建为最新版本后出现。

在我的例子中，S3托管在一个普通的AWS存储桶中。

这可能与之有关吗？

hosna · 2024 年11 月 11 日 08:17

我使用 minio 对象存储来处理我的 s3，并收到此错误：

rake aborted!
Aws::S3::Errors::Http504Error: Aws::S3::Errors::Http504Error (Aws::S3::Errors::Http504Error)
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/dualstack.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/accelerate.rb:43:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/response_target.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/request.rb:72:in `send_request'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/client.rb:11285:in `list_objects_v2'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/bucket.rb:1304:in `block (2 levels) in objects'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/user_agent.rb:28:in `feature'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/bucket.rb:1303:in `block in objects'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:101:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:101:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:101:in `block in non_empty_batches'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:52:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:52:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:52:in `block in each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:58:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:58:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/resources/collection.rb:58:in `each'
/var/www/discourse/lib/tasks/s3.rake:14:in `map'
/var/www/discourse/lib/tasks/s3.rake:14:in `existing_assets'
/var/www/discourse/lib/tasks/s3.rake:210:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => s3:expire_missing_assets
(See full trace by running task with --trace)
I, [2024-11-11T08:02:51.942337 #1]  INFO -- : Checking for stale S3 assets...

I, [2024-11-11T08:02:51.944197 #1]  INFO -- : Terminating async processes
I, [2024-11-11T08:02:51.944334 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 39
I, [2024-11-11T08:02:51.944432 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 107
2024-11-11 08:02:51.944 UTC [39] LOG:  received fast shutdown request
107:signal-handler (1731312171) Received SIGTERM scheduling shutdown...
2024-11-11 08:02:51.945 UTC [39] LOG:  aborting any active transactions
2024-11-11 08:02:51.949 UTC [39] LOG:  background worker "logical replication launcher" (PID 54) exited with exit code 1
2024-11-11 08:02:51.950 UTC [49] LOG:  shutting down
107:M 11 Nov 2024 08:02:51.960 # User requested shutdown...
107:M 11 Nov 2024 08:02:51.960 * Saving the final RDB snapshot before exiting.
2024-11-11 08:02:52.047 UTC [39] LOG:  database system is shut down
107:M 11 Nov 2024 08:02:52.207 * DB saved on disk
107:M 11 Nov 2024 08:02:52.208 # Redis is now ready to exit, bye bye...

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse &amp;&amp; sudo -E -u discourse bundle exec rake s3:expire_missing_assets failed with return #&lt;Process::Status: pid 3550 exit 1&gt;
Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"cd"=>"$home", "cmd"=>["sudo -E -u discourse bundle exec rake s3:upload_assets", "sudo -E -u discourse bundle exec rake s3:expire_missing_assets"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
4f601885ffad64bcd29f6dbd06df1f0f86ad301341d40e752df7447809a32eff

@martin

merefield · 2024 年11 月 11 日 23:16

另一个问题是您无法再导航到备份：

计算报告 `storage_stats` 时出错：对 nil 找不到方法 `start_with?'
/var/www/discourse/app/models/site_setting.rb:172:in `use_dualstack_endpoint'
/var/www/discourse/lib/s3_helper.rb:269:in `s3_options'

merefield · 2024 年11 月 11 日 23:24

这会是原因吗？

[7] pry(main)> SiteSetting.Upload.s3_region
=> nil
[8] pry(main)> SiteSetting.s3_region
=> "eu-west-2"
[9] pry(main)>

我的实例上的 SiteSetting.Upload.s3_region 为空，因此 starts_with? 抛出了错误。

github.com/discourse/discourse

app/models/site_setting.rb

4816e8e0e


      
          
          def self.enable_s3_transfer_acceleration
            if SiteSetting.enable_s3_uploads
              SiteSetting.enable_s3_transfer_acceleration
            else
              GlobalSetting.enable_s3_transfer_acceleration
            end
          end
          
          def self.use_dualstack_endpoint
            SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region.start_with?("cn-")
          end
          
          def self.enable_s3_uploads
            SiteSetting.enable_s3_uploads || GlobalSetting.use_s3?
          end
          
          def self.s3_base_url
            path = self.s3_upload_bucket.split("/", 2)[1]
            "#{self.absolute_base_url}#{path ? "/" + path : ""}"
          end

merefield · 2024 年11 月 11 日 23:30

在此 PR 中：

github.com/discourse/discourse

FIX: S3 custom endpoint incompatible with dualstack

main ← issue/fix-s3-endpoint-dualstack

opened 12:41AM - 08 Nov 24 UTC

martin-brennan

+22 -5

Followup 0568d36133081e52f25f05585c1a568c3b828d79 S3 itself and other S3-comp…atible providers do not allow using an S3 custom endpoint and dualstack at the same time, so this commit fixes that by not using dualstack when the endpoint is present. c.f. https://meta.discourse.org/t/rebuild-issue-cannot-set-dual-stack-in-combination-with-a-custom-endpoint/334902

spec 使用 SiteSetting.s3_region，但代码使用 SiteSetting.Upload.s3_region

martin · 2024 年11 月 12 日 05:56

@hosna 和 @merefield ，你们两位的 SiteSetting.enable_s3_uploads 都设置为 true 了吗？如果是的话，我们应该直接使用 SiteSetting.s3_region：

github.com/discourse/discourse

app/models/site_setting.rb

f9d92fa87


      
          def self.s3_region
            SiteSetting.enable_s3_uploads ? SiteSetting.s3_region : GlobalSetting.s3_region
          end

那 SiteSetting::Upload.enable_s3_uploads 呢，它返回什么？

我可以做一个像这样的修复，使用安全导航来处理 s3_region：

def self.use_dualstack_endpoint
  SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region&.start_with?("cn-")
end

但我想了解为什么你们的 SiteSetting.Upload.s3_region 是空的。

merefield · 2024 年11 月 12 日 06:40

好的，这变得越来越奇怪了，竟然有两个设置？

我正在使用 S3 进行备份，但不用于上传（这是一个有效的选择，而且多年来一直如此？）
界面并没有强制启用两者。
我多年来没有更改过任何设置。

因此 SiteSetting::Upload.s3_region 是空白的，因为我没有使用它？

虽然作为旁注，上传应该与备份捆绑在一起，但 SiteSetting.s3_region 应该足够了？

我的备份仍然失败

hosna · 2024 年11 月 12 日 11:42

如果你指的是这个设置，对我来说没有启用

我在我的 app.yml 文件中这样配置了 s3

这在这里有提到

@martin

omarfilip · 2024 年11 月 13 日 03:03

仅报告上述相同的问题。无法访问 /admin/backups（错误 500），备份失败。

S3 备份已启用，但 S3 上传未启用 - 这种情况已经持续多年并成功运行。

错误日志

### Message (14 copies reported)

Error while computing report `storage_stats`: undefined method `start_with?' for nil /var/www/discourse/app/models/site_setting.rb:172:in `use_dualstack_endpoint' /var/www/discourse/lib/s3_helper.rb:269:in `s3_options' /var/www/discourse/lib/backup_restore/s3_backup_store.rb:14:in `initialize' /var/www/discourse/lib/backup_restore/backup_store.rb:17:in `new' /var/www/discourse/lib/backup_restore/backup_store.rb:17:in `create' /var/www/discourse/app/models/concerns/reports/storage_stats.rb:10:in `report_storage_stats' /var/www/discourse/app/models/report.rb:269:in `public_send' /var/www/discourse/app/models/report.rb:269:in `block in find' /var/www/discourse/app/models/report.rb:165:in `block in wrap_slow_query' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.1.5/lib/active_record/connection_adapters/abstract/transaction.rb:535:in `block in within_new_transaction' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.1.5/lib/active_support/concurrency/null_lock.rb:9:in `synchronize' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.1.5/lib/active_record/connection_adapters/abstract/transaction.rb:532:in `within_new_transaction' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:344:in `transaction' /var/www/discourse/app/models/report.rb:160:in `wrap_slow_query' /var/www/discourse/app/models/report.rb:267:in `find' /var/www/discourse/app/controllers/admin/reports_controller.rb:97:in `block (2 levels) in bulk' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/actionpack-7.1.5/lib/action_controller/metal/strong_parameters.rb:400:in `block in each_pair' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/actionpack-7.1.5/lib/action_controller/metal/strong_parameters.rb:399:in `each_pair' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/actionpack-7.1.5/lib/action_controller/metal/strong_parameters.rb:399:in `each_pair' /var/www/discourse/app/contro...

### Backtrace

activesupport-7.1.5/lib/active_support/broadcast_logger.rb:134:in `block in error'

activesupport-7.1.5/lib/active_support/broadcast_logger.rb:231:in `block in dispatch'

activesupport-7.1.5/lib/active_support/broadcast_logger.rb:231:in `each'

activesupport-7.1.5/lib/active_support/broadcast_logger.rb:231:in `dispatch'

activesupport-7.1.5/lib/active_support/broadcast_logger.rb:134:in `error'

/var/www/discourse/app/models/report.rb:295:in `rescue in find'

/var/www/discourse/app/models/report.rb:262:in `find'

/var/www/discourse/app/controllers/admin/reports_controller.rb:97:in `block (2 levels) in bulk'

actionpack-7.1.5/lib/action_controller/metal/strong_parameters.rb:400:in `block in each_pair'

actionpack-7.1.5/lib/action_controller/metal/strong_parameters.rb:399:in `each_pair'

actionpack-7.1.5/lib/action_controller/metal/strong_parameters.rb:399:in `each_pair'

/var/www/discourse/app/controllers/admin/reports_controller.rb:76:in `block in bulk'

/var/www/discourse/lib/hijack.rb:64:in `instance_eval'

/var/www/discourse/lib/hijack.rb:64:in `block in hijack'

concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/promises.rb:911:in `callback_on_resolution'

concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/promises.rb:797:in `call_callback'

concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/promises.rb:803:in `call_callbacks'

concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/promises.rb:692:in `resolve_with'

concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/promises.rb:1325:in `resolve'

/var/www/discourse/lib/scheduler/defer.rb:115:in `block in do_work'

rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'

rails_multisite-6.1.0/lib/rails_multisite/connection_management.rb:21:in `with_connection'

/var/www/discourse/lib/scheduler/defer.rb:109:in `do_work'

/var/www/discourse/lib/scheduler/defer.rb:97:in `block (2 levels) in start_thread'

### Env

1/14

|hostname|forum-app|
| --- | --- |
|process_id|849|
|application_version|234133bd3b0750b2675a1c2c6745616c899df990|
|HTTP_HOST|forum.glasair-owners.com|
|REQUEST_METHOD|GET|
|HTTP_USER_AGENT|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36|
|HTTP_ACCEPT|application/json, text/javascript, */*; q=0.01|
|HTTP_REFERER|https://forum.glasair-owners.com/admin|
|HTTP_X_FORWARDED_FOR|xxx|
|HTTP_X_REAL_IP|xxx|
|username|xxx|
|time|6:47 pm|
||params null|

Chet_McDoniel · 2024 年11 月 13 日 03:30

我也是同样的问题……

pfaffman · 2024 年11 月 14 日 17:07

没有。我确信这就是问题所在。他们正在使用 S3 进行备份，但不是用于上传，这对于自托管用户来说是很常见的设置。配置 S3 备份很容易，而且不需要 CDN 或更改 YML 来将资产上传到预编译。

它是空的，因为他们没有将 S3 用于上传。

      SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region.start_with?("cn-")

来自上面的 github 链接，应该是

      GlobalSettings.use_s3 && SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region.start_with?("cn-")

或者也许

      SiteSetting.enable_s3_uploads && SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region.start_with?("cn-")

我不太确定 use_s3 和 enable_s3_uploads 是如何关联的。

pfaffman · 2024 年11 月 14 日 17:18

我认为\n\ndiscourse/app/models/site_setting.rb at 4816e8e0ecebe7048a0b2ea65bf52d66eabe4515 · discourse/discourse · GitHub def self.enable_s3_uploads?（因为它报告的是 s3_uploads 是否已启用，而不是启用 s3 uploads），然后\n\n\n SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region.start_with?(\"cn-\")\n\n\n应该改为\n\n use_s3_uploads? && SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region.start_with?(\"cn-\")\n\n另一方面，这是 Sam 7 年前命名的，而且似乎一直都很好 \n\n所以，就像函数现在命名的那样\n\n\n use_s3_uploads && SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region.start_with?(\"cn-\")\n

merefield · 2024 年11 月 14 日 18:50

难道不应该是这样的吗？：

use_s3_uploads? && !SiteSetting.Upload.s3_endpoint.blank? && !SiteSetting.Upload.s3_region.start_with?("cn-")

即 s3 上传端点不为空？

hosna · 2024 年11 月 14 日 19:04

但我也将 S3 用于上传。我在我的 app.yml 中进行了配置。

MarcP · 2024 年11 月 14 日 19:29

我也有同样的问题。登录到我的存储提供商以检查我是否支付了账单，我确实支付了。

为什么会在没有任何公告的情况下进行如此重大的更改？

我们现在应该在 app.yml 中输入所有详细信息吗？

为什么所有设置都从管理员后端移除了？我们如何找回这些设置？我几年前设置过一次，完全不知道我的密钥或密码是什么，甚至不确定我是否可以在不重置密钥的情况下找回它（这会影响我使用相同 s3 端点的其他服务……）

merefield · 2024 年11 月 14 日 20:12

我相信设置仍在 UI 中（备份位置、频率、s3 存储桶等），只是它不再像以前那样运行了。

我不认为这是一个故意的功能更改，它只是坏了？

肯定仍然应该可以在不将上传存储在 S3 的情况下备份到 S3——这两个用例虽然有些关联，但却是分开的……

Al_Ni · 2024 年11 月 14 日 21:36

同样的问题
之前使用 S3 进行备份，而不是用于上传

pfaffman · 2024 年11 月 14 日 23:02

因为它本不应该破坏任何东西，但 s3 代码必须支持大量边缘情况，除非你编写了那段代码或关注了它多年的开发，否则很难介入并看到问题的真正严重程度。

martin · 2024 年11 月 15 日 02:06

抱歉各位，我病了几天。这应该能解决问题：

github.com/discourse/discourse

FIX: Admin backups erroring because of S3 dualstack

main ← issue/fix-s3-backup-dualstack

opened 01:53AM - 15 Nov 24 UTC

martin-brennan

+30 -10

Followup 0568d36133081e52f25f05585c1a568c3b828d79 Followup 97cf069a061e350edfc5…310266e80e58532a39a4 Due to the S3 dualstack endpoint change, sites with S3 backups configured but _not_ S3 uploads were erroring, with admins unable to access the backups page. This commit fixes the error by not enabling S3 dualstack endpoints if S3 uploads have not been enabled, backups don't need to use them. c.f. https://meta.discourse.org/t/unable-to-backup-or-navigate-to-backups/335899

我在本地尝试过，在修复前后都禁用了 S3 上传并启用了 S3 备份，能够复现相同的错误然后解决它。

omarfilip · 2024 年11 月 15 日 03:42

已成功重建和备份。

谢谢 @martin。

话题		回复	浏览量
Rebuild issue: [Cannot set dual-stack in combination with a custom endpoint.] Bug s3 , fixed	15	294	2024 年11 月 18 日
Intended path to migrate S3 to local Installation uploads , s3	5	786	2023 年5 月 13 日
Forum rendered unusable Installation s3	10	821	2023 年9 月 4 日
Restore process cancelled at migrating uploads to S3 step Support	25	2536	2021 年4 月 24 日
No access to site after set S3 credentials Support	11	833	2020 年4 月 4 日

无法备份或导航到备份

相关话题