自托管论坛频繁出现502和503错误

我自行托管的论坛 (https://intfiction.org/) 在过去几周开始变得非常缓慢,并且频繁出现 502 和 503 错误。

我使用的是稳定分支,版本为 2026.1.5。

在此搜索这些错误并未发现任何可能的原因:

  • 论坛构建和启动正常
  • 服务器似乎有足够的容量(13.5GB 磁盘空间可用,1.6/1.9G 内存,1.7/2.2G 交换空间)

检查 htop,负载平均值确实看起来很高:1.67, 1.55, 3.12。此进程有时使用超过 40% 的 CPU:unicorn worker[0] -E production -c config/unicorn.conf.rb

在论坛日志中,我看到大量如下错误:

Job exception: execution expired

net-smtp-0.5.1/lib/net/smtp.rb:663:in 'TCPSocket#initialize' 
net-smtp-0.5.1/lib/net/smtp.rb:663:in 'IO.open' 
net-smtp-0.5.1/lib/net/smtp.rb:663:in 'Net::SMTP#tcp_socket' 
net-smtp-0.5.1/lib/net/smtp.rb:672:in 'block in Net::SMTP#do_start' 
timeout-0.5.0/lib/timeout.rb:222:in 'block in Timeout.timeout' 
timeout-0.5.0/lib/timeout.rb:229:in 'Timeout.timeout' 
net-smtp-0.5.1/lib/net/smtp.rb:671:in 'Net::SMTP#do_start' 
net-smtp-0.5.1/lib/net/smtp.rb:642:in 'Net::SMTP#start' 
mail-2.9.0/lib/mail/network/delivery_methods/smtp.rb:154:in 'Mail::SMTP#start_smtp_session' 
mail-2.9.0/lib/mail/network/delivery_methods/smtp.rb:108:in 'Mail::SMTP#deliver!' 
mail-2.9.0/lib/mail/message.rb:269:in 'Mail::Message#deliver!' 
/usr/local/lib/ruby/3.4.0/delegate.rb:87:in 'Delegator#method_missing'
/var/www/discourse/lib/email/sender.rb:296:in 'Email::Sender#send' 
/var/www/discourse/lib/email/processor.rb:151:in 'Email::Processor#handle_failure' 
/var/www/discourse/lib/email/processor.rb:31:in 'Email::Processor#process!' 
/var/www/discourse/lib/email/processor.rb:13:in 'Email::Processor.process!' 
/var/www/discourse/app/jobs/regular/process_email.rb:8:in 'Jobs::ProcessEmail#execute' 
/var/www/discourse/app/jobs/base.rb:318:in 'block (2 levels) in Jobs::Base#perform' 
rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in 'RailsMultisite::ConnectionManagement::NullInstance#with_connection'
rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in 'RailsMultisite::ConnectionManagement.with_connection'
/var/www/discourse/app/jobs/base.rb:305:in 'block in Jobs::Base#perform' 
/var/www/discourse/app/jobs/base.rb:301:in 'Array#each' 
/var/www/discourse/app/jobs/base.rb:301:in 'Jobs::Base#perform' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:220:in 'Sidekiq::Processor#execute_job' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:185:in 'block (4 levels) in Sidekiq::Processor#process' 
sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:180:in 'Sidekiq::Middleware::Chain#traverse' 
sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in 'block in Sidekiq::Middleware::Chain#traverse' 
/var/www/discourse/lib/sidekiq/suppress_user_email_errors.rb:6:in 'Sidekiq::SuppressUserEmailErrors#call' 
sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in 'Sidekiq::Middleware::Chain#traverse' 
sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in 'block in Sidekiq::Middleware::Chain#traverse' 
/var/www/discourse/lib/sidekiq/discourse_event.rb:6:in 'Sidekiq::DiscourseEvent#call' 
sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in 'Sidekiq::Middleware::Chain#traverse' 
sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in 'block in Sidekiq::Middleware::Chain#traverse' 
/var/www/discourse/lib/sidekiq/pausable.rb:131:in 'Sidekiq::Pausable#call' 
sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in 'Sidekiq::Middleware::Chain#traverse' 
sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in 'block in Sidekiq::Middleware::Chain#traverse' 
sidekiq-7.3.9/lib/sidekiq/job/interrupt_handler.rb:9:in 'Sidekiq::Job::InterruptHandler#call' 
sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in 'Sidekiq::Middleware::Chain#traverse' 
sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in 'block in Sidekiq::Middleware::Chain#traverse' 
sidekiq-7.3.9/lib/sidekiq/metrics/tracking.rb:26:in 'Sidekiq::Metrics::ExecutionTracker#track' 
sidekiq-7.3.9/lib/sidekiq/metrics/tracking.rb:134:in 'Sidekiq::Metrics::Middleware#call' 
sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in 'Sidekiq::Middleware::Chain#traverse' 
sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:173:in 'Sidekiq::Middleware::Chain#invoke' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:184:in 'block (3 levels) in Sidekiq::Processor#process' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:145:in 'block (6 levels) in Sidekiq::Processor#dispatch' 
sidekiq-7.3.9/lib/sidekiq/job_retry.rb:118:in 'Sidekiq::JobRetry#local' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:144:in 'block (5 levels) in Sidekiq::Processor#dispatch' 
sidekiq-7.3.9/lib/sidekiq/config.rb:39:in 'block in <class:Config>' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:139:in 'block (4 levels) in Sidekiq::Processor#dispatch' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:281:in 'Sidekiq::Processor#stats' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:134:in 'block (3 levels) in Sidekiq::Processor#dispatch' 
sidekiq-7.3.9/lib/sidekiq/job_logger.rb:15:in 'Sidekiq::JobLogger#call' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:133:in 'block (2 levels) in Sidekiq::Processor#dispatch' 
sidekiq-7.3.9/lib/sidekiq/job_retry.rb:85:in 'Sidekiq::JobRetry#global' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:132:in 'block in Sidekiq::Processor#dispatch' 
sidekiq-7.3.9/lib/sidekiq/job_logger.rb:40:in 'Sidekiq::JobLogger#prepare' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:131:in 'Sidekiq::Processor#dispatch' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:183:in 'block (2 levels) in Sidekiq::Processor#process' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:182:in 'Thread.handle_interrupt' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:182:in 'block in Sidekiq::Processor#process' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:181:in 'Thread.handle_interrupt' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:181:in 'Sidekiq::Processor#process' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:86:in 'Sidekiq::Processor#process_one' 
sidekiq-7.3.9/lib/sidekiq/processor.rb:76:in 'Sidekiq::Processor#run' 
sidekiq-7.3.9/lib/sidekiq/component.rb:10:in 'Sidekiq::Component#watchdog' 
sidekiq-7.3.9/lib/sidekiq/component.rb:19:in 'block in Sidekiq::Component#safe_thread' 

我还看到许多关于 Redis 的消息:

Job exception: Connection timed out - user specified timeout: 1.0s (redis://localhost:6379)

您的 Redis 网络连接性能极差。
最近 RTT 读数为 [403941, 62224, 151840, 1536008, 3440226],理想情况下应 < 1000。
确保 Redis 运行在同一 AZ 中

但我不知道这些是显示 Redis 是原因还是只是症状。

我刚刚运行了 ./launcher rebuild app,情况并没有好转。有什么关于如何诊断此服务器上出现问题的想法吗?

我建议先稍微增加一点内存,这可能会有所帮助。

在观察 top 时又注意到了一些情况:

postgres: 15/main: discourse discourse [local] idle 占用了高达 90% 的 CPU,并且这种情况持续了一分钟以上。
编辑:论坛当时正在进行每日备份,所以我认为这是造成这种情况的原因。

我在这台服务器上运行着邮件接收服务,似乎它正遭受垃圾邮件的轮番攻击:

那些源源不断的垃圾邮件可帮不上忙。

但我同意——如果可能的话,增加内存。直接翻倍吧。