错误:整数超出范围

我在 Sidekiq(重试和死信列表)中经常遇到这个错误:

Jobs::HandledExceptionWrapper: 包装的 ActiveRecord::RangeError: PG::NumericValueOutOfRange: 错误:整数超出范围

我注意到出现此相同错误的作业包括:
Jobs::PostAlert
Jobs::ProcessPost
Jobs::NotifyCategoryChange

此前在此处曾对此进行过一些讨论:Feedback on the new Review Queue (2019) - #250 by markersocial

2 个赞

能否请您运行以下命令:

cd /var/discourse/
./launcher enter app
su postgres
psql
\x
\connect discourse 
SELECT id FROM notifications ORDER BY 1 DESC LIMIT 1;
\q
exit
4 个赞

谢谢 @Falco,我刚刚运行了它。结果如下:

-[ RECORD 1 ]--
id | 2147483496
4 个赞

好的,这就是著名的整数最大值问题。我们需要改用 bigint 来解决。我会查看一下这个问题。

6 个赞

目前,您的变通方法是运行以下命令:

cd /var/discourse/
./launcher enter app
su postgres
psql
\x
\connect discourse 
ALTER TABLE notifications ALTER COLUMN id SET DATA TYPE bigint
\q
exit

这是新安装系统的默认配置,但旧安装的数据类型不正确。

运行此变通方法可能较为困难,因为它会锁定表。您可能需要先降低 Web 负载。

2 个赞

感谢 @Falco@sam —— 非常感谢 :slight_smile:

关于这个变通方案,这样做应该相对安全吧?我不担心停机时间,只是怕弄坏什么东西。

它使用的是标准的单容器 app.yml。为了减轻 Web 负载,你认为在执行变通方案之前,先启用只读模式并运行 ./launcher stop app./launcher start app 应该就足够了,对吗?

这不会弄坏任何东西,最坏的情况只是会“卡住”很长时间。

2 个赞

谢谢 Sam,我已经执行了变通方案。不过,在输入以下命令后并未收到任何反馈:

ALTER TABLE notifications ALTER COLUMN id SET DATA TYPE bigint

不确定是否仍在处理中。目前,针对以下作业(在 /sidekiq 死信列表中,最近一次重试时间为“刚刚”),仍出现相同的错误:

Jobs::PostAlert
Jobs::ProcessPost
Jobs::NotifyCategoryChange

看起来您还需要运行包含 post_id 列的查询。

1 个赞

谢谢!:slight_smile:

为确保无误,请确认以下内容是否正确?

ALTER TABLE notifications ALTER COLUMN post_id SET DATA TYPE bigint

是的,这应该是安全的。一旦我们添加了官方迁移,就会支持它。

1 个赞

完美,谢谢 :slight_smile:

我运行了该操作(没有收到确认或反馈,和之前一样)。论坛没有变慢,所以不确定是否生效。目前仍然遇到相同的错误。

也许最好等官方迁移后再进行。

1 个赞

是的,看起来这在 post_alerts 表中,我们需要遍历许多表。

1 个赞

好奇你们现在对这个问题的进展如何?错误停止了吗?

最初我们曾考虑在此进行一些官方迁移,但风险远大于收益。我们发现,遇到拥有超过 2,147,483,647 条帖子的数据库的情况极为罕见。21 亿确实是一个非常大的数字。

全面增加字段大小的缺点在于存储需求会上升。

目前的状况是,我们正在考虑添加一个 rake 任务,用于在极端情况下“腾出空间”,即当您的 Discourse 表中含有(或曾含有)20 亿行数据时。

1 个赞

感谢跟进 @sam

我刚升级到 2.8.0.beta6,但仍然收到“整数超出范围”的错误。

我认为问题在于通知数量已经变得非常庞大,与帖子数量相比,这更容易达到上限。许多大型话题包含大量回复、点赞等来自不同用户的操作,会导致产生大量的通知。

运行一个 rake 任务听起来太棒了 :slight_smile:

我知道这是一个旧的帖子——

我们也在我们的设置(devforum.roblox.com)上遇到了这个问题!我们目前运行的是 v2.8.9,但很快就会更新到 3.0.1。

当用户在尝试点赞/取消点赞帖子时看到 403/500 错误时,我们注意到有些不对劲。

然后我偶然发现了这个帖子,并检查了我们的通知表:

=> SELECT id FROM notifications ORDER BY 1 DESC LIMIT 1;
     id     
------------
 2147483647
(1 row)

@sam 上面的解决方法仍然是最好的建议吗,还是自 2021 年 9 月以来对 rake 任务给予了更多考虑?

更多信息 –

在修改了 notifications.id 列后,我遇到了一个与 Jobs::PostAlert 作业分开的问题。

Job exception: 2147498514 超出范围 ActiveModel::Type::Integer 的 4 字节限制

也许我遗漏了另一个表/列?或者 Ruby 中有地方仍然期望整数数据类型?

backtrace
activemodel-6.1.6.1/lib/active_model/type/integer.rb:49:in `ensure_in_range'

activemodel-6.1.6.1/lib/active_model/type/integer.rb:28:in `serialize'

activemodel-6.1.6.1/lib/active_model/attribute.rb:56:in `value_for_database'

activemodel-6.1.6.1/lib/active_model/attribute.rb:68:in `forgetting_assignment'

activemodel-6.1.6.1/lib/active_model/attribute_set.rb:90:in `transform_values'

activemodel-6.1.6.1/lib/active_model/attribute_set.rb:90:in `map'

activemodel-6.1.6.1/lib/active_model/dirty.rb:262:in `forget_attribute_assignments'

activemodel-6.1.6.1/lib/active_model/dirty.rb:154:in `changes_applied'

activerecord-6.1.6.1/lib/active_record/attribute_methods/dirty.rb:202:in `_create_record'

activerecord-6.1.6.1/lib/active_record/callbacks.rb:461:in `block in _create_record'

activesupport-6.1.6.1/lib/active_support/callbacks.rb:106:in `run_callbacks'

activesupport-6.1.6.1/lib/active_support/callbacks.rb:824:in `_run_create_callbacks'

activerecord-6.1.6.1/lib/active_record/callbacks.rb:461:in `_create_record'

activerecord-6.1.6.1/lib/active_record/timestamp.rb:108:in `_create_record'

activerecord-6.1.6.1/lib/active_record/persistence.rb:900:in `create_or_update'

activerecord-6.1.6.1/lib/active_record/callbacks.rb:457:in `block in create_or_update'

activesupport-6.1.6.1/lib/active_support/callbacks.rb:106:in `run_callbacks'

activesupport-6.1.6.1/lib/active_support/callbacks.rb:824:in `_run_save_callbacks'

activerecord-6.1.6.1/lib/active_record/callbacks.rb:457:in `create_or_update'

activerecord-6.1.6.1/lib/active_record/timestamp.rb:126:in `create_or_update'

activerecord-6.1.6.1/lib/active_record/persistence.rb:507:in `save!'

activerecord-6.1.6.1/lib/active_record/validations.rb:53:in `save!'

activerecord-6.1.6.1/lib/active_record/transactions.rb:302:in `block in save!'

activerecord-6.1.6.1/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'

activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'

activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'

activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'

activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'

activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'

activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'

activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'

activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'

activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'

activerecord-6.1.6.1/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'

activerecord-6.1.6.1/lib/active_record/transactions.rb:302:in `save!'

activerecord-6.1.6.1/lib/active_record/suppressor.rb:48:in `save!'

/app/app/models/notification.rb:40:in `tap'

/app/app/models/notification.rb:40:in `consolidate_or_create!'

activerecord-6.1.6.1/lib/active_record/relation/delegation.rb:67:in `block in consolidate_or_create!'

activerecord-6.1.6.1/lib/active_record/relation.rb:406:in `block in scoping'

activerecord-6.1.6.1/lib/active_record/relation.rb:804:in `_scoping'

activerecord-6.1.6.1/lib/active_record/relation.rb:406:in `scoping'

activerecord-6.1.6.1/lib/active_record/associations/collection_proxy.rb:1109:in `scoping'

activerecord-6.1.6.1/lib/active_record/relation/delegation.rb:67:in `consolidate_or_create!'

/app/app/services/post_alerter.rb:496:in `create_notification'

/app/app/services/post_alerter.rb:825:in `block in notify_post_users'

/app/app/services/post_alerter.rb:838:in `block (2 levels) in each_user_in_batches'

activerecord-6.1.6.1/lib/active_record/relation/delegation.rb:88:in `each'

activerecord-6.1.6.1/lib/active_record/relation/delegation.rb:88:in `each'

/app/app/services/post_alerter.rb:838:in `block in each_user_in_batches'

/app/app/services/post_alerter.rb:837:in `each'

/app/app/services/post_alerter.rb:837:in `each_slice'

/app/app/services/post_alerter.rb:837:in `each_user_in_batches'

/app/app/services/post_alerter.rb:821:in `notify_post_users'

/app/app/services/post_alerter.rb:162:in `after_save_post'

/app/app/jobs/regular/post_alert.rb:11:in `execute'

/app/app/jobs/base.rb:232:in `block (2 levels) in perform'

/app/lib/rails_multisite/connection_management.rb:80:in `with_connection'

/app/app/jobs/base.rb:221:in `block in perform'

/app/app/jobs/base.rb:217:in `each'

/app/app/jobs/base.rb:217:in `perform'

sidekiq-6.3.1/lib/sidekiq/processor.rb:196:in `execute_job'

sidekiq-6.3.1/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'

sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:138:in `block in invoke'

/app/lib/sidekiq/pausable.rb:138:in `call'

sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:140:in `block in invoke'

sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:143:in `invoke'

sidekiq-6.3.1/lib/sidekiq/processor.rb:163:in `block in process'

sidekiq-6.3.1/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'

sidekiq-6.3.1/lib/sidekiq/job_retry.rb:112:in `local'

sidekiq-6.3.1/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'

sidekiq-6.3.1/lib/sidekiq/rails.rb:14:in `block in call'

activesupport-6.1.6.1/lib/active_support/execution_wrapper.rb:91:in `wrap'

activesupport-6.1.6.1/lib/active_support/reloader.rb:72:in `block in wrap'

activesupport-6.1.6.1/lib/active_support/execution_wrapper.rb:91:in `wrap'

activesupport-6.1.6.1/lib/active_support/reloader.rb:71:in `wrap'

sidekiq-6.3.1/lib/sidekiq/rails.rb:13:in `call'

sidekiq-6.3.1/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'

sidekiq-6.3.1/lib/sidekiq/processor.rb:257:in `stats'

sidekiq-6.3.1/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'

sidekiq-6.3.1/lib/sidekiq/job_logger.rb:13:in `call'

sidekiq-6.3.1/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'

sidekiq-6.3.1/lib/sidekiq/job_retry.rb:79:in `global'

sidekiq-6.3.1/lib/sidekiq/processor.rb:124:in `block in dispatch'

sidekiq-6.3.1/lib/sidekiq/logger.rb:11:in `with'

sidekiq-6.3.1/lib/sidekiq/job_logger.rb:33:in `prepare'

sidekiq-6.3.1/lib/sidekiq/processor.rb:123:in `dispatch'

sidekiq-6.3.1/lib/sidekiq/processor.rb:162:in `process'

sidekiq-6.3.1/lib/sidekiq/processor.rb:78:in `process_one'

sidekiq-6.3.1/lib/sidekiq/processor.rb:68:in `run'

sidekiq-6.3.1/lib/sidekiq/util.rb:43:in `watchdog'

sidekiq-6.3.1/lib/sidekiq/util.rb:52:in `block in safe_thread'

是的,这仍然是这里唯一的解决方法。我担心在核心中更改它,但我想如果我们不解决这个问题,这种情况只会发生在巨大的论坛上。
缺点是存储增加。

1 个赞

post_actions 服务(或通知流程中的其他地方)是否存在可能在运行 ALTER 后仍期望整数的用法?

我们看到对 /post_actions 的点赞/取消点赞调用出现 5xx 错误,响应为:

{"errors":["找不到请求的 URL 或资源。"],"error_type":"not_found"}

此外,还有一些与通知相关的作业失败(Jobs::BookmarkReminderNotifications、Jobs::GrantAnniversaryBadges、Jobs::PostAlert)。

我在上一条消息中添加了 PostAlert 的回溯信息;看起来 notification.rb 中的 consolidate_or_createnotification.rb 中抛出了一个关于整数限制的问题。

对我们来说,如果能恢复功能,增加存储空间并不是太大的问题 :crossed_fingers:

也许可以尝试重启容器,可能是内存中有缓存的东西。