Users reporting lots of 502 errors when attempting to post due to "max consecutive replies" check

Bug
1

Over at the Quarter to Three forum, we have a number of users that report troubles with posting to various topics. When attempting to post, they are told that there was a 502 error. It’s being discussed in this topic. I’ve tried updating the VM, rebooting it, and rebuilding Discourse multiple times. We’ve also tried closing some of the older/longer topics. However, 502 error reports continue to trickle in.

The only thing unusual that I see in the logs is multiple occurrences of:

Info:

Error while computing report `time_to_first_response`: undefined method `clear' for nil:NilClass
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_sql-0.1.10/lib/mini_sql/connection.rb:97:in `ensure in query_hash'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_sql-0.1.10/lib/mini_sql/connection.rb:97:in `query_hash'
(eval):6:in `query_hash'
/var/www/discourse/app/models/topic.rb:1252:in `time_to_first_response'
/var/www/discourse/app/models/topic.rb:1260:in `time_to_first_response_total'
/var/www/discourse/app/models/report.rb:402:in `report_time_to_first_response'
/var/www/discourse/app/models/report.rb:170:in `block in find'
/var/www/discourse/app/models/report.rb:64:in `block in wrap_slow_query'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/database_statements.rb:259:in `block in transaction'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
/usr/local/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/database_statements.rb:259:in `transaction'
/var/www/discourse/app/models/report.rb:61:in `wrap_slow_query'
/var/www/discourse/app/models/report.rb:168:in `find'
/var/www/discourse/app/controllers/admin/reports_controller.rb:40:in `block (2 levels) in bulk'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/strong_parameters.rb:338:in `block in each_pair'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/strong_parameters.rb:337:in `each_pair'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/strong_parameters.rb:337:in `each_pair'
/var/www/discourse/app/controllers/admin/reports_controller.rb:29:in `block in bulk'
/var/www/discourse/lib/hijack.rb:56:in `instance_eval'
/var/www/discourse/lib/hijack.rb:56:in `block in hijack'
/var/www/discourse/lib/scheduler/defer.rb:94:in `block in do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.4/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/lib/scheduler/defer.rb:89:in `do_work'
/var/www/discourse/lib/scheduler/defer.rb:79:in `block (2 levels) in start_thread'

backtrace:

/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-1.3.1/lib/logster/logger.rb:101:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-1.3.1/lib/logster/logger.rb:52:in `add'
/usr/local/lib/ruby/2.5.0/logger.rb:545:in `error'
/var/www/discourse/app/models/report.rb:193:in `rescue in find'
/var/www/discourse/app/models/report.rb:163:in `find'
/var/www/discourse/app/controllers/admin/reports_controller.rb:40:in `block (2 levels) in bulk'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/strong_parameters.rb:338:in `block in each_pair'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/strong_parameters.rb:337:in `each_pair'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/strong_parameters.rb:337:in `each_pair'
/var/www/discourse/app/controllers/admin/reports_controller.rb:29:in `block in bulk'
/var/www/discourse/lib/hijack.rb:56:in `instance_eval'
/var/www/discourse/lib/hijack.rb:56:in `block in hijack'
/var/www/discourse/lib/scheduler/defer.rb:94:in `block in do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.4/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/lib/scheduler/defer.rb:89:in `do_work'
/var/www/discourse/lib/scheduler/defer.rb:79:in `block (2 levels) in start_thread'

Env:

|hostname|forum-app|
|---|---|
|process_id|[21574, 13757, 8035, 26650, 19829, 21512, 9214, 21212, 14569, 1006]|
|application_version|[efcea148ebaaf1f7b593ad4db35de6803dbd0e9c, 2909e7fbdf7faecc4316e235bf8018b2d48c98a9, 7b253dbe4a92f7656bbc779d68e07ec93c56dfed]|
|HTTP_HOST|forum.quartertothree.com|
|REQUEST_URI|[/admin/reports/bulk?reports%5Btime_to_first_response%5D%5Bcache%5D=true&reports%5Btime_to_first_response%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Btime_to_first_response%5D%5Bstart_date%5D=2018-11-19T00%3A00%3A00.000Z&reports%5Btime_to_first_response%5D%5Bend_date%5D=2018-12-19T23%3A59%3A59.999Z&reports%5Blikes%5D%5Bcache%5D=true&reports%5Blikes%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Blikes%5D%5Bstart_date%5D=2018-11-19T00%3A00%3A00.000Z&reports%5Blikes%5D%5Bend_date%5D=2018-12-19T23%3A59%3A59.999Z&reports%5Bflags%5D%5Bcache%5D=true&reports%5Bflags%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bflags%5D%5Bstart_date%5D=2018-11-19T00%3A00%3A00.000Z&reports%5Bflags%5D%5Bend_date%5D=2018-12-19T23%3A59%3A59.999Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bcache%5D=true&reports%5Buser_to_user_private_messages_with_replies%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Buser_to_user_private_messages_with_replies%5D%5Bstart_date%5D=2018-11-19T00%3A00%3A00.000Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bend_date%5D=2018-12-19T23%3A59%3A59.999Z&_=1545186988758, /admin/reports/bulk?reports%5Btime_to_first_response%5D%5Bcache%5D=true&reports%5Btime_to_first_response%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Btime_to_first_response%5D%5Bstart_date%5D=2018-11-19T00%3A00%3A00.000Z&reports%5Btime_to_first_response%5D%5Bend_date%5D=2018-12-19T23%3A59%3A59.999Z&reports%5Blikes%5D%5Bcache%5D=true&reports%5Blikes%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Blikes%5D%5Bstart_date%5D=2018-11-19T00%3A00%3A00.000Z&reports%5Blikes%5D%5Bend_date%5D=2018-12-19T23%3A59%3A59.999Z&reports%5Bflags%5D%5Bcache%5D=true&reports%5Bflags%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bflags%5D%5Bstart_date%5D=2018-11-19T00%3A00%3A00.000Z&reports%5Bflags%5D%5Bend_date%5D=2018-12-19T23%3A59%3A59.999Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bcache%5D=true&reports%5Buser_to_user_private_messages_with_replies%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Buser_to_user_private_messages_with_replies%5D%5Bstart_date%5D=2018-11-19T00%3A00%3A00.000Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bend_date%5D=2018-12-19T23%3A59%3A59.999Z&_=1545195058138, /admin/reports/bulk?reports%5Btime_to_first_response%5D%5Bcache%5D=true&reports%5Btime_to_first_response%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Btime_to_first_response%5D%5Bstart_date%5D=2018-11-19T00%3A00%3A00.000Z&reports%5Btime_to_first_response%5D%5Bend_date%5D=2018-12-19T23%3A59%3A59.999Z&reports%5Blikes%5D%5Bcache%5D=true&reports%5Blikes%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Blikes%5D%5Bstart_date%5D=2018-11-19T00%3A00%3A00.000Z&reports%5Blikes%5D%5Bend_date%5D=2018-12-19T23%3A59%3A59.999Z&reports%5Bflags%5D%5Bcache%5D=true&reports%5Bflags%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bflags%5D%5Bstart_date%5D=2018-11-19T00%3A00%3A00.000Z&reports%5Bflags%5D%5Bend_date%5D=2018-12-19T23%3A59%3A59.999Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bcache%5D=true&reports%5Buser_to_user_private_messages_with_replies%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Buser_to_user_private_messages_with_replies%5D%5Bstart_date%5D=2018-11-19T00%3A00%3A00.000Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bend_date%5D=2018-12-19T23%3A59%3A59.999Z&_=1545172252827, /admin/reports/bulk?reports%5Btime_to_first_response%5D%5Bcache%5D=true&reports%5Btime_to_first_response%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Btime_to_first_response%5D%5Bstart_date%5D=2018-11-19T00%3A00%3A00.000Z&reports%5Btime_to_first_response%5D%5Bend_date%5D=2018-12-19T23%3A59%3A59.999Z&reports%5Blikes%5D%5Bcache%5D=true&reports%5Blikes%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Blikes%5D%5Bstart_date%5D=2018-11-19T00%3A00%3A00.000Z&reports%5Blikes%5D%5Bend_date%5D=2018-12-19T23%3A59%3A59.999Z&reports%5Bflags%5D%5Bcache%5D=true&reports%5Bflags%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bflags%5D%5Bstart_date%5D=2018-11-19T00%3A00%3A00.000Z&reports%5Bflags%5D%5Bend_date%5D=2018-12-19T23%3A59%3A59.999Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bcache%5D=true&reports%5Buser_to_user_private_messages_with_replies%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Buser_to_user_private_messages_with_replies%5D%5Bstart_date%5D=2018-11-19T00%3A00%3A00.000Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bend_date%5D=2018-12-19T23%3A59%3A59.999Z&_=1545244138815, /admin/reports/bulk?reports%5Btime_to_first_response%5D%5Bcache%5D=true&reports%5Btime_to_first_response%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Btime_to_first_response%5D%5Bstart_date%5D=2018-11-20T00%3A00%3A00.000Z&reports%5Btime_to_first_response%5D%5Bend_date%5D=2018-12-20T23%3A59%3A59.999Z&reports%5Blikes%5D%5Bcache%5D=true&reports%5Blikes%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Blikes%5D%5Bstart_date%5D=2018-11-20T00%3A00%3A00.000Z&reports%5Blikes%5D%5Bend_date%5D=2018-12-20T23%3A59%3A59.999Z&reports%5Bflags%5D%5Bcache%5D=true&reports%5Bflags%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bflags%5D%5Bstart_date%5D=2018-11-20T00%3A00%3A00.000Z&reports%5Bflags%5D%5Bend_date%5D=2018-12-20T23%3A59%3A59.999Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bcache%5D=true&reports%5Buser_to_user_private_messages_with_replies%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Buser_to_user_private_messages_with_replies%5D%5Bstart_date%5D=2018-11-20T00%3A00%3A00.000Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bend_date%5D=2018-12-20T23%3A59%3A59.999Z&_=1545321503751, /admin/reports/bulk?reports%5Btime_to_first_response%5D%5Bcache%5D=true&reports%5Btime_to_first_response%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Btime_to_first_response%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Btime_to_first_response%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&reports%5Blikes%5D%5Bcache%5D=true&reports%5Blikes%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Blikes%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Blikes%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&reports%5Bflags%5D%5Bcache%5D=true&reports%5Bflags%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bflags%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Bflags%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bcache%5D=true&reports%5Buser_to_user_private_messages_with_replies%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Buser_to_user_private_messages_with_replies%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&_=1545357428353, /admin/reports/bulk?reports%5Btime_to_first_response%5D%5Bcache%5D=true&reports%5Btime_to_first_response%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Btime_to_first_response%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Btime_to_first_response%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&reports%5Blikes%5D%5Bcache%5D=true&reports%5Blikes%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Blikes%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Blikes%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&reports%5Bflags%5D%5Bcache%5D=true&reports%5Bflags%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bflags%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Bflags%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bcache%5D=true&reports%5Buser_to_user_private_messages_with_replies%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Buser_to_user_private_messages_with_replies%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&_=1545368018982, /admin/reports/bulk?reports%5Btime_to_first_response%5D%5Bcache%5D=true&reports%5Btime_to_first_response%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Btime_to_first_response%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Btime_to_first_response%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&reports%5Blikes%5D%5Bcache%5D=true&reports%5Blikes%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Blikes%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Blikes%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&reports%5Bflags%5D%5Bcache%5D=true&reports%5Bflags%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bflags%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Bflags%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bcache%5D=true&reports%5Buser_to_user_private_messages_with_replies%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Buser_to_user_private_messages_with_replies%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&_=1545276710471, /admin/reports/bulk?reports%5Btime_to_first_response%5D%5Bcache%5D=true&reports%5Btime_to_first_response%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Btime_to_first_response%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Btime_to_first_response%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&reports%5Blikes%5D%5Bcache%5D=true&reports%5Blikes%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Blikes%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Blikes%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&reports%5Bflags%5D%5Bcache%5D=true&reports%5Bflags%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bflags%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Bflags%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bcache%5D=true&reports%5Buser_to_user_private_messages_with_replies%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Buser_to_user_private_messages_with_replies%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&_=1545412654073, /admin/reports/bulk?reports%5Btime_to_first_response%5D%5Bcache%5D=true&reports%5Btime_to_first_response%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Btime_to_first_response%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Btime_to_first_response%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&reports%5Blikes%5D%5Bcache%5D=true&reports%5Blikes%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Blikes%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Blikes%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&reports%5Bflags%5D%5Bcache%5D=true&reports%5Bflags%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bflags%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Bflags%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bcache%5D=true&reports%5Buser_to_user_private_messages_with_replies%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Buser_to_user_private_messages_with_replies%5D%5Bstart_date%5D=2018-11-21T00%3A00%3A00.000Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bend_date%5D=2018-12-21T23%3A59%3A59.999Z&_=1545427483602, /admin/reports/bulk?reports%5Btime_to_first_response%5D%5Bcache%5D=true&reports%5Btime_to_first_response%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Btime_to_first_response%5D%5Bstart_date%5D=2018-11-22T00%3A00%3A00.000Z&reports%5Btime_to_first_response%5D%5Bend_date%5D=2018-12-22T23%3A59%3A59.999Z&reports%5Blikes%5D%5Bcache%5D=true&reports%5Blikes%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Blikes%5D%5Bstart_date%5D=2018-11-22T00%3A00%3A00.000Z&reports%5Blikes%5D%5Bend_date%5D=2018-12-22T23%3A59%3A59.999Z&reports%5Bflags%5D%5Bcache%5D=true&reports%5Bflags%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Bflags%5D%5Bstart_date%5D=2018-11-22T00%3A00%3A00.000Z&reports%5Bflags%5D%5Bend_date%5D=2018-12-22T23%3A59%3A59.999Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bcache%5D=true&reports%5Buser_to_user_private_messages_with_replies%5D%5Bfacets%5D%5B%5D=prev_period&reports%5Buser_to_user_private_messages_with_replies%5D%5Bstart_date%5D=2018-11-22T00%3A00%3A00.000Z&reports%5Buser_to_user_private_messages_with_replies%5D%5Bend_date%5D=2018-12-22T23%3A59%3A59.999Z&_=1545511108112]|
|REQUEST_METHOD|GET|
|HTTP_USER_AGENT|[Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; LCJB; rv:11.0) like Gecko, Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko, Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:65.0) Gecko/20100101 Firefox/65.0, Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1, Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0, Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36, Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.2 Safari/605.1.15]|
|HTTP_ACCEPT|application/json, text/javascript, */*; q=0.01|
|HTTP_REFERER|https://forum.quartertothree.com/admin|
|HTTP_X_FORWARDED_FOR|[173.172.34.176, 172.69.68.230, 173.172.34.176, 172.69.69.45, 74.201.132.19, 108.162.219.117, 2606:a000:1124:c42d:d829:8b46:b05f:3e55, 172.69.62.105, 173.172.34.176, 172.69.68.56, 71.183.193.192, 162.158.63.231, 74.201.132.19, 162.158.63.237, 2606:a000:1124:c42d:9152:df85:1097:dbaf, 162.158.79.40, 2606:a000:1124:c42d:c4c5:a768:826e:4073, 172.69.62.69]|
|HTTP_X_REAL_IP|[172.69.68.230, 172.69.69.45, 108.162.219.117, 172.69.62.105, 172.69.68.56, 162.158.63.231, 162.158.63.237, 162.158.79.40, 172.69.62.69]|
|username|[redacted]|
|params|null|
|_|[1545186988758, 1545195058138, 1545172252827, 1545244138815, 1545321503751, 1545357428353, 1545368018982, 1545276710471, 1545412654073, 1545427483602, 1545511108112]|

Any ideas what’s going on or how we can address the issue? I ran ./discourse-doctor and all looks fine… Again, this is a recent problem. It seems to have started with an update performed in the last week or two.

2

Error while computing report time_to_first_response

Looks like there may be migrated data that is now breaking this report, somehow? Everything you posted is about that report.

1 Like
3

Are you still seeing this?

2 Likes
4

Yes, unfortunately.

(I mean the 502 errors reported by users, not the problem appearing in the log dump in the original post.)

We’re also seeing a lot of ember errors now.

5

It could be the server hardware is underpowered? What does htop show for cpu usage? What do disk speed throughput tests show? Is swap churning due to memory pressure?

3 Likes
6

6 vCPUs, usage varies between 10 and 30% except when rebakes are running.

No active swapping. In fact, I can’t seem to make Discourse use all the RAM we do have. 16GB on the VM and we still have 11.5G in filecache and 850M available, even after I increased DB buffers to 10GB and db_work_mem to 60M.

Disk is a SSD, uncached read speeds 640MB/sec, uncached writes 817MB/sec.

Also <1% IOwait.

Do not believe it’s an issue with the VM. It’s not stressed, and no signs over being oversold.

2 Likes
7

My only other guess is that the database is large enough in this case to cause Postgres special pain, perhaps on some obscure background job. The backtrace above does seem to mention one specific job a lot.

If this was the case, we would typically see this on our hosting as well, though. Especially if you have ruled out any slow disk or ram swapping issues.

3 Likes
8

Also … is this a shared VM? Is there invisible shared tenancy fighting for resources here that would affect disk perf, memory, etc? We call this “noisy neighbor” in cloud terms, and it does happen.

9

Like I said, there’s no signs they oversold the VM. I can grab a ton of CPU when I need to, the disks are fast, no iowait, and no real system load. Does not appear to be constrained by hardware resources.

10

It’s tough because nobody else is reporting this (that I know of) and we don’t see it on our infra, but our infra is bare metal colocation and very fast.

11

It’s really a very fast VM, and the hardware does not appear to be a constraint by any measure I can see, but it is indeed not bare-metal.

12

You could try running some of these benchmarks and report back what you get?

13

Sure. Didn’t rebuild the app as it causes an outage, but here’s the other ones. All seem pretty reasonable. Remember these were all run without taking down Discourse, so the machine was somewhat loaded.

sysbench cpu --cpu-max-prime=20000 run
3606

sysbench cpu --cpu-max-prime=40000 --num-threads=8 run
6710

ioping -RD -w 10 .
9.86 k iops, 38.5 MiB/s

dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
830 MB/s

hdparm -Tt /dev/sda
 Timing cached reads:   15936 MB in  1.99 seconds = 7991.03 MB/sec
 Timing buffered disk reads: 1820 MB in  3.00 seconds = 606.30 MB/sec
2 Likes
14

Yes, those seem like reasonable numbers. My gut says that Postgres is somehow suffering in an unusual way, given the large database … it would be a good idea to familiarize yourself with Postgres slow query and load diagnosis type tooling?

Also, how many unicorns (e.g. concurrent web requests) are you running?

15

We’re at 10 unicorns. I tried increasing that to better use our resources also.

16

You might have too many web workers (unicorns), if the point of contention is the database as I suspect it could be?

17

OK, I thought 2 per CPU was the upper limit, and we have 6 vCPUs. I’ll try dropping it from 10 down to 6.

Edit: Done, and rebuild time was 7m38s.

Edit2: Did not fix the issue, still getting 502s.

18

Do you see anything in the rails logs or the javascript console?

1 Like
19

Look for the 502s in the nginx logs, what are some example errors? Is there anything in our discourse logs that correlates?

My guess is that something is locking up and timing out. Can you confirm you are on latest?

5 Likes
20

I’ll take a look tonight or tomorrow - just back from vacation and trying to catch up on some work! Thanks for the help.

5 Likes