Sidekiq errors continue to plague our deployment


(Paul Apostolos) #1

We continue to see errors in the logs and users are not receiving messages.

Here is an example of a recurring error. (It’s happening for multiple users)

exception: Infinity
context: {"retry"=>true, "queue"=>"default", "class"=>"Jobs::SendSystemMessage", "args"=>[{"user_id"=>1769, "message_type"=>"welcome_user", "current_site_id"=>"default"}], "jid"=>"3321f9cebf8f568bf96b2f4c", "enqueued_at"=>1399468690.1663618, "error_message"=>"Infinity", "error_class"=>"FloatDomainError", "failed_at"=>1399468690.3582952, "retry_count"=>6, "retried_at"=>1399470153.411549}
backtrace: /var/www/discourse/lib/text_sentinel.rb:19:in `to_i'
/var/www/discourse/lib/text_sentinel.rb:19:in `body_sentinel'
/var/www/discourse/lib/validators/post_validator.rb:31:in `raw_quality'
/var/www/discourse/lib/validators/post_validator.rb:7:in `validate'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.4/lib/active_support/callbacks.rb:283:in `_callback_before_39'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.4/lib/active_support/callbacks.rb:477:in `_run__2208640808541755120__validate__callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.4/lib/active_support/callbacks.rb:80:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activemodel-4.0.4/lib/active_model/validations.rb:373:in `run_validations!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activemodel-4.0.4/lib/active_model/validations/callbacks.rb:106:in `block in run_validations!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.4/lib/active_support/callbacks.rb:373:in `_run__2208640808541755120__validation__callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.4/lib/active_support/callbacks.rb:80:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activemodel-4.0.4/lib/active_model/validations/callbacks.rb:106:in `run_validations!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activemodel-4.0.4/lib/active_model/validations.rb:314:in `valid?'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.4/lib/active_record/validations.rb:70:in `valid?'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.4/lib/active_record/validations.rb:77:in `perform_validations'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.4/lib/active_record/validations.rb:51:in `save'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.4/lib/active_record/attribute_methods/dirty.rb:32:in `save'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.4/lib/active_record/transactions.rb:270:in `block (2 levels) in save'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.4/lib/active_record/transactions.rb:330:in `block in with_transaction_returning_status'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:211:in `transaction'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.4/lib/active_record/transactions.rb:209:in `transaction'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.4/lib/active_record/transactions.rb:327:in `with_transaction_returning_status'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.4/lib/active_record/transactions.rb:270:in `block in save'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.4/lib/active_record/transactions.rb:285:in `rollback_active_record_state!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.4/lib/active_record/transactions.rb:269:in `save'
/var/www/discourse/lib/post_creator.rb:224:in `save_post'
/var/www/discourse/lib/post_creator.rb:60:in `block in create'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:221:in `within_new_transaction'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.4/lib/active_record/transactions.rb:209:in `transaction'
/var/www/discourse/lib/post_creator.rb:56:in `create'
/var/www/discourse/lib/post_creator.rb:87:in `create'
/var/www/discourse/lib/system_message.rb:22:in `create'
/var/www/discourse/app/jobs/regular/send_system_message.rb:16:in `execute'
/var/www/discourse/app/jobs/base.rb:131:in `block (2 levels) in perform'

(Kane York) #2

@sam What if you could get Logster to be recording all errors in jobs? I can’t help but feel that a lot of the job errors go unnoticed.


(Paul Apostolos) #3

To make this long story short. The issue here is the site setting “send welcome message” was causing this failure because the SiteContent “welcome_user” is missing. (It literally is nowhere, but the method to send the message relies on it).

It’s not in this file’s add_content_type declarations

app/models/site_content.rb

(Jeff Atwood) #4

We should definitely add protection there. Probably an admin console warning too @neil.


(Sam Saffron) #5

already happening … :slight_smile: will be more tightly integrated into the admin panel as we go.


(Jeff Atwood) #6