Sidekiq down probably because getting PG UniqueViolation error


(Kevin Yang) #1

Sidekiq is showing ‘down’ in dashboard panel. and i check the logs, i think it might related to following errors. Can someone tell me how to debug that error and make sidekiq back.

PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_application_requests_on_date_and_req_type"
DETAIL:  Key (date, req_type)=(2015-12-08, 0) already exists.
: INSERT INTO "application_requests" ("date", "req_type") VALUES ('2015-12-08', 0) RETURNING "id"
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-1.0.1/lib/logster/logger.rb:74:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-1.0.1/lib/logster/logger.rb:35:in `add'
/usr/local/lib/ruby/2.0.0/logger.rb:454:in `error'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/connection_adapters/abstract_adapter.rb:460:in `translate_exception_class'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/connection_adapters/abstract_adapter.rb:475:in `rescue in log'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/connection_adapters/abstract_adapter.rb:473:in `log'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/connection_adapters/postgresql_adapter.rb:596:in `exec_no_cache'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/connection_adapters/postgresql_adapter.rb:588:in `execute_and_clear'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:192:in `exec_insert'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/connection_adapters/abstract/database_statements.rb:108:in `insert'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/connection_adapters/abstract/query_cache.rb:14:in `insert'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/relation.rb:64:in `insert'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/persistence.rb:524:in `_create_record'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/counter_cache.rb:139:in `_create_record'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/locking/optimistic.rb:75:in `_create_record'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/attribute_methods/dirty.rb:132:in `_create_record'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/callbacks.rb:306:in `block in _create_record'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/callbacks.rb:778:in `_run_create_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/callbacks.rb:306:in `_create_record'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/timestamp.rb:57:in `_create_record'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/persistence.rb:504:in `create_or_update'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/callbacks.rb:302:in `block in create_or_update'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/callbacks.rb:778:in `_run_save_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/callbacks.rb:302:in `create_or_update'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/persistence.rb:142:in `save!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/validations.rb:43:in `save!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/attribute_methods/dirty.rb:29:in `save!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/transactions.rb:291:in `block in save!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/transactions.rb:220:in `transaction'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/transactions.rb:348:in `with_transaction_returning_status'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/transactions.rb:291:in `save!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/persistence.rb:51:in `create!'
/var/www/discourse/app/models/application_request.rb:93:in `req_id'
/var/www/discourse/app/models/application_request.rb:66:in `block in write_cache!'
/var/www/discourse/app/models/application_request.rb:52:in `each'
/var/www/discourse/app/models/application_request.rb:52:in `write_cache!'
/var/www/discourse/app/models/application_request.rb:41:in `write_cache!'
/var/www/discourse/app/models/application_request.rb:35:in `increment!'
/var/www/discourse/lib/middleware/request_tracker.rb:31:in `log_request'
/var/www/discourse/lib/middleware/request_tracker.rb:11:in `block in log_request_on_site'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rails_multisite-1.0.3/lib/rails_multisite/connection_management.rb:44:in `with_hostname'
/var/www/discourse/lib/middleware/request_tracker.rb:10:in `log_request_on_site'
/var/www/discourse/lib/middleware/request_tracker.rb:88:in `block in log_later'
/var/www/discourse/lib/scheduler/defer.rb:68:in `call'
/var/www/discourse/lib/scheduler/defer.rb:68:in `do_work'
/var/www/discourse/lib/scheduler/defer.rb:46:in `do_all_work'
/var/www/discourse/lib/scheduler/defer.rb:86:in `process_client'
/var/www/discourse/lib/middleware/unicorn_oobgc.rb:95:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:674:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:529:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:540:in `maintain_worker_count'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:294:in `join'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.9.0/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'

(Sam Saffron) #2

The error seems totally unrelated.

Are you running latest? When did you last rebuild your container? How much free memory do you have? How much free diskspace do you have?


(Hans Erik Lange) #3

I have the same error, and another, possibly unrelated.

Backtrace for the sidekiq error:

/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-1.0.1/lib/logster/logger.rb:74:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-1.0.1/lib/logster/logger.rb:35:in `add'
/usr/local/lib/ruby/2.0.0/logger.rb:445:in `warn'
config/unicorn.conf.rb:129:in `check_sidekiq_heartbeat'
config/unicorn.conf.rb:146:in `master_sleep'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:295:in `join'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.9.0/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'

Backtrace for the job exception:

/var/www/discourse/lib/scheduler/schedule_info.rb:79:in `schedule!'
/var/www/discourse/lib/scheduler/manager.rb:221:in `schedule_next_job'
/var/www/discourse/lib/scheduler/manager.rb:199:in `block in tick'
/var/www/discourse/lib/scheduler/manager.rb:246:in `block in lock'
/var/www/discourse/lib/distributed_mutex.rb:21:in `synchronize'
/var/www/discourse/lib/scheduler/manager.rb:245:in `lock'
/var/www/discourse/lib/scheduler/manager.rb:198:in `tick'
/var/www/discourse/config/initializers/100-sidekiq.rb:35:in `block (2 levels) in <top (required)>'

My version is at v1.5.0.beta7 +39, updated today after I saw the error. The container was rebuildt a week or two ago after I upgraded the host (Ubuntu 14.04 LTS) with the latest patches.

I assume that since the Sidekiq jobs is not running, it won’t pick up replies by email?

ETA: Stats on the host.
Memory:

MemTotal:        3525408 kB 
MemFree:          779556 kB

Disk:

Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda1       30202916 25974840   2962044  90% /

(Kevin Yang) #4

@sam, my latest version is v1.5.0.beta6 +101, i just upgrade to v1.5.0.beta7 +44, and sidekiq is running ok now. Sorry to interrupt.


(Kevin Yang) #5

It seems the follow command will stop sidekiq starting up.

./launcher destroy app
./launcher start app

after i upgrade, sidekiq is back, but when i change smtp setting and run commands above, sidekiq down agaign. even after i restart app again.

rebuild app will work.


(Hans Erik Lange) #6

With great help from @zogstrip I did the following:

./launcher stop app
./launcher rebuild app
./launcher enter app
redis-cli flushall

and now, some 15 minutes later, no errors. :clap: