After seeing Discourse exhibit some odd behavior, we’ve been watching it closer lately. Noticed 3 errors in the logs, hoping someone might be able to shed some light on them. #1 (Error), 4 occurrences: Message (4 copies reported) ActiveRecord::StatementInvalid (PG::UndefinedColumn: ERROR: column topics.bookmark_count does not exist LINE 1: ..., "topics"."highest_staff_post_number" AS t0_r48, "topics"."... ^ : SELECT "topics"."id" AS t0_r0, "topics"."title" AS t0_r1, "topics"."last_posted_at" AS t0_r2, "topics"."created_at" AS t0_r3, "topics"."updated_at" AS t0_r4, "topics"."views" AS t0_r5, "topics"."posts_count" AS t0_r6, "topics"."user_id" AS t0_r7, "topics"."last_post_user_id" AS t0_r8, "topics"."reply_count" AS t0_r9, "topics"."featured_user1_id" AS t0_r10, "topics"."featured_user2_id" AS t0_r11, "topics"."featured_user3_id" AS t0_r12, "topics"."avg_time" AS t0_r13, "topics"."deleted_at" AS t0_r14, "topics"."highest_post_number" AS t0_r15, "topics"."image_url" AS t0_r16, "topics"."like_count" AS t0_r17, "topics"."incoming_link_count" AS t0_r18, "topics"."category_id" AS t0_r19, "topics"."visible" AS t0_r20, "topics"."moderator_posts_count" AS t0_r21, "topics"."closed" AS t0_r22, "topics"."archived" AS t0_r23, "topics"."bumped_at" AS t0_r24, "topics"."has_summary" AS t0_r25, "topics"."vote_count" AS t0_r26, "topics"."archetype" AS t0_r27, "topics"."featured_user4_id" AS t0_r28, "topics"."notify_moderators_count" AS t0_r29, "topics"."spam_count" AS t0_r30, "topics"."pinned_at" AS t0_r31, "topics"."score" AS t0_r32, "topics"."percent_rank" AS t0_r33, "topics"."subtype" AS t0_r34, "topics"."slug" AS t0_r35, "topics"."auto_close_at" AS t0_r36, "topics"."auto_close_user_id" AS t0_r37, "topics"."auto_close_started_at" AS t0_r38, "topics"."deleted_by_id" AS t0_r39, "topics"."participant_count" AS t0_r40, "topics"."word_count" AS t0_r41, "topics"."excerpt" AS t0_r42, "topics"."pinned_globally" AS t0_r43, "topics"."auto_close_based_on_last_post" AS t0_r44, "topics"."auto_close_hours" AS t0_r45, "topics"."pinned_until" AS t0_r46, "topics"."fancy_title" AS t0_r47, "topics"."highest_staff_post_number" AS t0_r48, "topics"."bookmark_count" AS t0_r49, "topics"."off_topic_count" AS t0_r50, "topics"."illegal_count" AS t0_r51, "topics"."inappropriate_count" AS t0_r52, "topics"."notify_user_count" AS t0_r53, "topics"."featured_link" AS t0_r54, "categories"."id" AS t1_r0, "categories"."name" AS t1_r1, "categories"."color" AS t1_r2, "categories"."topic_id" AS t1_r3, "categories"."topic_count" AS t1_r4, "categories"."created_at" AS t1_r5, "categories"."updated_at" AS t1_r6, "categories"."user_id" AS t1_r7, "categories"."topics_year" AS t1_r8, "categories"."topics_month" AS t1_r9, "categories"."topics_week" AS t1_r10, "categories"."slug" AS t1_r11, "categories"."description" AS t1_r12, "categories"."text_color" AS t1_r13, "categories"."read_restricted" AS t1_r14, "categories"."auto_close_hours" AS t1_r15, "categories"."post_count" AS t1_r16, "categories"."latest_post_id" AS t1_r17, "categories"."latest_topic_id" AS t1_r18, "categories"."position" AS t1_r19, "categories"."parent_category_id" AS t1_r20, "categories"."posts_year" AS t1_r21, "categories"."posts_month" AS t1_r22, "categories"."posts_week" AS t1_r23, "categories"."email_in" AS t1_r24, "categories"."email_in_allow_strangers" AS t1_r25, "categories"."topics_day" AS t1_r26, "categories"."posts_day" AS t1_r27, "categories"."logo_url" AS t1_r28, "categories"."background_url" AS t1_r29, "categories"."allow_badges" AS t1_r30, "categories"."name_lower" AS t1_r31, "categories"."auto_close_based_on_last_post" AS t1_r32, "categories"."topic_template" AS t1_r33, "categories"."suppress_from_homepage" AS t1_r34, "categories"."contains_messages" AS t1_r35, "categories"."sort_order" AS t1_r36, "categories"."sort_ascending" AS t1_r37, "categories"."uploaded_logo_id" AS t1_r38, "categories"."uploaded_background_id" AS t1_r39, "categories"."topic_featured_link_allowed" AS t1_r40, "categories"."all_topics_wiki" AS t1_r41 FROM "topics" LEFT OUTER JOIN "categories" ON "categories"."id" = "topics"."category_id" LEFT OUTER JOIN topic_users AS tu ON (topics.id = tu.topic_id AND tu.user_id = 46) WHERE (categories.id = 5 OR (categories.parent_category_id = 5 AND categories.topic_id <> topics.id)) AND (topics.archetype <> 'private_message') AND (topics.deleted_at IS NULL) AND (COALESCE(tu.notification_level,1) > 0) AND ( NOT EXISTS ( SELECT 1 FROM category_users cu WHERE cu.user_id = 46 AND cu.category_id = topics.category_id AND cu.notification_level = 0 AND cu.category_id <> 5 AND (tu.notification_level IS NULL OR tu.notification_level tu.cleared_pinned_at OR tu.cleared_pinned_at IS NULL)) ORDER BY topics.bumped_at DESC LIMIT 30) /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-mini-profiler-0.10.1/lib/patches/db/pg.rb:90:in `exec' Backtrace /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-mini-profiler-0.10.1/lib/patches/db/pg.rb:90:in `exec' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-mini-profiler-0.10.1/lib/patches/db/pg.rb:90:in `async_exec' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `block in exec_no_cache' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `exec_no_cache' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:584:in `execute_and_clear' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/query_cache.rb:68:in `block in select_all' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/query_cache.rb:83:in `cache_sql' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/query_cache.rb:68:in `select_all' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/finder_methods.rb:356:in `find_with_associations' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation.rb:639:in `exec_queries' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation.rb:515:in `load' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation.rb:243:in `to_a' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/delegation.rb:132:in `method_missing' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/delegation.rb:99:in `method_missing' /var/www/discourse/lib/topic_query.rb:268:in `prioritize_pinned_topics' /var/www/discourse/lib/topic_query.rb:285:in `create_list' /var/www/discourse/lib/topic_query.rb:125:in `list_latest' /var/www/discourse/app/controllers/list_controller.rb:68:in `public_send' /var/www/discourse/app/controllers/list_controller.rb:68:in `block (2 levels) in ' /var/www/discourse/app/controllers/list_controller.rb:92:in `block (2 levels) in ' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_controller/metal/implicit_render.rb:4:in `send_action' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/abstract_controller/base.rb:198:in `process_action' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_controller/metal/rendering.rb:10:in `process_action' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/abstract_controller/callbacks.rb:20:in `block in process_action' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:117:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:505:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:92:in `__run_callbacks__' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:81:in `run_callbacks' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/abstract_controller/callbacks.rb:19:in `process_action' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_controller/metal/rescue.rb:29:in `process_action' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications.rb:164:in `block in instrument' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications.rb:164:in `instrument' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_controller/metal/instrumentation.rb:30:in `process_action' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_controller/metal/params_wrapper.rb:250:in `process_action' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/controller_runtime.rb:18:in `process_action' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/abstract_controller/base.rb:137:in `process' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionview-4.2.7.1/lib/action_view/rendering.rb:30:in `process' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-mini-profiler-0.10.1/lib/mini_profiler/profiling_methods.rb:76:in `block in profile_method' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_controller/metal.rb:196:in `dispatch' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_controller/metal.rb:237:in `block in action' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/routing/route_set.rb:74:in `dispatch' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/routing/route_set.rb:43:in `serve' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/journey/router.rb:43:in `block in serve' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/journey/router.rb:30:in `each' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/journey/router.rb:30:in `serve' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/routing/route_set.rb:817:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/builder.rb:63:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/conditionalget.rb:25:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/head.rb:13:in `call' /var/www/discourse/lib/middleware/anonymous_cache.rb:136:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/params_parser.rb:27:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/flash.rb:260:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:225:in `context' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:220:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/cookies.rb:560:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/query_cache.rb:36:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:88:in `__run_callbacks__' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:778:in `_run_call_callbacks' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:81:in `run_callbacks' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/callbacks.rb:27:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/remote_ip.rb:78:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/logster-1.2.7/lib/logster/middleware/reporter.rb:31:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/rack/logger.rb:38:in `call_app' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/rack/logger.rb:22:in `call' /var/www/discourse/config/initializers/100-quiet_logger.rb:17:in `call_with_quiet_assets' /var/www/discourse/config/initializers/100-silence_logger.rb:29:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/request_id.rb:21:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/methodoverride.rb:22:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/runtime.rb:18:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/sendfile.rb:113:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-mini-profiler-0.10.1/lib/mini_profiler/profiler.rb:171:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.2/lib/message_bus/rack/middleware.rb:62:in `call' /var/www/discourse/lib/middleware/request_tracker.rb:73:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/engine.rb:518:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/application.rb:165:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/railtie.rb:194:in `public_send' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/railtie.rb:194:in `method_missing' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/urlmap.rb:66:in `block in call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/urlmap.rb:50:in `each' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/urlmap.rb:50:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/unicorn-5.2.0/lib/unicorn/http_server.rb:562:in `process_client' /var/www/discourse/lib/scheduler/defer.rb:85:in `process_client' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/unicorn-5.2.0/lib/unicorn/http_server.rb:658:in `worker_loop' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/unicorn-5.2.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/unicorn-5.2.0/lib/unicorn/http_server.rb:132:in `start' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/unicorn-5.2.0/bin/unicorn:126:in ` ' /var/www/discourse/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load' /var/www/discourse/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in ` ' Env hostname aw0v-prodstaffdiscourse2-app process_id [5206, 5228, 5283, 5248] application_version 675cb1c71569193b079440a0b2d476d804fc18c0 HTTP_HOST **redacted** REQUEST_URI /c/important-updates.json?api_key=**redacted** REQUEST_METHOD GET HTTP_ACCEPT */* HTTP_X_FORWARDED_FOR **redacted** HTTP_X_REAL_IP **redacted** username MotherBrain params api_key **redacted** api_username MotherBrain #2 (Fatal), 40 Occurences: Message (40 copies reported) Job exception: PG::UndefinedColumn: ERROR: column users.auth_token does not exist LINE 1: ...e" AS t1_r10, "users"."username_lower" AS t1_r11, "users"."a... ^ : SELECT "user_avatars"."id" AS t0_r0, "user_avatars"."user_id" AS t0_r1, "user_avatars"."custom_upload_id" AS t0_r2, "user_avatars"."gravatar_upload_id" AS t0_r3, "user_avatars"."last_gravatar_download_attempt" AS t0_r4, "user_avatars"."created_at" AS t0_r5, "user_avatars"."updated_at" AS t0_r6, "users"."id" AS t1_r0, "users"."username" AS t1_r1, "users"."created_at" AS t1_r2, "users"."updated_at" AS t1_r3, "users"."name" AS t1_r4, "users"."seen_notification_id" AS t1_r5, "users"."last_posted_at" AS t1_r6, "users"."email" AS t1_r7, "users"."password_hash" AS t1_r8, "users"."salt" AS t1_r9, "users"."active" AS t1_r10, "users"."username_lower" AS t1_r11, "users"."auth_token" AS t1_r12, "users"."last_seen_at" AS t1_r13, "users"."admin" AS t1_r14, "users"."last_emailed_at" AS t1_r15, "users"."trust_level" AS t1_r16, "users"."approved" AS t1_r17, "users"."approved_by_id" AS t1_r18, "users"."approved_at" AS t1_r19, "users"."previous_visit_at" AS t1_r20, "users"."suspended_at" AS t1_r21, "users"."suspended_till" AS t1_r22, "users"."date_of_birth" AS t1_r23, "users"."views" AS t1_r24, "users"."flag_level" AS t1_r25, "users"."ip_address" AS t1_r26, "users"."moderator" AS t1_r27, "users"."blocked" AS t1_r28, "users"."title" AS t1_r29, "users"."uploaded_avatar_id" AS t1_r30, "users"."locale" AS t1_r31, "users"."primary_group_id" AS t1_r32, "users"."registration_ip_address" AS t1_r33, "users"."trust_level_locked" AS t1_r34, "users"."staged" AS t1_r35, "users"."first_seen_at" AS t1_r36, "users"."auth_token_updated_at" AS t1_r37 FROM "user_avatars" INNER JOIN "users" ON "users"."id" = "user_avatars"."user_id" WHERE "user_avatars"."last_gravatar_download_attempt" IS NULL ORDER BY users.last_posted_at DESC LIMIT 5000 Backtrace /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-mini-profiler-0.10.1/lib/patches/db/pg.rb:90:in `exec' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-mini-profiler-0.10.1/lib/patches/db/pg.rb:90:in `async_exec' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `block in exec_no_cache' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `exec_no_cache' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:584:in `execute_and_clear' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/query_cache.rb:70:in `select_all' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/finder_methods.rb:356:in `find_with_associations' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation.rb:639:in `exec_queries' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation.rb:515:in `load' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation.rb:243:in `to_a' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/delegation.rb:46:in `each' /var/www/discourse/app/jobs/scheduled/create_missing_avatars.rb:12:in `execute' /var/www/discourse/app/jobs/base.rb:154:in `block (2 levels) in perform' Env hostname aw0v-prodstaffdiscourse2-app process_id 5335 application_version 675cb1c71569193b079440a0b2d476d804fc18c0 current_db default current_hostname **redacted** job Jobs::CreateMissingAvatars problem_db default opts #3 (Error), 3 Occurrences: Message (3 copies reported) ActionController::RoutingError (uninitialized constant ManifestJsonController) /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/inflector/methods.rb:261:in `const_get' Backtrace /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/inflector/methods.rb:261:in `const_get' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/inflector/methods.rb:261:in `block in constantize' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/inflector/methods.rb:259:in `each' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/inflector/methods.rb:259:in `inject' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/inflector/methods.rb:259:in `constantize' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/routing/route_set.rb:70:in `controller_reference' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/routing/route_set.rb:60:in `controller' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/routing/route_set.rb:39:in `serve' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/journey/router.rb:43:in `block in serve' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/journey/router.rb:30:in `each' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/journey/router.rb:30:in `serve' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/routing/route_set.rb:817:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/omniauth-1.3.1/lib/omniauth/builder.rb:63:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/conditionalget.rb:25:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/head.rb:13:in `call' /var/www/discourse/lib/middleware/anonymous_cache.rb:136:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/params_parser.rb:27:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/flash.rb:260:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:225:in `context' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:220:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/cookies.rb:560:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/query_cache.rb:36:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:88:in `__run_callbacks__' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:778:in `_run_call_callbacks' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:81:in `run_callbacks' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/callbacks.rb:27:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/remote_ip.rb:78:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/logster-1.2.7/lib/logster/middleware/reporter.rb:31:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/rack/logger.rb:38:in `call_app' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/rack/logger.rb:22:in `call' /var/www/discourse/config/initializers/100-quiet_logger.rb:17:in `call_with_quiet_assets' /var/www/discourse/config/initializers/100-silence_logger.rb:29:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/request_id.rb:21:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/methodoverride.rb:22:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/runtime.rb:18:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/sendfile.rb:113:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-mini-profiler-0.10.1/lib/mini_profiler/profiler.rb:171:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.2/lib/message_bus/rack/middleware.rb:62:in `call' /var/www/discourse/lib/middleware/request_tracker.rb:73:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/engine.rb:518:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/application.rb:165:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/railtie.rb:194:in `public_send' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/railtie.rb:194:in `method_missing' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/urlmap.rb:66:in `block in call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/urlmap.rb:50:in `each' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-1.6.5/lib/rack/urlmap.rb:50:in `call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/unicorn-5.2.0/lib/unicorn/http_server.rb:562:in `process_client' /var/www/discourse/lib/scheduler/defer.rb:85:in `process_client' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/unicorn-5.2.0/lib/unicorn/http_server.rb:658:in `worker_loop' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/unicorn-5.2.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/unicorn-5.2.0/lib/unicorn/http_server.rb:132:in `start' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/unicorn-5.2.0/bin/unicorn:126:in ` ' /var/www/discourse/vendor/bundle/ruby/2.3.0/bin/unicorn:22:in `load' /var/www/discourse/vendor/bundle/ruby/2.3.0/bin/unicorn:22:in ` ' Env hostname aw0v-prodstaffdiscourse2-app process_id [183, 135, 110] application_version 040e10a6275e687452ef2119de1cd93bc3fa5854 HTTP_HOST **redacted** REQUEST_URI /manifest.json REQUEST_METHOD GET HTTP_ACCEPT */* HTTP_REFERER [https://**redacted**/login/, https://**redacted**/admin/site_settings/category/required, https://**redacted**/] HTTP_X_FORWARDED_FOR **redacted** HTTP_X_REAL_IP **redacted**

[image] jomaxro: Is your thinking that these errors are completely unrelated to the earlier incident I reported, and not at all an issue? Pretty much. If they never happen again, there’s high chances they were due to a rebuild. (Even though we make our best to do “rolling builds”, it’s never an easy task and sometimes a commit might generate errors unless all the containers have the same version of the code)

Not sure about #3, but the first two are related to the database schema being different from what the app expects. Specifically, given that the code is expecting users.auth_token, but that got nuked not that long ago, I’m thinking the database schema has been migrated, but the code that is actually running is an older version.

Thanks for the info. Any chance there’s an easy solution?

Apart from “run the correct version of the code”, you mean?

What code? We did a rebuild today, and according to docker manager are running the latest tests_passed…

What is the value of the content attribute of the tag in your generated pages? Also, I recall there may have been some confusion on your server about two installs of Discourse; is there any chance that two different versions are running in different places, somehow?

[image] mpalmer: What is the value of the content attribute of the tag in your generated pages? "Discourse 1.8.0.beta6 - https://github.com/discourse/discourse version 3e4dd513c9885c50df36a40e8a567fe7bc66556e" [image] mpalmer: Also, I recall there may have been some confusion on your server about two installs of Discourse; is there any chance that two different versions are running in different places, somehow? That was suggested by someone else, not me. I believe the concern was how long ago our Discourse was launched and whether it was run from /var/docker/ or /var/discourse. He even said “but this is just a guess!”. According to IT, we are only running one version. We do have 2 sites (dev and prod), but they’re not connected, don’t even use the same database.

[image] jomaxro: discourse version 3e4dd513c9885c50df36a40e8a567fe7bc66556e That is, indeed, an extremely up-to-date version. [image] jomaxro: That was suggested by someone else, not me. Ah, sorry, misremembered the background. [image] jomaxro: We do have 2 sites (dev and prod), but they’re not connected, don’t even use the same database. Hmm, it would be worth double-checking that. Looking at the environment for the errors you posted (don’t know why I didn’t scroll down before), I see, for example: application_version 675cb1c71569193b079440a0b2d476d804fc18c0 Something is using an up-to-date database with a code version from five days ago. Do the PIDs listed in those errors exist (remember, they’ll be PIDs inside the Docker container, not the ones on the system itself)? Interestingly, the third error comes from a much newer version (from 7 hours ago, so presumably just before you did another update to the current version).

[image] mpalmer: Do the PIDs listed in those errors exist (remember, they’ll be PIDs inside the Docker container, not the ones on the system itself)? Unfortunately, I don’t have SSH access, so I have no way of checking. I can ask IT, but I’ll need more specifics on exactly what to ask them to check. [image] mpalmer: Interestingly, the third error comes from a much newer version (from 7 hours ago, so presumably just before you did another update to the current version). The errors come from different dates. #1 is from Monday 10:23 AM, #2 is from Tuesday, 1:50 PM, and #3 is Wednesday, 11:10 PM. There were multiple updates today, IT did a full rebuild, and I did a docker_manager update this evening as we were seeing the entire /admin section “untranslated”. Does that explain anything? It’s quite possible the last update (before today), was five days ago…

Can you check in 24 hours and let us know if the number of occurrences increased? They might be transient errors due to a rebuild happening. [image] jomaxro: and I did a docker_manager update this evening as we were seeing the entire /admin section “untranslated”. Is that still happening? Would love to check that out if possible.

[image] zogstrip: Can you check in 24 hours and let us know if the number of occurrences increased? They might be transient errors due to a rebuild happening. I’d be glad to. Will look tonight. [image] zogstrip: [image] jomaxro: and I did a docker_manager update this evening as we were seeing the entire /admin section “untranslated”. Is that still happening? Would love to check that out if possible. Not currently an issue. We’ve seen it a few times, each time we simply did an update from docker_manager and the issue disappeared. I’ll hold off on an immediate update if I catch it again and ping you!

40 Fatals and 7 Errors in recent logs

Product Support

jomaxro (Joshua Rosenfeld) February 17, 2017, 8:16pm 12

No additional errors have occurred. Is your thinking that these errors are completely unrelated to the earlier incident I reported, and not at all an issue?

Topic		Replies	Views
While trying to load /admin/docker/repos Error code: 500 error Bug	6	546	January 9, 2026
Stuck with 500 error after weird bugs and a rebuild Self-hosting	17	2341	January 8, 2024
Update Failing: column "require_message" does not exist Bug	17	728	November 11, 2024
Update Not Completed and Discourse not opening Self-hosting	16	829	June 17, 2024
Plugin causing errors during rebuild Self-hosting	20	2437	October 11, 2022

40 Fatals and 7 Errors in recent logs

Related topics