One category throws 500 Internal Server Error after update

I updated discourse on krita-artists.org few hours ago and I noticed that the #lounge category doesn’t open and shows internal server error. I see this error in the log file

Job exception: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is in recovery mode

and

NoMethodError (undefined method `include?' for nil:NilClass) lib/excerpt_parser.rb:139:in `start_element' lib/excerpt_parser.rb:36:in `block in get_excerpt' lib/excerpt_parser.rb:35:in `catch' lib/exc 

Is there any other place I should for errors? Can someone help me find the issue?

Thank you

Hi Raghavendra!

Your Postgres database seems to be in a recovery mode that does not accept normal connections, so nothing will work until that is fixed.

How have you installed Discourse? Are you using the standard Docker installation method?

If so, you could try entering the container with launcher enter app and then issuing sv restart postgres for a low effort attempt to restart the DB. If that doesn’t work, then you’ll have to debug the Postgres issue further by looking at its logs in /var/log/postgres/current again inside the container.

Note that it is unlikely that the Discourse upgrade directly caused Postgres to break; is it possible you have run out of disk space or some other environmental issue?

2 Likes

Thanks for the quick reply, I will check the logs and places you mentioned.

Edit:

@leonardo I ran the postgres restart but the issue persists. My disk is 57% utilised out of 76G disk space. My memory has 976Mi available out of 3.7 Gi.

I will check the logs of postgres.

The logs show

discourse@discourse FATAL:  the database system is shutting down

I will check more

That does not speak to the root cause. Can you paste the new lines logged into that file when you do a sv restart postgres?

I get these immediately after the command

2022-12-01 02:23:39.575 UTC [572] LOG:  received smart shutdown request
2022-12-01 02:23:57.342 UTC [35345] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:24:04.475 UTC [35354] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:24:04.482 UTC [35355] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:24:04.488 UTC [35356] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:24:09.160 UTC [23505] discourse@discourse LOG:  duration: 122.355 ms  statement: WITH tags_included_cte AS (
          SELECT 
                   DISTINCT topics.id as topic_id,
                   u.id as user_id,
                   topics.created_at,
                   topics.updated_at,
                   topics.highest_post_number,
                   last_read_post_number,
                   c.id as category_id,
                   tu.notification_level,
                   us.first_unread_at,
                   GREATEST(
                      CASE
                      WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -1 THEN u.created_at
                      WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -2 THEN COALESCE(
                        u.previous_visit_at,u.created_at
                      )
                      ELSE ('2022-12-01 02:24:09.036349'::timestamp - INTERVAL '1 MINUTE' * COALESCE(uo.new_topic_duration_minutes, 2880))
                      END, u.created_at, '2022-11-22 15:21:19'
                   ) AS treat_as_new_topic_start_date
        FROM topics
        JOIN users u on u.id = 2541
        JOIN user_stats AS us ON us.user_id = u.id
        JOIN user_options AS uo ON uo.user_id = u.id
        JOIN categories c ON c.id = topics.category_id
        LEFT JOIN topic_users tu ON tu.topic_id = topics.id AND tu.user_id = u.id
        LEFT JOIN dismissed_topic_users ON dismissed_topic_users.topic_id = topics.id AND dismissed_topic_users.user_id = 2541

        WHERE u.id = 2541 AND
              
              topics.archetype <> 'private_message' AND
              ((1=0) OR ("topics"."deleted_at" IS NULL AND (topics.created_at >= (GREATEST(CASE
                          WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -1 THEN u.created_at
                          WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -2 THEN COALESCE(u.previous_visit_at,u.created_at)
                          ELSE ('2022-12-01 02:24:09.033128'::timestamp - INTERVAL '1 MINUTE' * COALESCE(uo.new_topic_duration_minutes, 2880))
                       END, u.created_at, '2022-11-22 15:21:19'))) AND (tu.last_read_post_number IS NULL) AND (COALESCE(tu.notification_level, 2) >= 2) AND topics.created_at > '2022-11-22 15:21:19' AND dismissed_topic_users.id IS NULL)) AND
              (topics.visible OR u.admin OR u.moderator) AND
              
              topics.deleted_at IS NULL AND
              (
         NOT c.read_restricted OR u.admin OR c.id IN (
            SELECT c2.id FROM categories c2
            JOIN category_groups cg ON cg.category_id = c2.id
            JOIN group_users gu ON gu.user_id = 2541 AND cg.group_id = gu.group_id
            WHERE c2.read_restricted )
        ) AND

              NOT (
                last_read_post_number IS NULL AND
                (
                  topics.category_id IN (SELECT "categories"."id" FROM "categories" LEFT JOIN categories categories2 ON categories2.id = categories.parent_category_id LEFT JOIN category_users ON category_users.category_id = categories.id AND category_users.user_id = 2541 LEFT JOIN category_users category_users2 ON category_users2.category_id = categories2.id AND category_users2.user_id = 2541 WHERE ((category_users.id IS NULL AND COALESCE(category_users2.notification_level, 1) = 0) OR COALESCE(category_users.notification_level, 1) = 0))
                  AND tu.notification_level <= 1
                )
              )

        UNION ALL

        SELECT 
                   DISTINCT topics.id as topic_id,
                   u.id as user_id,
                   topics.created_at,
                   topics.updated_at,
                   topics.highest_post_number,
                   last_read_post_number,
                   c.id as category_id,
                   tu.notification_level,
                   us.first_unread_at,
                   GREATEST(
                      CASE
                      WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -1 THEN u.created_at
                      WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -2 THEN COALESCE(
                        u.previous_visit_at,u.created_at
                      )
                      ELSE ('2022-12-01 02:24:09.036349'::timestamp - INTERVAL '1 MINUTE' * COALESCE(uo.new_topic_duration_minutes, 2880))
                      END, u.created_at, '2022-11-22 15:21:19'
                   ) AS treat_as_new_topic_start_date
        FROM topics
        JOIN users u on u.id = 2541
        JOIN user_stats AS us ON us.user_id = u.id
        JOIN user_options AS uo ON uo.user_id = u.id
        JOIN categories c ON c.id = topics.category_id
        LEFT JOIN topic_users tu ON tu.topic_id = topics.id AND tu.user_id = u.id


        WHERE u.id = 2541 AND
               topics.updated_at >= us.first_unread_at AND 
              topics.archetype <> 'private_message' AND
              (("topics"."deleted_at" IS NULL AND (tu.last_read_post_number < topics.highest_post_number) AND (COALESCE(tu.notification_level, 1) >= 2)) OR (1=0)) AND
              (topics.visible OR u.admin OR u.moderator) AND
              
              topics.deleted_at IS NULL AND
              (
         NOT c.read_restricted OR u.admin OR c.id IN (
            SELECT c2.id FROM categories c2
            JOIN category_groups cg ON cg.category_id = c2.id
            JOIN group_users gu ON gu.user_id = 2541 AND cg.group_id = gu.group_id
            WHERE c2.read_restricted )
        ) AND

              NOT (
                last_read_post_number IS NULL AND
                (
                  topics.category_id IN (SELECT "categories"."id" FROM "categories" LEFT JOIN categories categories2 ON categories2.id = categories.parent_category_id LEFT JOIN category_users ON category_users.category_id = categories.id AND category_users.user_id = 2541 LEFT JOIN category_users category_users2 ON category_users2.category_id = categories2.id AND category_users2.user_id = 2541 WHERE ((category_users.id IS NULL AND COALESCE(category_users2.notification_level, 1) = 0) OR COALESCE(category_users.notification_level, 1) = 0))
                  AND tu.notification_level <= 1
                )
              )

        )
        SELECT *, (
          SELECT ARRAY_AGG(name) from topic_tags
             JOIN tags on tags.id = topic_tags.tag_id
             WHERE topic_id = tags_included_cte.topic_id
          ) tags
        FROM tags_included_cte


         LIMIT 5000
2022-12-01 02:24:27.397 UTC [35382] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:24:29.357 UTC [35387] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:24:43.996 UTC [35406] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:24:44.001 UTC [35407] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:24:48.968 UTC [23505] discourse@discourse LOG:  duration: 109.101 ms  execute <unnamed>: SELECT topics.*, COALESCE((SELECT 1 FROM discourse_voting_votes WHERE user_id = 2541 AND topic_id = topics.id), 0) AS current_user_voted, "topics"."id" AS t0_r0, "categories"."id" AS t1_r0, "categories"."name" AS t1_r1, "categories"."color" AS t1_r2, "categories"."topic_id" AS t1_r3, "categories"."topic_count" AS t1_r4, "categories"."created_at" AS t1_r5, "categories"."updated_at" AS t1_r6, "categories"."user_id" AS t1_r7, "categories"."topics_year" AS t1_r8, "categories"."topics_month" AS t1_r9, "categories"."topics_week" AS t1_r10, "categories"."slug" AS t1_r11, "categories"."description" AS t1_r12, "categories"."text_color" AS t1_r13, "categories"."read_restricted" AS t1_r14, "categories"."auto_close_hours" AS t1_r15, "categories"."post_count" AS t1_r16, "categories"."latest_post_id" AS t1_r17, "categories"."latest_topic_id" AS t1_r18, "categories"."position" AS t1_r19, "categories"."parent_category_id" AS t1_r20, "categories"."posts_year" AS t1_r21, "categories"."posts_month" AS t1_r22, "categories"."posts_week" AS t1_r23, "categories"."email_in" AS t1_r24, "categories"."email_in_allow_strangers" AS t1_r25, "categories"."topics_day" AS t1_r26, "categories"."posts_day" AS t1_r27, "categories"."allow_badges" AS t1_r28, "categories"."name_lower" AS t1_r29, "categories"."auto_close_based_on_last_post" AS t1_r30, "categories"."topic_template" AS t1_r31, "categories"."contains_messages" AS t1_r32, "categories"."sort_order" AS t1_r33, "categories"."sort_ascending" AS t1_r34, "categories"."uploaded_logo_id" AS t1_r35, "categories"."uploaded_background_id" AS t1_r36, "categories"."topic_featured_link_allowed" AS t1_r37, "categories"."all_topics_wiki" AS t1_r38, "categories"."show_subcategory_list" AS t1_r39, "categories"."num_featured_topics" AS t1_r40, "categories"."default_view" AS t1_r41, "categories"."subcategory_list_style" AS t1_r42, "categories"."default_top_period" AS t1_r43, "categories"."mailinglist_mirror" AS t1_r44, "categories"."minimum_required_tags" AS t1_r45, "categories"."navigate_to_first_post_after_read" AS t1_r46, "categories"."search_priority" AS t1_r47, "categories"."allow_global_tags" AS t1_r48, "categories"."reviewable_by_group_id" AS t1_r49, "categories"."read_only_banner" AS t1_r50, "categories"."default_list_filter" AS t1_r51, "categories"."allow_unlimited_owner_edits_on_first_post" AS t1_r52, "categories"."default_slow_mode_seconds" AS t1_r53, "categories"."uploaded_logo_dark_id" AS t1_r54, "discourse_voting_topic_vote_count"."id" AS t2_r0, "discourse_voting_topic_vote_count"."topic_id" AS t2_r1, "discourse_voting_topic_vote_count"."votes_count" AS t2_r2, "discourse_voting_topic_vote_count"."created_at" AS t2_r3, "discourse_voting_topic_vote_count"."updated_at" AS t2_r4 FROM "topics" LEFT OUTER JOIN "categories" ON "categories"."id" = "topics"."category_id" LEFT OUTER JOIN "discourse_voting_topic_vote_count" ON "discourse_voting_topic_vote_count"."topic_id" = "topics"."id" LEFT OUTER JOIN topic_users AS tu ON (topics.id = tu.topic_id AND tu.user_id = 2541) WHERE (topics.archetype <> 'private_message') AND (topics.category_id IS NULL or topics.category_id IN (44,4,39,18,41,6,9,1,37,38,15,33,36,16,10,13,32,28,29,24,20,35,2,26,19,7,30,42,8,5,34,27,14,31,40,22,11,17,36)) AND "topics"."visible" = TRUE AND (topics.deleted_at IS NULL) AND (tu.last_read_post_number < topics.highest_post_number) AND (COALESCE(tu.notification_level, 1) >= 2) AND (topics.updated_at >= '2022-09-02 02:24:48.854073') AND (topics.id NOT IN (50776)) AND "topics"."visible" = TRUE ORDER BY CASE WHEN topics.user_id = tu.user_id THEN 1 ELSE 2 END, CASE WHEN topics.category_id = 22 THEN 0 ELSE 1 END, topics.bumped_at DESC LIMIT 5
2022-12-01 02:24:58.325 UTC [35305] discourse@discourse LOG:  duration: 883.515 ms  statement:  WITH w as (
           INSERT INTO user_badges(badge_id, user_id, granted_at, granted_by_id, created_at, post_id)
           SELECT 103, q.user_id, q.granted_at, -1, current_timestamp, NULL
             FROM (
                    SELECT id user_id, current_timestamp granted_at
        FROM users
        WHERE id  IN (
               SELECT p1.user_id
               FROM post_custom_fields pc
               JOIN badge_posts p1 ON p1.id = pc.post_id
               JOIN topics t1 ON p1.topic_id = t1.id
               WHERE p1.user_id <> t1.user_id AND
                            name = 'is_accepted_answer' AND
                    p1.user_id IN (
                           SELECT user_id
                           FROM posts
                           WHERE FALSE OR  p1.id IN (157913)
                    )
                GROUP BY p1.user_id
                HAVING COUNT(*) > 9
        )

                  ) q
        LEFT JOIN user_badges ub ON ub.badge_id = 103 AND ub.user_id = q.user_id
           
           WHERE (ub.badge_id IS NULL AND q.user_id > 0)
           ON CONFLICT DO NOTHING
           RETURNING id, user_id, granted_at
         )
         SELECT w.*, username, locale, (u.admin OR u.moderator) AS staff, uo.skip_new_user_tips
           FROM w
           JOIN users u on u.id = w.user_id
           JOIN user_options uo ON uo.user_id = w.user_id

2022-12-01 02:24:58.498 UTC [35424] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:24:58.517 UTC [35425] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:24:58.720 UTC [35057] discourse@discourse LOG:  duration: 110.735 ms  statement: WITH tags_included_cte AS (
          SELECT 
                   DISTINCT topics.id as topic_id,
                   u.id as user_id,
                   topics.created_at,
                   topics.updated_at,
                   topics.highest_post_number,
                   last_read_post_number,
                   c.id as category_id,
                   tu.notification_level,
                   us.first_unread_at,
                   GREATEST(
                      CASE
                      WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -1 THEN u.created_at
                      WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -2 THEN COALESCE(
                        u.previous_visit_at,u.created_at
                      )
                      ELSE ('2022-12-01 02:24:58.609758'::timestamp - INTERVAL '1 MINUTE' * COALESCE(uo.new_topic_duration_minutes, 2880))
                      END, u.created_at, '2022-11-22 15:21:19'
                   ) AS treat_as_new_topic_start_date
        FROM topics
        JOIN users u on u.id = 1984
        JOIN user_stats AS us ON us.user_id = u.id
        JOIN user_options AS uo ON uo.user_id = u.id
        JOIN categories c ON c.id = topics.category_id
        LEFT JOIN topic_users tu ON tu.topic_id = topics.id AND tu.user_id = u.id
        LEFT JOIN dismissed_topic_users ON dismissed_topic_users.topic_id = topics.id AND dismissed_topic_users.user_id = 1984

        WHERE u.id = 1984 AND
              
              topics.archetype <> 'private_message' AND
              ((1=0) OR ("topics"."deleted_at" IS NULL AND (topics.created_at >= (GREATEST(CASE
                          WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -1 THEN u.created_at
                          WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -2 THEN COALESCE(u.previous_visit_at,u.created_at)
                          ELSE ('2022-12-01 02:24:58.608748'::timestamp - INTERVAL '1 MINUTE' * COALESCE(uo.new_topic_duration_minutes, 2880))
                       END, u.created_at, '2022-11-22 15:21:19'))) AND (tu.last_read_post_number IS NULL) AND (COALESCE(tu.notification_level, 2) >= 2) AND topics.created_at > '2022-11-22 15:21:19' AND dismissed_topic_users.id IS NULL)) AND
              (topics.visible OR u.admin OR u.moderator) AND
              
              topics.deleted_at IS NULL AND
              (
         NOT c.read_restricted OR u.admin OR c.id IN (
            SELECT c2.id FROM categories c2
            JOIN category_groups cg ON cg.category_id = c2.id
            JOIN group_users gu ON gu.user_id = 1984 AND cg.group_id = gu.group_id
            WHERE c2.read_restricted )
        ) AND

              NOT (
                last_read_post_number IS NULL AND
                (
                  topics.category_id IN (SELECT "categories"."id" FROM "categories" LEFT JOIN categories categories2 ON categories2.id = categories.parent_category_id LEFT JOIN category_users ON category_users.category_id = categories.id AND category_users.user_id = 1984 LEFT JOIN category_users category_users2 ON category_users2.category_id = categories2.id AND category_users2.user_id = 1984 WHERE ((category_users.id IS NULL AND COALESCE(category_users2.notification_level, 1) = 0) OR COALESCE(category_users.notification_level, 1) = 0))
                  AND tu.notification_level <= 1
                )
              )

        UNION ALL

        SELECT 
                   DISTINCT topics.id as topic_id,
                   u.id as user_id,
                   topics.created_at,
                   topics.updated_at,
                   topics.highest_post_number,
                   last_read_post_number,
                   c.id as category_id,
                   tu.notification_level,
                   us.first_unread_at,
                   GREATEST(
                      CASE
                      WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -1 THEN u.created_at
                      WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -2 THEN COALESCE(
                        u.previous_visit_at,u.created_at
                      )
                      ELSE ('2022-12-01 02:24:58.609758'::timestamp - INTERVAL '1 MINUTE' * COALESCE(uo.new_topic_duration_minutes, 2880))
                      END, u.created_at, '2022-11-22 15:21:19'
                   ) AS treat_as_new_topic_start_date
        FROM topics
        JOIN users u on u.id = 1984
        JOIN user_stats AS us ON us.user_id = u.id
        JOIN user_options AS uo ON uo.user_id = u.id
        JOIN categories c ON c.id = topics.category_id
        LEFT JOIN topic_users tu ON tu.topic_id = topics.id AND tu.user_id = u.id


        WHERE u.id = 1984 AND
               topics.updated_at >= us.first_unread_at AND 
              topics.archetype <> 'private_message' AND
              (("topics"."deleted_at" IS NULL AND (tu.last_read_post_number < topics.highest_post_number) AND (COALESCE(tu.notification_level, 1) >= 2)) OR (1=0)) AND
              (topics.visible OR u.admin OR u.moderator) AND
              
              topics.deleted_at IS NULL AND
              (
         NOT c.read_restricted OR u.admin OR c.id IN (
            SELECT c2.id FROM categories c2
            JOIN category_groups cg ON cg.category_id = c2.id
            JOIN group_users gu ON gu.user_id = 1984 AND cg.group_id = gu.group_id
            WHERE c2.read_restricted )
        ) AND

              NOT (
                last_read_post_number IS NULL AND
                (
                  topics.category_id IN (SELECT "categories"."id" FROM "categories" LEFT JOIN categories categories2 ON categories2.id = categories.parent_category_id LEFT JOIN category_users ON category_users.category_id = categories.id AND category_users.user_id = 1984 LEFT JOIN category_users category_users2 ON category_users2.category_id = categories2.id AND category_users2.user_id = 1984 WHERE ((category_users.id IS NULL AND COALESCE(category_users2.notification_level, 1) = 0) OR COALESCE(category_users.notification_level, 1) = 0))
                  AND tu.notification_level <= 1
                )
              )

        )
        SELECT *, (
          SELECT ARRAY_AGG(name) from topic_tags
             JOIN tags on tags.id = topic_tags.tag_id
             WHERE topic_id = tags_included_cte.topic_id
          ) tags
        FROM tags_included_cte


         LIMIT 5000
2022-12-01 02:24:59.146 UTC [35305] discourse@discourse LOG:  duration: 810.816 ms  statement:  WITH w as (
           INSERT INTO user_badges(badge_id, user_id, granted_at, granted_by_id, created_at, post_id)
           SELECT 102, q.user_id, q.granted_at, -1, current_timestamp, q.post_id
             FROM (
                    SELECT p.user_id, p.id post_id, p.updated_at granted_at
        FROM badge_posts p
        WHERE p.post_number > 1 AND
            p.id IN (
              SELECT post_id FROM (
               SELECT pc.post_id, row_number()
               OVER (PARTITION BY p1.user_id ORDER BY pc.created_at) as rnum
               FROM post_custom_fields pc
               JOIN badge_posts p1 ON p1.id = pc.post_id
               JOIN topics t1 ON p1.topic_id = t1.id
               WHERE name = 'is_accepted_answer' AND
                            p1.user_id <> t1.user_id AND
                (
                  FALSE OR
                   p1.user_id IN (
                           select user_id from posts where p1.id IN (157913)
                   )
               )
        ) X  WHERE rnum = 1)

                  ) q
        LEFT JOIN user_badges ub ON ub.badge_id = 102 AND ub.user_id = q.user_id
           AND (q.post_id = ub.post_id OR NOT FALSE)
           WHERE (ub.badge_id IS NULL AND q.user_id > 0)
           ON CONFLICT DO NOTHING
           RETURNING id, user_id, granted_at
         )
         SELECT w.*, username, locale, (u.admin OR u.moderator) AS staff, uo.skip_new_user_tips
           FROM w
           JOIN users u on u.id = w.user_id
           JOIN user_options uo ON uo.user_id = w.user_id

2022-12-01 02:25:02.870 UTC [35439] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:03.388 UTC [35441] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:03.408 UTC [35442] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:03.415 UTC [35443] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:03.438 UTC [35444] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:06.533 UTC [35448] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:15.093 UTC [35459] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:28.501 UTC [35474] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:29.428 UTC [35477] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:33.640 UTC [35484] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:33.641 UTC [35485] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:33.648 UTC [35487] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:33.649 UTC [35486] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:33.760 UTC [35490] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:38.707 UTC [35496] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:41.961 UTC [35500] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:43.143 UTC [35503] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:53.888 UTC [35515] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:53.892 UTC [35516] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:53.895 UTC [35517] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:25:59.124 UTC [35305] discourse@discourse LOG:  duration: 570.233 ms  statement:  WITH w as (
           INSERT INTO user_badges(badge_id, user_id, granted_at, granted_by_id, created_at, post_id)
           SELECT 103, q.user_id, q.granted_at, -1, current_timestamp, NULL
             FROM (
                    SELECT id user_id, current_timestamp granted_at
        FROM users
        WHERE id  IN (
               SELECT p1.user_id
               FROM post_custom_fields pc
               JOIN badge_posts p1 ON p1.id = pc.post_id
               JOIN topics t1 ON p1.topic_id = t1.id
               WHERE p1.user_id <> t1.user_id AND
                            name = 'is_accepted_answer' AND
                    p1.user_id IN (
                           SELECT user_id
                           FROM posts
                           WHERE FALSE OR  p1.id IN (196496)
                    )
                GROUP BY p1.user_id
                HAVING COUNT(*) > 9
        )

                  ) q
        LEFT JOIN user_badges ub ON ub.badge_id = 103 AND ub.user_id = q.user_id
           
           WHERE (ub.badge_id IS NULL AND q.user_id > 0)
           ON CONFLICT DO NOTHING
           RETURNING id, user_id, granted_at
         )
         SELECT w.*, username, locale, (u.admin OR u.moderator) AS staff, uo.skip_new_user_tips
           FROM w
           JOIN users u on u.id = w.user_id
           JOIN user_options uo ON uo.user_id = w.user_id

2022-12-01 02:25:59.953 UTC [35305] discourse@discourse LOG:  duration: 824.648 ms  statement:  WITH w as (
           INSERT INTO user_badges(badge_id, user_id, granted_at, granted_by_id, created_at, post_id)
           SELECT 102, q.user_id, q.granted_at, -1, current_timestamp, q.post_id
             FROM (
                    SELECT p.user_id, p.id post_id, p.updated_at granted_at
        FROM badge_posts p
        WHERE p.post_number > 1 AND
            p.id IN (
              SELECT post_id FROM (
               SELECT pc.post_id, row_number()
               OVER (PARTITION BY p1.user_id ORDER BY pc.created_at) as rnum
               FROM post_custom_fields pc
               JOIN badge_posts p1 ON p1.id = pc.post_id
               JOIN topics t1 ON p1.topic_id = t1.id
               WHERE name = 'is_accepted_answer' AND
                            p1.user_id <> t1.user_id AND
                (
                  FALSE OR
                   p1.user_id IN (
                           select user_id from posts where p1.id IN (196496)
                   )
               )
        ) X  WHERE rnum = 1)

                  ) q
        LEFT JOIN user_badges ub ON ub.badge_id = 102 AND ub.user_id = q.user_id
           AND (q.post_id = ub.post_id OR NOT FALSE)
           WHERE (ub.badge_id IS NULL AND q.user_id > 0)
           ON CONFLICT DO NOTHING
           RETURNING id, user_id, granted_at
         )
         SELECT w.*, username, locale, (u.admin OR u.moderator) AS staff, uo.skip_new_user_tips
           FROM w
           JOIN users u on u.id = w.user_id
           JOIN user_options uo ON uo.user_id = w.user_id

2022-12-01 02:26:28.600 UTC [35569] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:26:37.384 UTC [35580] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:26:39.492 UTC [35583] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:26:46.274 UTC [35591] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:26:46.277 UTC [35592] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:26:46.282 UTC [35593] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:27:02.604 UTC [35612] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:27:08.817 UTC [35622] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:27:10.616 UTC [35628] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:27:29.701 UTC [35649] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:27:38.709 UTC [35662] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:28:06.998 UTC [35305] discourse@discourse LOG:  duration: 690.969 ms  statement: UPDATE posts
        SET percent_rank = X.percent_rank
        FROM (
          SELECT posts.id, Y.percent_rank
          FROM posts
          JOIN (
            SELECT id, percent_rank()
                         OVER (PARTITION BY topic_id ORDER BY SCORE DESC) as percent_rank
            FROM posts
           ) Y ON Y.id = posts.id
           JOIN topics ON posts.topic_id = topics.id
          WHERE (posts.percent_rank IS NULL OR Y.percent_rank <> posts.percent_rank) AND (topics.bumped_at > '2022-11-30 02:28:06.284754' ) AND (topics.posts_count < 500)
          LIMIT 20000
        ) AS X
        WHERE posts.id = X.id

2022-12-01 02:28:07.191 UTC [35305] discourse@discourse LOG:  duration: 192.483 ms  statement: UPDATE topics AS topics
        SET has_summary = (topics.like_count >= 1 AND
                           topics.posts_count >= 50 AND
                           x.max_score >= 15),
            score = x.avg_score
        FROM (SELECT p.topic_id,
                     MAX(p.score) AS max_score,
                     AVG(p.score) AS avg_score
              FROM posts AS p
              GROUP BY p.topic_id) AS x
              WHERE (x.topic_id = topics.id AND
        (
          (topics.score <> x.avg_score OR topics.score IS NULL) OR
          (topics.has_summary IS NULL OR topics.has_summary <> (
            topics.like_count >= 1 AND
            topics.posts_count >= 50 AND
            x.max_score >= 15
          ))
        )
        ) AND (topics.bumped_at > '2022-11-30 02:28:06.284754' ) AND (topics.posts_count < 500)

2022-12-01 02:28:08.564 UTC [35697] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:28:16.968 UTC [35707] discourse@discourse FATAL:  the database system is shutting down
2022-12-01 02:28:29.802 UTC [35722] discourse@discourse FATAL:  the database system is shutting down

Sorry, I failed to see the forest for the trees! The site is very clearly up so Postgres is not broken, and the PG error you first quoted must have been some transient error in the past.

This is a bug which has been fixed by this commit just a short while ago:

If you update Discourse again now, the error should go away.

4 Likes

That solved the issue really thanks.

Just to be sure about that postgres being in recovery mode I will keep an eye on the logs.

1 Like

Glad to hear it!

On a personal note, I use Krita for all my complex photography retouching needs (basically anything that Rawtherapee is too coarse for) and I love it :heart_eyes:

2 Likes

Wow! I am really glad to find a krita user here on meta, its early morning here and this made my day . If you have any issue with krita please ask in the community you are most welcome there :bouquet:

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.