Over the last ~18h, our server is using close to 100% CPU.
Looking at Sidekiq, it’s busy running the same 5 jobs:
It’s always in “Just now” and they seem to be always re-running over those same 3 topic ids.
I tried restarting Discourse, but the situation remained.
Not sure if it’s related, but this started ~24h after we updated from 3.3.3 to 3.3.4. I don’t see any recent changes to app/jobs/regular/sync_topic_user_bookmarked.rb and I couldn’t spot any change on 3.3.4 that could be related, but I don’t know the codebase.
Any clue on what might be happening?
An update to this. We just noticed that the 3 topic ids over which the tasks keep re-running are topics to which we activated auto-delete replies yesterday, close to when this started. They are all very large topics.
So we know the trigger, but the SQL being repeated doesn’t seem to be doing any batching, so I don’t see why it’s being re-run over and over again.
Is this being triggered by some other job that is still deleting the large number of replies and Jobs::SyncTopicUserBookmarked gets executed for every single deleted post? Or, is this a bug?
We removed the auto-delete from one of the large topics and it didn’t seem to have an effect on the repeated calls of Jobs::SyncTopicUserBookmarked over that topic id.
I reduced SIDEKIQ_WORKERS to 2 and, not too long after, the queue (with many pending things) cleared up and there’s no more Jobs::SyncTopicUserBookmarked running… 
Before trying that, I caught the query below running multiple times at once (using same topic id):
SELECT bookmarks.user_id, COUNT(*)
INTO TEMP TABLE tmp_sync_topic_user_bookmarks
FROM bookmarks
LEFT JOIN posts ON posts.id = bookmarks.bookmarkable_id AND bookmarks.bookmarkable_type = 'Post'
LEFT JOIN topics ON (topics.id = bookmarks.bookmarkable_id AND bookmarks.bookmarkable_type = 'Topic') OR
(topics.id = posts.topic_id)
WHERE (topics.id = 51303 OR posts.topic_id = 51303)
AND posts.deleted_at IS NULL AND topics.deleted_at IS NULL
GROUP BY bookmarks.user_id;
UPDATE topic_users
SET bookmarked = true
FROM tmp_sync_topic_user_bookmarks
WHERE topic_users.user_id = tmp_sync_topic_user_bookmarks.user_id AND
topic_users.topic_id = 51303 AND
tmp_sync_topic_user_bookmarks.count > 0;
UPDATE topic_users
SET bookmarked = false
FROM tmp_sync_topic_user_bookmarks
WHERE topic_users.topic_id = 51303 AND
topic_users.bookmarked = true AND
topic_users.user_id NOT IN (
SELECT tmp_sync_topic_user_bookmarks.user_id
FROM tmp_sync_topic_user_bookmarks
);
DROP TABLE tmp_sync_topic_user_bookmarks;
I’m not clear how PostgreSQL runs multiple statements in a single query, but if it is executed multiple times concurrently, could it cause a loop somehow?
