benjaoming reported a critical bug where an automatic backup process, initiated in September 2019, remained stuck until January 2020 with no timeout mechanism, causing a backlog of email digests and password resets to be sent en masse upon cancellation. The backup log showed a stray pg_dump message after “Finalizing backup…”, raising suspicion about process handling. eviltrout questioned whether the database size contributed, but benjaoming clarified the .dump was only 112 MB. Stephen asked about deployment, while gerhard noted the behavior was odd and pointed to possible Sidekiq pausing issues in older versions. benjaoming confirmed the issue occurred on Discourse v1.8, clarified that only new_version emails were sent (not password resets), and suggested implementing sanity checks for delayed email delivery. gerhard emphasized that v1.8 is extremely outdated and that no such bug exists in current versions, noting Discourse has no unattended upgrades. benjaoming corrected a typo, and gerhard closed the topic, requesting it be reopened only if reproducible on the latest version.
There is seemingly no timeouts for backups. So an automatic backup was stuck since September
Once canceled, all our users received email digests and password resets aggregated since September.
Here’s the log:
[2019-09-26 03:35:25] pg_dump: creating INDEX "public.idx_tag_users_ix1"
[2019-09-26 03:35:25] pg_dump: creating INDEX "public.idx_tag_users_ix2"
[2019-09-26 03:35:25] pg_dump: creating INDEX "public.idx_topic_id_public_type_deleted_at"
[2019-09-26 03:35:25] pg_dump: creating INDEX "public.idx_topics_front_page"
[2019-09-26 03:35:25] pg_dump: creating INDEX "public.idx_topics_user_id_deleted_at"
[2019-09-26 03:35:25] pg_dump: creating INDEX "public.idx_unique_actions"
[2019-09-26 03:35:25] pg_dump: creating INDEX "public.idx_unique_flags"
[2019-09-26 03:35:25] Finalizing backup...
[2019-09-26 03:35:25] Creating archive: our-community-2019-09-26-033520-v20171214040346.tar.gz
[2019-09-26 03:35:25] Making sure archive does not already exist...
[2019-09-26 03:35:25] pg_dump: creating INDEX "public.idx_unique_post_uploads"
[2020-01-24 16:02:39] Backup process was cancelled!
[2020-01-24 16:02:39] Notifying 'system' of the end of the backup...
This is very weird. There’s a stray log message from “pg_dump” after the “Finalizing backup…” message.
Also, I’m not sure why the system didn’t send emails anymore. Sidekiq should already be unpaused when “Finalizing backup…” appears in the logs.
What version of Discourse are you using? I assume you didn’t run any upgrades since September, otherwise I’m quite sure that would have stopped the backup.
Thanks for reading and responding! I noticed the issue while inspecting an old server, which has now been replaced. I still have access to the old instance.
It happened on 1.8
The only emails sent were new_version emails. No password resets, no notifications.
Logs are often written to and flushed in different threads/processes, so I’m not sure if that can explain the stray pg_dump message.. but it’s certainly very suspicious.
I think the actual issue would perhaps also be that there’s no sanity checks when sending out a pile of emails in a queue. What if a password reminder is 3 months old? Should we assume that system time has changed or indeed that emails have failed to send for a considerable duration?
Our new server is now running the latest version of Discourse, but it would of course be a shame if unattended backups once again get stuck. https://community.learningequality.org/