There is seemingly no timeouts for backups. So an automatic backup was stuck since September
Once canceled, all our users received email digests and password resets aggregated since September.
Here’s the log:
[2019-09-26 03:35:25] pg_dump: creating INDEX "public.idx_tag_users_ix1"
[2019-09-26 03:35:25] pg_dump: creating INDEX "public.idx_tag_users_ix2"
[2019-09-26 03:35:25] pg_dump: creating INDEX "public.idx_topic_id_public_type_deleted_at"
[2019-09-26 03:35:25] pg_dump: creating INDEX "public.idx_topics_front_page"
[2019-09-26 03:35:25] pg_dump: creating INDEX "public.idx_topics_user_id_deleted_at"
[2019-09-26 03:35:25] pg_dump: creating INDEX "public.idx_unique_actions"
[2019-09-26 03:35:25] pg_dump: creating INDEX "public.idx_unique_flags"
[2019-09-26 03:35:25] Finalizing backup...
[2019-09-26 03:35:25] Creating archive: our-community-2019-09-26-033520-v20171214040346.tar.gz
[2019-09-26 03:35:25] Making sure archive does not already exist...
[2019-09-26 03:35:25] pg_dump: creating INDEX "public.idx_unique_post_uploads"
[2020-01-24 16:02:39] Backup process was cancelled!
[2020-01-24 16:02:39] Notifying 'system' of the end of the backup...
What version of Discourse are you using? I assume you didn’t run any upgrades since September, otherwise I’m quite sure that would have stopped the backup.
Thanks for reading and responding! I noticed the issue while inspecting an old server, which has now been replaced. I still have access to the old instance.
It happened on 1.8
The only emails sent were new_version emails. No password resets, no notifications.
Logs are often written to and flushed in different threads/processes, so I’m not sure if that can explain the stray pg_dump message… but it’s certainly very suspicious.
I think the actual issue would perhaps also be that there’s no sanity checks when sending out a pile of emails in a queue. What if a password reminder is 3 months old? Should we assume that system time has changed or indeed that emails have failed to send for a considerable duration?
Our new server is now running the latest version of Discourse, but it would of course be a shame if unattended backups once again get stuck. https://community.learningequality.org/