Mailing list mode emails not being sent for some first posts

Bug
,
1

Thanks Cameron, great to hear! I’ll schedule this topic to close after
another month, if it is closed and another issue comes up feel free to
open another one :slight_smile:

I may have found one of my discrepancies.

I have the email for I can't install libraries - #2 by MRAB - Python Help - Discussions on Python.org
but not the email for the first post. I’ve appended the header as
received here below.

Could you see what your records say about email for the first post
please? It should have message-id <discourse/post/72724@discuss.python.org>
but that message-id does not show in my mail filer logs here.

Thanks,
Cameron Simpson cs@cskk.id.au

The headers on the second post are these:

Return-Path: <incoming+verp-7a67e4e43670863710e0bebf82466d20@python1.discoursemail.com>
X-Original-To: cs@cskk.id.au
Delivered-To: cs@cskk.id.au
Received: from mx-out-01b.sea2.discourse.cloud (mx-out-01b.sea2.discourse.cloud [184.105.176.58])
         (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits))
         (No client certificate requested)
         by mail.cskk.id.au (Postfix) with ESMTPS id 2F85142106
         for <cs@cskk.id.au>; Tue,  1 Nov 2022 02:06:11 +0000 (UTC)
Received: from localhost.localdomain (unknown [IPv6:2602:fd3f:3:207:0:242:ac11:b])
         by mx-out-01b.sea2.discourse.cloud (Postfix) with ESMTP id 66C61100455
         for <cs@cskk.id.au>; Tue,  1 Nov 2022 02:06:09 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=discoursemail.com;
         s=sea2; t=1667268369;
         bh=dqc9opBcaL3Sz5b7apIIcM2hRhdys9o0z9IGxAzk4i4=;
         h=Date:From:Reply-To:To:In-Reply-To:References:Subject:
         List-Unsubscribe:List-ID:List-Archive;
         b=nuQKrBE6H2us2CerZzzR2yDWOnGG7R4kRgmH21VHpzOh/onYgIPtSFsguZ1nDYXXr
         t/+2pfIXGpl/wRRPuAppWH7IOLzJt5bv/+QbAflcKduBbATnRoVnEokbVLRJjg7fmS
         vvYcR3BPcaWV5LbVhw3/9F32/XqDtyHvb04kS0lmS230xvEN+IVJ1F43fAq+OOfp2j
         xboY8fyRJJ245gCTO07Bx/AE5tdrJ5sFrOeTmlz0gn9NJaKYgQxEOxqtWSulX8IyT0
         erHwnaEg6prJFWjGtxrhiZfe5EtqMVop5XaTI4hbxD++9ndJIHS5rCNn9jjurwWB4u
         5UToaINnQoDsQ==
1 Like
2

Followup. It looks like Discourse truncated my header paste.
I’ll put them in here via the web forum instead of via email:

It might be a display issue. The raw text shown when I edit this post has the full headers. I’ve uploaded them as a file. I think. - Cameron
hdrs.txt (2.7 KB)

1 Like
3

@martin Did you see the post above? Looks like a first-post didn’t go to email. - Cameron

1 Like
4

Yes thank you Cameron, my bad I forgot to reply, I’ve just been swamped with other work this week and I also had a day off midweek. I have a reminder set to go through this on Monday and respond properly then.

1 Like
5

No worries, just checking. - Cameron

2 Likes
6

Confirmed that is the correct Message-ID in our email logs for python. I think maybe what has happened in this case is that the OP was posted under the “Packaging” category and then not long after changed to the “Python Help” category, which could have affected the sending of emails. It looks like only 5 people were emailed for the OP, and 30 were emailed (including you) for the second post. Maybe just keep an eye out and see if this happens again and we can see if a pattern develops – there are many short circuits in our email system to prevent emails being sent in certain cases, and we may have just run into one here.

1 Like
7

[quote=“Cameron Simpson, post:94, topic:233499,
username:cameron-simpson”]
It should have message-id <discourse/post/72724@discuss.python.org>
but that message-id does not show in my mail filer logs here.
[/quote]

Confirmed that is the correct Message-ID in our email logs for python.

Good.

I think maybe what has happened in this case is that the OP was posted
under the “Packaging” category and then not long after changed to the
“Python Help” category, which could have affected the sending of
emails. It looks like only 5 people were emailed for the OP, and 30
were emailed (including you) for the second post. Maybe just keep an
eye out and see if this happens again and we can see if a pattern
develops – there are many short circuits in our email system to
prevent emails being sent in certain cases, and we may have just run
into one here.

That’s interesting. I’ve got mailing list mode turned on, and AFAIK the
Packaging category isn’t muted for me. Should I not have received an
email for the OP?

Is it fruitful to use me as a known unemailed case to examine the short
circuits?

Cheers,
Cameron Simpson cs@cskk.id.au

1 Like
8

Here’s another example where I received the second post but not the OP:

2 Likes
9

And another example where I received the second post and not the OP.

2 Likes
10

And yet another example: Why I can't do 'conda activate <env>' in JupyterLab? - Python Help - Discussions on Python.org where I have the second post but not the first. Although the OP is from August.

1 Like
11

And another: os.scandir should have an option to also yield the path itself - Ideas - Discussions on Python.org

For this thread I only have post 6.

… and @martin can we undo the autoclose for this topic please?

2 Likes
12

Apologies, lots going on at the moment internally. Thanks for sending through more examples, I will try to find a pattern from these.

Do you mean this one on meta? Or the one you have linked to os.scandir should have an option to also yield the path itself - Ideas - Discussions on Python.org ? Neither seem to have autoclose on.

14

Apologies, lots going on at the moment internally. Thanks for sending
through more examples, I will try to find a pattern from these.

No worries. And no hurry.

Do you mean this one on meta?

This thread on meta. Yesterday(?) it was saying it would autoclose in 2
weeks, from your setting here:

But that seems now to be turned off, so all good.

Cheers,
Cameron Simpson cs@cskk.id.au

1 Like
15

Sorry, that was me. :slightly_smiling_face: I removed the timer after I read your post asking for it to be turned off. I popped a little Like in as an acknowledgement, but it may have been too subtle a nod. :slightly_smiling_face:

3 Likes
16

Too subtle for me. As you might guess from the thread I’m an email-first user.

But also, I read Snowcrash and didn’t notice the main character’s name was a pun. So you’re fighting an uphill battle with me if you’re going to use subtlety :slight_smile:

Cheers,
Cameron

5 Likes
17

I am using this one as an example now. So far nothing has jumped out at me from stepping through the code. No one got an email for the OP of that topic according to our EmailLog table but it’s not clear why, since if I run through the same steps as discourse/app/jobs/regular/notify_mailing_list_subscribers.rb at 23ac0cf8f1f5d7aede3991c6947530dc1af7f7a2 · discourse/discourse · GitHub and stop short of sending the emails, it looks like the correct thing will happen. There are no records for the OP in SkippedEmailLog either. I suppose it’s possible that the job failed and then failed on every subsequent retry, but I doubt that.

I’ve just tweaked some things on our hosting for python to allow me to see more detailed logging information for these jobs. Please keep an eye out and let me know if you notice this happening again within the next couple of weeks, hopefully I will be able to hunt down the issue.

1 Like
19

[quote=“Cameron Simpson, post:102, topic:233499,
username:cameron-simpson”]
And another example where I received the second post and not the OP.
[/quote]

I am using this one as an example now. So far nothing has jumped out at me from stepping through the code. No one got an email for the OP of that topic according to our EmailLog table but it’s not clear why, since if I run through the same steps as discourse/app/jobs/regular/notify_mailing_list_subscribers.rb at 23ac0cf8f1f5d7aede3991c6947530dc1af7f7a2 · discourse/discourse · GitHub and stop short of sending the emails, it looks like the correct thing will happen. There are no records for the OP in SkippedEmailLog either. I suppose it’s possible that the job failed and then failed on every subsequent retry, but I doubt that.

Right. I can’t see anything wrong with that code either, unless there
are weird issues with the NOT EXISTS stuff. But it looks ok to my
naive eye.

I’ve just tweaked some things on our hosting for python to allow me to see more detailed logging information for these jobs. Please keep an eye out and let me know if you notice this happening again within the next couple of weeks, hopefully I will be able to hunt down the issue.

Thank you. I’ll keep you posted as I see new items.

Cheers,
Cameron Simpson cs@cskk.id.au

1 Like
20

Another example today: Printing only duplicates from a list - Python Help - Discussions on Python.org

I received the second post but not the first:

[~/var/log/mailfiler]fleet2*> agf discourse/post/78942@discuss.python.org *log
main.log
5551250:2022-12-24 08:25:19,937 Level 25 mailfiler: monitor: +spool:   +spool/new/1671830717.M998823P71474Q174.fleet2.l <discourse/post/78942@discuss.python.org>
5552015:2022-12-24 08:25:37,173 Level 25 mailfiler: monitor: +spool-in:   +spool-in/new/1671830719.#5973M944325P789 <discourse/post/78942@discuss.python.org>

spool-in.log
3606809:2022-12-24 08:25:37,173 Level 25   +spool-in/new/1671830719.#5973M944325P789 <discourse/post/78942@discuss.python.org>

spool.log
3243627:2022-12-24 08:25:19,937 Level 25   +spool/new/1671830717.M998823P71474Q174.fleet2.l <discourse/post/78942@discuss.python.org>
[~/var/log/mailfiler]fleet2*> agf discourse/post/78934@discuss.python.org *log
[~/var/log/mailfiler]fleet2*>

shows grepping for the second post message-id in my logs and then the first post message-id, whcih is not seen.

Cheers,
Cameron

21

And another today: Serve HTML from extensionless URLs in http.server - #2 by Rosuav - Ideas - Discussions on Python.org

The timing coincidence (this and the preceeding instance are both this morning, after a period of apparent quiet) makes me wonder if we’re not looking at a plain old “should this go out?” logic bug but some wider circumstance where that doesn’t work, eg a restart of the Discourse instance or some other resource issue.

Similar grep of the reply message-id and then the in-reply-to (OP) message-id:

[~/var/log/mailfiler]fleet2*> agf discourse/post/78998@discuss.python.org *log
main.log
5553805:2022-12-24 08:42:48,500 Level 25 mailfiler: monitor: +spool:   +spool/new/1671831759.M979332P32679Q9.fleet2.l <discourse/post/78998@discuss.python.org>
5553809:2022-12-24 08:42:48,522 Level 25 mailfiler: monitor: +spool-in:   +spool-in/new/1671831768.#6332M514534P789 <discourse/post/78998@discuss.python.org>

spool-in.log
3607754:2022-12-24 08:42:48,522 Level 25   +spool-in/new/1671831768.#6332M514534P789 <discourse/post/78998@discuss.python.org>

spool.log
3244519:2022-12-24 08:42:48,500 Level 25   +spool/new/1671831759.M979332P32679Q9.fleet2.l <discourse/post/78998@discuss.python.org>
[~/var/log/mailfiler]fleet2*> agf discourse/post/78967@discuss.python.org *log
[~/var/log/mailfiler]fleet2*>

Cheers,
Cameron

2 Likes
22

Thanks for giving these two examples Cameron. It definitely could be something like that, perhaps something to do with our background jobs in Sidekiq erroring or having something else happen to them since they are stored in Redis.

Confirmed there is no EmailLog record for your user in that topic for the OP. Nothing more is standing out to me so far though, this is a very frustrating bug to track down the cause of.

Unfortunately this was only for 1 month so it since turned off again via an automated system, I just turned our detailed logging on again for 3 months then we can see if any more examples come in for Python that I can look at.

1 Like