Missing space characters in activity summary email text

[ToddZ]

I’ve just sent myself some preview summaries, since I don’t normally get them.

I’m seeing some random—but consistent—missing space characters in headings and text within the email. These spaces are not missing in the forum content, but the same ones are consistently dropped in multiple generated preview summaries, as viewed in various email clients.

I’ve tried deleting and re-adding the original spaces to no effect.

Excerpts:

I’ve reviewed the summaries I receive from some other Discourse forums, and I don’t see this anywhere else.

Has anyone else seen this, or have an idea what’s happening?

[supermathie (Michael Brown)]

Is it possible this could be a font / display problem? Have you checked the underlying raw content?

[ToddZ]

Hm. I’m not sure how to diagnose a font/display problem. The emails render the same in multiple mail clients and browsers across Windows and Linux.

I’ve attached .json to the forum post URLs, and there’s nothing off about the “topic_slug” or “cooked” content…

Is there something else I could be checking for in the raw content?

[supermathie (Michael Brown)]

@ToddZ

Hm. I’m not sure how to diagnose a font/display problem. The emails render the same in multiple mail clients and browsers across Windows and Linux.

You’ll need to check the raw email rather than the post.

[ToddZ]

Ok - l looked at the raw email, and where the HTML version is missing spaces, the text version has the correct spaces. However, the text version is missing other space characters. There’s no rhyme or reason to it.

Maybe it could be a character-encoding glitch in the course of copy/pasting the affected topics from a legacy platform…? EDIT: Nope. It’s continuing with current posts, and also with other emails—not just the summary.

More recent Discourse summaries featuring current posts are not having the same issue, so I won’t worry much unless I see it continue. EDIT: It’s continuing.

(Side note: just to watch for this stuff, I now wish I could force a comprehensive summary sent to my admin account on a daily basis – regardless of me being constantly logged in.)

[supermathie (Michael Brown)]

Could you forward one of those emails to me as an attachment?

EDIT: done

[supermathie (Michael Brown)]

OK, here’s what I see in the raw text email:

[Mis/Disinformation starts to overwhelm civilization][2]

 The dark side of generative AI is that it enables the production of misinformation (due to confabulation) and disinformation (i.e. deliberate production of fake news to achieve an end) at industrial scale.   Renderingof web pages in the style of authoritative sources is straighforward, and progress in deep fakes will make video story complements easier.  Asidefrom Vinge’s clouds of fake information to hide information (Rainbows End), which I don’t believe posed a solution, have any SF authors thought about this and how it might be tackled?

note:

• to overwhelm
• a paperback
• Renderingof
• Asidefrom

and in the HTML version:

<a href="https://forum.tasat.org/t/mis-disinformation-starts-to-overwhelm-civilization/66" style="text-decoration: none; font-weight: bold; color: #006699;; font-weight:400;line-height:1.3;margin:0;padding:0;text-decoration:none">
<strong>Mis/Disinformation starts tooverwhelm civilization</strong>
…
Rendering of web pages in the style of authoritative sources is straighforward, and progress in deep fakes will make video story complements easier.  Aside from

note:

• tooverwhelm
• apaperback
• Rendering of
• Aside from

In the rawest (i.e. encoded) form, these errors are still there:

[Mis/Disinformation starts to overwhelm civilization][2]
            
 The dark = 
side of generative AI is that it enables the production of misinformation (=
due to confabulation) and disinformation (i.e. deliberate production of fak=
e news to achieve an end) at industrial scale.   Renderingof web pages in t=
he style of authoritative sources is straighforward, and progress in deep f=
akes will make video story complements easier.  Asidefrom Vinge=E2=80=99s c=
louds of fake information to hide information (Rainbows End), which I don=
=E2=80=99t believe posed a solution, have any SF authors thought about this=
 and how it might be tackled?
Ken

                                  <a href=3D"https://foru=
m.tasat.org/t/mis-disinformation-starts-to-overwhelm-civilization/66" style=
=3D"text-decoration: none; font-weight: bold; color: #006699;; font-weight:=
400;line-height:1.3;margin:0;padding:0;text-decoration:none">
           =
                         <strong>Mis/Disinformation starts tooverwhelm civi=
lization</strong>

these are not in the raw/cooked:

000000d0: 5265 6e64 6572 696e 6720 6f66 2077 6562  Rendering of web
000000e0: 2070 6167 6573 2069 6e20 7468 6520 7374   pages in the st
000000f0: 796c 6520 6f66 2061 7574 686f 7269 7461  yle of authorita
00000100: 7469 7665 2073 6f75 7263 6573 2069 7320  tive sources is 
00000110: 7374 7261 6967 6866 6f72 7761 7264 2c20  straighforward, 
00000120: 616e 6420 7072 6f67 7265 7373 2069 6e20  and progress in 
00000130: 6465 6570 2066 616b 6573 2077 696c 6c20  deep fakes will 
00000140: 6d61 6b65 2076 6964 656f 2073 746f 7279  make video story
00000150: 2063 6f6d 706c 656d 656e 7473 2065 6173   complements eas
00000160: 6965 722e 2020 4173 6964 6520 6672 6f6d  ier.  Aside from

Not that I didn’t believe you

So… spaces are being occasionally removed from the email body, be it the text part or the HTML part. And not in the same places!

I posit that these errors could have been introduced at one of four places:

• in Discourse, generating the email
• transmitting the email to a the email submission server
• transmitting the email to an intermediate/end server
• delivering to the user mailbox

It’s probably easiest to start at the beginning.

Can you have Discourse submit mail to a local MTA where you can inspect it in the queue prior to the MTA sending it off to your “actual” email delivery server?

[ToddZ]

Thanks for the analysis, Michael!

@supermathie

Can you have Discourse submit mail to a local MTA where you can inspect it in the queue prior to the MTA sending it off to your “actual” email delivery server?

I’m not an advanced email admin – I’m running the typical recommended self-installation, with actual outbound email via MailerSend.net, and have carefully configured DKIM/DMARC etc. to a working state. From what I’m reading, incorporating a local MTA like sendmail or Postfix is an advanced move that’s discouraged in most cases… I’m a little apprehensive about mucking around and possibly disrupting a functioning pipeline.

Is there an easily revertible, troubleshooting kind of MTA implementation I might consider?

[ToddZ]

As noted in edits above, this problem continues with current user-posted content, not just admin cut-&-pasted content — and is now observed with summary, user_replied, and user_posted emails.

MailerSend support confirmed the space characters are missing when they receive the request from Discourse — so it would seem the error is with Discourse generation of the email…?

FWIW, space characters are not missing when previewing a generated summary — only when they’re received as emails.

At the same time, I’m having this issue with summary emails, reported by others starting in February:

Multiple repeated summary mail entries

I’m seeing the same behavior: “Popular Posts” and “New for you” topics being repeated. In this case one Popular Post is tripled, another is doubled: [image] Self-hosted standard installation here, current at 3.4.0.beta2-dev (f2121cc437). I see this report dates back to Feb. – anything happening? Any further details I could provide to help?

These repeated posts are present in the generated summary previews.

---

I’ve done a command-line update & rebuild to see if that would flush out any kinks, but it had no effect.

If these things aren’t happening to everyone who’s up-to-date on the tests-passed branch, what might I look into in my setup?