Troubleshooting omitted content from incoming emails

[Simon_Manning]

I’m looking into some posts that have content omitted from them after importing emails from a mailing list. Some emails wrap headings/sections with “~~~~~~~~~~~~~~~~” lines and the first occurrence of that on its own line, plus everything below gets omitted from the Discourse post.

Inspecting the email on the post, I can see the original base64 encoded email and switching to the Text tab, I can see that it has decoded that without omitting anything.

./launcher enter app
rails c
Post.find_by(id: 154678)
=> #<Post:0x000055b8da925e08
...

Using rails, I can see that the raw column stops immediately before the first “~~~~~~~~~~~~~~~~” and the cooked column obviously contains the same as raw but appropriate for HTML. I’ve tried changing some settings related to trimming emails but I’m unsure how to test if those make a difference.

As I understand it, using Rebuild HTML on the post or issuing Post.find_by(id: 154678).rebake! in rails will take the contents of raw and build HTML, setting cooked. I don’t know if it will take the contents of raw_email and process that again though, so I’m unsure how to test if the changed settings are having any effect.

How should I tell Discourse to reprocess raw_email of the post if rebake! doesn’t do that, so that I can be sure I’m troubleshooting correctly?

If you have any thoughts on how to prevent the content from being omitted, those are of course welcome as well.

[Simon_Manning]

Continuing to search for related topics, I stumbled upon the Advanced Test feature under Emails in Admin. I suspected this would be the thing doing it but using that feature, I confirmed that it is the trim incoming emails setting causing this, described as Trim part of the incoming emails that isn't relevant.

With that setting turned off, I get the correct results for those emails but rebaking doesn’t appear to reprocess the emails so the question remains: How do I tell Discourse to reprocess posts from their incoming emails?

[Simon_Manning]

Digging through the repository looking for any code that deals with handling emails, I eventually found the rake task posts:refresh_emails which takes an optional topic ID. Using the following, as the name implies, reprocessed the emails and also rebaked all the posts in the specified topic.

cd /var/discourse
./launcher enter app
rake posts:refresh_emails['24961']

This provided the posts in that topic untrimmed as expected with trim incoming emails disabled. That did expose another issue I need to deal with before I can proceed but that’s not for this topic. To refresh all emails across all topics, I will be able to use the rake task as follows.

cd /var/discourse
./launcher enter app
rake posts:refresh_emails