Email processing stopped after getting: incompatible character encodings: UTF-8 and ASCII-8BIT


(Dirk Jahnke) #1

It happens occasionally, but unfortunately the processing of received emails stopped. No further emails are processed, they stay in the inbox untouched. The error log shows this message:

incompatible character encodings: UTF-8 and ASCII-8BIT

The details are:

/var/www/discourse/lib/email/processor.rb:107:in log_email_process_failure’
/var/www/discourse/lib/email/processor.rb:25:inrescue in process!’
/var/www/discourse/lib/email/processor.rb:15:in process!’
/var/www/discourse/lib/email/processor.rb:11:inprocess!’
/var/www/discourse/app/jobs/scheduled/poll_mailbox.rb:25:in process_popmail’
/var/www/discourse/app/jobs/scheduled/poll_mailbox.rb:36:inblock (2 levels) in poll_pop3’
/usr/local/lib/ruby/2.3.0/net/pop.rb:689:in block in delete_all’
/usr/local/lib/ruby/2.3.0/net/pop.rb:688:ineach’
/usr/local/lib/ruby/2.3.0/net/pop.rb:688:in delete_all’
/var/www/discourse/app/jobs/scheduled/poll_mailbox.rb:35:inblock in poll_pop3’
/usr/local/lib/ruby/2.3.0/net/pop.rb:532:in start’
/var/www/discourse/app/jobs/scheduled/poll_mailbox.rb:34:inpoll_pop3’
/var/www/discourse/app/jobs/scheduled/poll_mailbox.rb:16:in execute’
/var/www/discourse/app/jobs/base.rb:154:inblock (2 levels) in perform’

We have discourse version v1.7.0.beta4 +1 running, but observed this behaviour even before.

We can solve this by:

  • removing the mail from the inbox manually, that has caused this error
  • restarting discourse, otherwise the email processing does not seem to start again

The email content is the following (I removed/changed any personalized content):

Received: from [12.34.56.78] (helo=mx11.ispgateway.de)
by baradla.ispgateway.de with esmtp (Exim 4.68)
(envelope-from kind.user@users-mail-provider.tld)
id 1bjuPl-00078r-6G; Tue, 13 Sep 2016 22:34:45 +0200
Return-path: kind.user@users-mail-provider.tld
X-Envelope-To: replies+93a6ad869cf52da60516d912fdc38ed5@replies-to-our-forum.tld
Received: from [12.34.136.195] (helo=dd15524.someserver.com)
by mx11.ispgateway.de with esmtps (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256)
(Exim 4.84)
(envelope-from kind.user@users-mail-provider.tld)
id 1bjuPl-00008H-1B
for replies+93a6ad869cf52da60516d912fdc38ed5@replies-to-our-forum.tld; Tue, 13 Sep 2016 22:34:45 +0200
Received: from [127.0.0.1] (ip1f10d11d.dynamic.kabel-deutschland.de [12.34.56.78])
by dd15524.kasserver.com (Postfix) with ESMTPA id 7B3E342C017E
for replies+93a6ad869cf52da60516d912fdc38ed5@reply-to-our-forum.tld; Tue, 13 Sep 2016 22:34:43 +0200 (CEST)
Subject: Re: [Our Forum] [Sonstiges/Moderatoren / Admin Bereich] PNR (Point
of no Return)?
To: Our Forum replies+93a6ad869cf52da60516d912fdc38ed5@reply-to-our-forum.tld
References: topic/351@our-forum.tld topic/351/897@our-forum.tld
From: Kind User kind.user@users-email-provider.tld
Message-ID: aaf18f35-e873-a4ab-8d4d-d707d34ac0a4@users-email-provider.tld
Date: Tue, 13 Sep 2016 22:34:43 +0200
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:45.0) Gecko/20100101
Thunderbird/45.3.0
MIME-Version: 1.0
In-Reply-To: topic/351/897@our-forum.tld
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Antivirus: avast! (VPS 160913-3, 13.09.2016), Outbound message
X-Antivirus-Status: Clean
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
spamfilter14.ispgateway.de
X-Spam-Level:
X-Spam-Status: No, hits=-1.0 required=9999.0 tests=BAYES_40 autolearn=disabled
version=3.3.1
X-Spam-CMAETAG: v=2.1 cv=RdUeCjdv c=1 sm=0 tr=0
a=2Fy9FXaE3rn+nnIb8O4l/A==:17 a=IkcTkHD0fZMA:10 a=1oJP67jkp3AA:10
a=ZZnuYtJkoWoA:10 a=GW1xBdLrtEIA:10 a=EYHFl4OH7dEKSkrgIHAA:9
a=QEXdDO2ut3YA:10 a=iXjr25m-gE0A:10 xcat=Undefined/Undefined
X-Spam-CMAECATEGORY: 0
X-Spam-CMAESUBCATEGORY: 0
X-Spam-CMAESCORE: 0

Moin,

Am 13.09.2016 um 20:38 schrieb another user:

[other.user] other.user
some text including as
message, nothing special here

Some reply including “Umlaut” like ö and ä
but that’s it

This is all we got collect for this case. Is there anything we can do to prevent this situation? Or is this a bug? At least I would expect, that the email processing does not stop, but this message is skipped, if it causes severe problems.

Thank you and best regards,
Dirk


Setting company_domain stops email processing?
(Jeff Atwood) #2

Can you send the raw email content as an attachment to @zogstrip?


(Régis Hanol) #3

I tried using the raw email you sent me and it worked fine.

Is there any chances I could be made admin on your Discourse to check a couple of things?


(Dirk Jahnke) #4

Thanks for your feedback, thus the email did not cause the behaviour.

We use SSO with our membership-management system, thus I cannot give you a login as long as you are not a member :wink: – but would it be feasibly to share my login session e.g. using TeamViewer? Or is there any other possiblity?


(Dirk Jahnke) #5

Details sent by direct message to @zogstrip
The problem occured again after ~2 weeks, thus not very often.


(Dirk Jahnke) #6

It just came again, solving it starts to become a problem. I needed to rebuild discourse, which means a downtime of 10-15 minutes. It feels like fishing as a blind. I do not know, what really helps, so far I start with just restarting the docker container, then setting the error log entries to solved, removing the email that seems to cause the problem, always with restarting the container. Finally I call the ./launcher rebuild app – which takes quite some time but seems to help.

This is the error log:

Message (2 copies reported)

Job exception: incompatible character encodings: UTF-8 and ASCII-8BIT
Backtrace

/var/www/discourse/lib/email/processor.rb:107:in `log_email_process_failure'
/var/www/discourse/lib/email/processor.rb:25:in `rescue in process!'
/var/www/discourse/lib/email/processor.rb:15:in `process!'
/var/www/discourse/lib/email/processor.rb:11:in `process!'
/var/www/discourse/app/jobs/scheduled/poll_mailbox.rb:25:in `process_popmail'
/var/www/discourse/app/jobs/scheduled/poll_mailbox.rb:36:in `block (2 levels) in poll_pop3'
/usr/local/lib/ruby/2.3.0/net/pop.rb:689:in `block in delete_all'
/usr/local/lib/ruby/2.3.0/net/pop.rb:688:in `each'
/usr/local/lib/ruby/2.3.0/net/pop.rb:688:in `delete_all'
/var/www/discourse/app/jobs/scheduled/poll_mailbox.rb:35:in `block in poll_pop3'
/usr/local/lib/ruby/2.3.0/net/pop.rb:532:in `start'
/var/www/discourse/app/jobs/scheduled/poll_mailbox.rb:34:in `poll_pop3'
/var/www/discourse/app/jobs/scheduled/poll_mailbox.rb:16:in `execute'
/var/www/discourse/app/jobs/base.rb:154:in `block (2 levels) in perform'
Env

hostname	forum-app
process_id	182
application_version	2a9331d061423f2b04d63ad7407f90fa6fd104cd
current_db	default
current_hostname	forum.fremo-net.eu
job	Jobs::PollMailbox
problem_db	default
opts

(Arash Dalir) #7

Hi
we have the same problem too and unfortunately we cannot give you access to the system either for the same reasons. I believe this problem should always occur when the incoming email contains UTF8 characters (written in UTF-8 Character-set). if that’s correct, then testing it would be really easy: it would be enough to sent an email containing for example german characters like Ö Ä Ü ß or maybe some persian/ arabic ones like پ چ س or something. Maybe, if I receive an answer via email to this thread, I can do the test and we might be able to reproduce this issue… :smiley:


(Sam Saffron) #8

Can you reproduce it here by replying to this email from your inbox?


(Arash Dalir) #9

To ensure this problem is really because of any utf8 character, i’ll first
send a message without any characters in it (i.e. this email), and then
send a second reply containing german and persian letters. If the theory is
right, then the first email should work and the second one should fail.


(Arash Dalir) #10

And now a second message containing german words like Gemütlichkeit,
Äußerungen,
And some persian ones:
درود، روز همگی خوش!
Let’s see what happens now…


(Arash Dalir) #11

So as seen here, my theory seems to be incorrect, both emails have been processed. Not sure if I should be happy or sad now :laughing:
A colleague of mine had the theory, that there is something wrong with the email headers, could that be something to think about?