UTF-8 / RTL e-mail reply issues


(Michael - DiscourseHosting.com) #1

One of our customers runs a Hebrew forum and is having massive issues with the reply-via-email functionality. We suspected many different things around the specific email provider that was being used but eventually the problem turned out to be about the content of the messages.

In sidekiq log we see the following

2013-10-28T11:33:10Z 537 TID-13hiec WARN: {"retry"=>false, "queue"=>"default", "class"=>"Jobs::PollMailbox", "args"=>[1382959920.0, 1382959980.0], "jid"=>"1f328c5953d6c04407283630", "enqueued_at"=>1382959988.7867837}
2013-10-28T11:33:10Z 537 TID-13hiec WARN: incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)
2013-10-28T11:33:10Z 537 TID-13hiec WARN: /var/www/discourse/lib/email/receiver.rb:96:in `block in discourse_email_parser'
/var/www/discourse/lib/email/receiver.rb:93:in `each'
/var/www/discourse/lib/email/receiver.rb:93:in `each_with_index'
/var/www/discourse/lib/email/receiver.rb:93:in `discourse_email_parser'
/var/www/discourse/lib/email/receiver.rb:31:in `process'
/var/www/discourse/app/jobs/scheduled/poll_mailbox.rb:26:in `block (2 levels) in poll_pop3s'
/usr/lib/ruby/1.9.1/net/pop.rb:665:in `each'
/usr/lib/ruby/1.9.1/net/pop.rb:665:in `each_mail'
/var/www/discourse/app/jobs/scheduled/poll_mailbox.rb:25:in `block in poll_pop3s'
/usr/lib/ruby/1.9.1/net/pop.rb:533:in `start'
/usr/lib/ruby/1.9.1/net/pop.rb:406:in `start'
/var/www/discourse/app/jobs/scheduled/poll_mailbox.rb:20:in `poll_pop3s'
/var/www/discourse/app/jobs/scheduled/poll_mailbox.rb:14:in `execute'
/var/www/discourse/app/jobs/base.rb:128:in `block (2 levels) in perform'

After an english email was received, this stopped.

Eventually, a private message which had only english characters, even
though the mail’s subject included the website title which has hebrew
characters were read and arrived in the right place
That also triggered reading another email message, which was read badly.
(before that it was laying in the mailbox for 13h).

Anyone?

Running v0.9.7.1 on ruby 1.9.3p0


(Robin Ward) #2

I was wondering if you could forward me the hebrew email that caused the error? I’ll then be able to debug and push out a fix :smile:

My email is my first name dot last name at google’s email server.


(Amit Friedmann) #3

Hi @eviltrout ,
I forwarded you an email correspondent that reproduces the exact problem (reproduced it right now) along with all the configuration I’ve been using as screenshot. @michaeld I have sent it to you as well.

Cheers.


(Robin Ward) #4

I’ve just committed a fix for this to master. Should work now!

Let us know if you have further problems.


(Robin Ward) #5