pfaffman reported an infinite loop in email_reply_trimmer.rb that caused issues with importing large emails. The issue, which occurred around line 175, was difficult to replicate and debug due to the size of the messages. After an upgrade, the infinite loop seemed to resolve, but emails imported from Nabble were not being parsed correctly.
gerhard was able to reproduce the problem and created a pull request to fix it. However, pfaffman still experienced issues with parsing long messages (> 500K bytes), which took over 30 seconds to parse. gerhard requested examples of these long messages to further investigate.
In the meantime, pfaffman implemented a workaround by truncating posts at 50K bytes, as the large messages were not necessary for the import process.
this is a 
, but I think only developers will care about it. There’s an infinite loop in one of these replacements. I found it while running the nabble importer. I banged at it a bit before just commenting it out to solve my immediate problem.
It looks like @zogstrip understands this. It seems that sometimes 183 gets stuck. I was wondering why these 170K messages need an entire holiday weekend to import.
Do you have an example that fails?
I just assumed that it’d be self evident, because those regexes are so simple.
Yes. I did get the script to spit out the errant one,but by the time I was sure this was the problem it was gone. I’ll try to run it again Real Soon Now.
And now it seems that maybe an upgrade made that problem go away, but now Nabble seems not to import any messages that have been emailed in (as opposed to created in the Nabble web interface).
If there’s some easy way to decode a MIME-encoded email, I’d love to know about it.
Not sure about the “MIME-encoded email”, but at least I can reproduce the problem in EmailReplyTrimmer. I’ll fix it tomorrow. This is now fixed.
This seems to improve things, though I still have messages that are taking >30 seconds to parse a message. They all look Really Long (> 500K bytes?).
Do you want them?
Sure, send me a few of those. I’ll take a look.
My solution was to truncate the posts at 50k. It turns out they didn’t want that list anyway (a bunch of cvs reports).
I wrote code that I think saves the bad messages to files, but really, who wants 500k posts?