User ruperty is experiencing import failures during a Discourse migration, primarily due to “Failed to map post” and “Parent message doesn’t exist” errors, affecting ~1000 of 66,909 messages. The root cause was traced to a bug in Discourse’s receiver.rb that threw undefined method 'hex' for nil:NilClass, which gerhard fixed via commit e84d88d. After upgrading, ~800 posts succeeded, but ~200 still failed due to missing Date headers or Discourse::InvalidAccess errors. ruperty confirmed missing dates in some mbox files and attempted to patch them, but changes didn’t reflect until gerhard advised deleting index.db to force re-indexing. ruperty also discovered that inline attachments with filenames lacking extensions (e.g., Conflict (was ... long live Wil) may trigger InvalidAccess, suggesting file extension validation issues. The importer uses In-Reply-To and References headers for threading unless group_messages_by_subject is enabled — which ruperty had set to true to enable grouping. gerhard recommended using index_only: true in settings.yml to safely edit index.db before full import. ruperty ultimately chose to edit mbox files directly, but only after deleting index.db did the updates take effect. The remaining failures are likely due to malformed attachments or missing metadata, requiring manual cleanup or code adjustments to handle edge cases.
I have 28 years of posts, with a folder for each year, with a mbox file for each month. There are 66909 messages in the mbox’s. The import shows 65895. Is the difference of 1014 due to the failures indicated in the import output?
The posts were converted from Eudora mbx’s to standard mbox’s with Aid4Mail.
For the “Parent message doesn’t exist” error I see 421 instances.
For the “Failed to map post” error I see 149 instances.
My split_regex is “^From .@. [0-9]{4}” which seems suitable for the headers such as,
I’ve seen the same error last week while working on a different problem. I’m going to fix it this week and will post an update here once it’s fixed.
Those are just warnings and probably appear due to the “Failed to map post” errors. It happens when a message references a post that doesn’t exist. I’m quite sure that fixing the other problem will fix most if not all of these warnings.
You could take a look at the index.db the import script creates. It’s an SQLite3 database. You could run the following query to see what the parser is working with. It selects the messages for the two Message-IDs you posted.
SELECT *
FROM email
WHERE msg_id IN ('bbe76bf7a9cab5a2ec2a06e6ef453555', '23a86e52-71ba-7435-1c9c-c4f2a134b90d@mmtaylor.net')
I guess the email_date and raw_message columns will be the most interesting columns for you. Maybe you can find what’s confusing the email parser…
For the first the date is null, and I see there is no date for that message in the mbx. I note that the reply (with :Re) appears before the “initial” message, which is why I thought the date was not missing. Does the import take the parent messages as the first one in the file with that subject?
Is the email date taken from the “Date:” line, such as?
Date: Wed, 25 Mar 1992 12:23:00 GMT
I’ll see if I can repair those with missing dates.
No, it uses the In-Reply-To and References header to match and sort by Message-ID unless you changed the importer’s group_messages_by_subject setting to true.
Yes.
My best guess is that there’s a problem with one of the attachments. Maybe the file extension isn’t allowed?
I did set group_messages_by_subject setting to true as without it there was no grouping at all.
That message has two inline images:
Content-Type: application/octet-stream;
name=“Conflict (was … long live Wil”
Content-Transfer-Encoding: base64
Content-Disposition: inline; filename=“Conflict (was … long live Wil”
Yes, that works. I did the same in the past. I’d recommend setting index_only to true in settings.yml, so that it doesn’t start to import immediately after indexing the messages. You can make all the needed changes in the database after the indexing has finished. Then change index_only to false again and restart the import.
I think I am misunderstanding something there. Hasn’t the indexing already been done, as index.db is already built?
I have transferred index.db to my desktop. I was going to update the dates then transfer index.db back to the server and then run import again. Is that not right?
I decided to go the route of editing the mbox files, adding a “Date” line, e.g. “Date: Wed, 25 Mar 1992 17:43:06”. I transferred the updated files and reran the import, twice. However, the email_date field was not updated.