Could one solution be for an option to rewrite the message-id to reflect the original id as created by the poster’s mail client plus an additional identifier, eg the mailing list To: address that Discourse already looks for? This would mean if a post came in from mailing list B but it referenced a message-id header that was also in a post from mailing list A it would get correctly filed by Discourse?
Eg. Original email sent to two mailing lists A and B simultaneously has unique message-id ‘1234567890gmail’.
Discourse receives two copies forwarded to the generic incoming Discourse address. First message is filed based on To: address being to List A and message-id gets extra characters appended by Discourse to become ‘1234567890gmailListA’. The second message is not filed as it looks like a duplicate. (I think this is current behaviour).
Someone on List B replies using the To: address for List B. Discourse gets a copy and spots there is an in reply to header that contains ‘1234567890gmail’. Discourse, seeing the email came in with a To: line reflecting List B appends characters to the end to make the in reply to header look like ‘1234567890gmailListB’. It then goes on to file this as a new message in the correct category as it no longer is tied to the message that was filed against List A.
This approach could work for transfering archives too where the same message-id might be found in several mailing lists if a user sent the message to several lists at the same time. At the point of import all message-id fields (and in reply to) could have unique text appended linked to the named mailing list to stop a sort of cross talk between archives when messages are indexed.