IMAP support for group inboxes

Quick update here. I merged these two PRs last week. First, the fix so that tagging does not need to be enabled for archiving and deleting to work:

https://github.com/discourse/discourse/pull/10426

Second, I am now writing all IMAP logs to the database for easier inspection. A job will be run daily to delete logs > 5 days old:

https://github.com/discourse/discourse/pull/10435

10 Likes

This is super sweet! Thanks.

I should say, I had to create an application password in my google email account for discourse to get it to accept my username/password

4 Likes

Are you running 2-factor authentication on the Google end? If so, this is standard practice.

1 Like

Yes. I believe it was required in order for me to create the app password (though I don’t remember for sure)

This is noted in the OP :wink: Otherwise you have to jump through crazy hoops to make your Gmail account accept “Less secure apps”. At some point we will need to support OAuth for Google as that looks like the way things are headed.

3 Likes

Oops! Did I mention that my reading comprehension skills suck? :woozy_face:

4 Likes

It’s okay! It is an important point so I moved it to the top of the Gmail list and added some warning indicators. Thank you for trying to let others know about this to save them trouble :slight_smile:

5 Likes

I’m trying to understand why I’m failing to sync mails with a dovecot IMAP server (unfortunately without experience in ruby or rails).

Reading the imap_sync_logs the import daemon hangs in a loop:

UIDVALIDITY = 1612565899 does not match expected 0, invalidating IMAP cache and resyncing emails for mailbox INBOX

So I am wondering, if @group.imap_uid_validity shouldnt be updated in this case:

https://github.com/discourse/discourse/blob/f34fa999a2c8a4f3d0c477bef9401817c538b405/lib/imap/sync.rb#L47

After a manual update of this value, the import daemon hangs in another loop.

Dovecot does not support 'LABELS' in

https://github.com/discourse/discourse/blob/f34fa999a2c8a4f3d0c477bef9401817c538b405/lib/imap/sync.rb#L238

yielding a Net::IMAP::BadResponseError which, unfortunately, does not find its way into the logs.

Now I’m wondering

  • if dovecots support of custom flags is sufficient for bi directional sync with discourse
  • how to detect dovecot in lib/imap/providers/detector.rb
  • how to emulate the label feature in a dovecot provider

4 Likes

IMAP syncing for group inboxes is still an early stage feature that currently only supports Gmail. Theoretically you could write a dovecot provider, but I noticed a couple of weeks back that there are some places in the main code that use Gmail-only features that need to be moved into provider-specific code. So sorry about that, but you may want to wait until this is a little closer to primetime.

6 Likes

IIUC 'LABELS'is an non-standard feature of gmail.
I would like to understand the difference of google imap and standard IMAP.

This is my summery so far:

standard IMAP google IMAP
archived mails live in subfolders all mails live in one folder
archived massages are missing the “\INBOX”-label
custom flags are stored along system flags stores custom flags as labels

Would it be sufficient

  • to emulate labels in standard IMAP by splitting flags into system and custom flags
  • handle email_is_archived by the provider (for example add and test a “archived” flag)

to get standard IMAP syncing going?

This would affect this lines:

https://github.com/discourse/discourse/blob/f34fa999a2c8a4f3d0c477bef9401817c538b405/lib/imap/sync.rb#L295

https://github.com/discourse/discourse/blob/f34fa999a2c8a4f3d0c477bef9401817c538b405/lib/imap/sync.rb#L357

https://github.com/discourse/discourse/blob/f34fa999a2c8a4f3d0c477bef9401817c538b405/lib/imap/sync.rb#L381

This line seems strage to me:
https://github.com/discourse/discourse/blob/f34fa999a2c8a4f3d0c477bef9401817c538b405/lib/imap/sync.rb#L399

Any system flags will be overriden by discourses tags.
Does this handle all IMAP flags well (`\Answered, \Deleted, \Draft, \Flagged, \Recent and \Seen / $Forwarded, $MDNSent, $SubmitPending and $Submitted)?

1 Like

I am revisiting this after some time away.

I have not been able to figure out how to get the reply-to header to work properly. I can set up a group to sync with a particular gmail account but when I try to reply via email, the reply-to address is set to the configured notification email from discourse and not the IMAP address.

Is there something that I’m missing?

Just tested this with GoDaddy email provider but no success. With Gmail works great sadly we want to use our own domain. Is there any updates for this be ready for primetime?

You can use your own domain with Gmail (for money), fwiw.

Yeah, will support for other email providers like Outlook and iCloud come sooner rather than later?

Oh, and that isn’t even counting lesser known email providers.

I agree this is the best option.

1 Like

Hi @martin,

Sorry if this was already asked, but could not find any reference on Meta.

According to:

Is it still under your plans to incorporate Outlook/Exchange Online for IMAP? If so, how Discourse will connect to Outlook/Exchange Online, giving the fact that MS is moving into the OAuth2 authentication direction ? (as per Authenticate an IMAP, POP or SMTP connection using OAuth | Microsoft Docs)

Cheers,
Ismael

1 Like

Sorry, we don’t currently have further improvements for IMAP support on our roadmap, nor do we have plans to add support for other providers at this time.

It turned out to be a far too complex piece with too many edge cases to worry about.

I recommend sticking with the simple to reason about POP3 integration.

3 Likes

Hi Sam,

Many thanks for your reply. Indeed we will like to stick to POP3 if possible, but if I’m not mistaken, Discourse communicates with basic-auth with POP3.

According to MS, basic-auth will be phased out as of October 2022 (please see Basic Authentication and Exchange Online – September 2021 Update - Microsoft Tech Community - Starting September 1, 2022, we will remove the opt out option, and starting October 1, 2022, we’ll begin turning off Basic Auth in all tenants, regardless of usage.).

Hence, re-formulating my question. Is it under your plans providing Oauth2 authentication for the current POP3 mail protocol?

Thanks,
Ismael

2 Likes

We actually do SSL by default :wink:

4 Likes

A post was split to a new topic: Add richer authentication support for POP3