IMAP support for group inboxes

martin · August 12, 2020, 5:03am

We are removing support for IMAP in Discourse. See this post for details IMAP support for group inboxes - #39 by martin .

We are happy to introduce an alpha version of IMAP support for group inboxes. This feature adds the ability to enter IMAP credentials and settings on a per-group basis to sync group inboxes with an email inbox.

This is an alpha-stage feature, and as such it is not 100% complete and polished. It is very likely to be broken, we are not currently maintaining it or using it in production. Use at your own risk!

Benefits

Your group can hit the ground running with using Discourse inboxes, because all your existing mail will be synced!
Your group can have their messages synced between Discourse and your email provider so there is no single point of failure. People can reply from the email account or from within Discourse and everything will sync up
You will no longer need to set up crazy forwarding rules from your email provider to your Discourse incoming mailbox.
Email labels will sync with Discourse tags so things can be nice and organized
You will be replying to people who email your group from the email address you set up, even if you reply from Discourse. No more confusion!

Features

All incoming and outgoing will be synced between the IMAP server and Discourse, with the appropriate topics and post replies created based on parsed emails. Replies can be created from Discourse or from the IMAP server and everything will sync up!
Emails can be replied to from the Discourse group inbox OR from the email account.
Tags applied to the Discourse topic will be created as labels and applied to the email on the IMAP server (this is provider-specific).
Emails archived on the IMAP server will be archived in the group inbox.
Group private message topics that are archived on Discourse will be archived on the IMAP server.
Emails deleted on the IMAP server will delete the topic in the group inbox.
Group private message topics that are deleted on Discourse will be deleted on the IMAP server.
Emails sent by groups with this feature enabled will have their reply-to address set to the same email username set up in the IMAP settings.

Getting started

First of all, there are some site settings you must configure before you can set up this functionality on a group.

enable imap and enable smtp both need to be enabled. IMAP is used for syncing with your mail server, and SMTP is used to send emails from your mail server.
tagging enabled and allow staff to tag pms - Must both be enabled, as label sync applies tags to PMs.
enable imap write - Must be enabled if you want changes made in Discourse to reflect in your mail server (for example tags, archiving topics, and deleting topics).
enable imap idle - Lets us receive live updates from your mail server IF your mail server supports IDLE. This greatly speeds up the reflection of changes in your email provider to Discourse. You should turn this on (Gmail supports it, for example). If you want all the gory details you can read about them in the RFC for IDLE.
imap polling period mins - If IDLE is not supported, this is the number of minutes we wait before checking the IMAP server for changes. It is also the time we wait between sending changes to the IMAP server from Discourse if enable imap write is enabled. The minimum is 1 minute.

These settings can be left at their defaults, and tweaked if you are finding issues with the syncing:

imap polling old emails - The maximum number of old emails (processed) to be updated every time an IMAP box is polled (0 for all).
imap polling new emails - The maximum number of new emails (unprocessed) to be updated every time an IMAP box is polled.
imap batch import email - The minimum number of new emails that trigger import mode (disables post alerts).

If you want labels/tags to sync with the IMAP server you need to enable the tagging enabled and allow staff to tag pms site settings enabled.

Next, you need to go into the group that you want to sync with your IMAP server and fill in the settings.

These settings will be provider-specific, see below for more details. Once you have entered the settings and credentials click “Save Changes” and we will validate the credentials against the IMAP provider’s servers. If successful, the list of mailboxes will be populated, and you need to choose which one you want to sync (guidance on this is also provider-specific):

If the credential validation was not successful, an error message will be shown on the page. To be clear, the SMTP server entered will be used to send emails on behalf of the account entered under username & password, not the main Discourse SMTP configured for bulk emails.

Finally, inside your app.yml file you will need to add DISCOURSE_ENABLE_EMAIL_SYNC_DEMON: true inside the env section and run ./launcher rebuild. This will enable the background worker to start up and start syncing emails!

Provider-specific information

Gmail

You must generate an app password, otherwise you need to turn on “Less secure app access” which Google is getting rid of at some point anyway. Use this password instead of your Gmail account password in the IMAP settings. See Sign in with app passwords - Gmail Help for more details.
Make sure you use these settings for SMTP and IMAP:
- IMAP port: 987
- SMTP port: 587
- IMAP server: imap.gmail.com
- SMTP server: smtp.gmail.com
- Use SSL for both SMTP and IMAP
It is strongly advised (soon to be a UI feature) that you only select the “[Gmail]/All Mail” mailbox to sync with.
We do not immediately delete emails in Gmail, only move them to the trash inbox, when the topic is deleted in Discourse. The “30 day” delete functionality inside Gmail will take over from there.
Tags applied to Discourse topics will create Labels in Gmail, and apply them to email threads. Labels in Gmail are also IMAP mailboxes!

Limitations

This is an alpha-stage feature, and as such it is not 100% complete and polished. Therefore, the following limitations currently apply:

Only Gmail is currently supported as an IMAP provider. We have some generic IMAP functionality but it is not guaranteed to work. Outlook online is our next big target for support.
Resurrecting deleted emails from Gmail’s Trash and reflecting the state in Discourse is currently flaky.
Changing the sync mailbox after emails have already synced is not advised and can lead to a lot of weird problems.
Only one mailbox can be synced per group.
Group-to-group communication is uncharted territory and will not work well (e.g. emailing team@yoursite.com from support@yoursite.com if each of those emails is set up for a different group).

There may also be other subtle gotchas and rough edges as this is an alpha feature.

Feedback & Roadmap

We would love feedback from anyone using this feature with Gmail. I will be the one working on hammering out any bugs/issues. Debugging is a little tricky right now but I am working on that!

The next things I will work on:

Improve debugging to make it easier to view IMAP logs and figure out things that go wrong.
Improve the group email interface to limit the mailboxes that can be selected from Gmail and to also discourage/disallow changing mailboxes.
Validation to prevent using the same IMAP details for multiple groups.
Possibly improvements to how the group IMAP settings are stored and a better UX for validating credentials.
Outlook support.

Currently while we work out the kinks this feature is not available to our hosted customers.

Special Thanks

Both @dan and @joffreyjaffeux are the primary contributors of this huge feature which has been worked on for a long time. Without their amazing work I would not have been able to bring the feature to this point and make this announcement .

hellekin · August 12, 2020, 7:09am

This is at the same time fantastic news and terrible news. Let me explain. First is great news to see IMAP integration with Discourse groups, as it brings a number of useful solutions to group identity (using the correct original address), to facilitate multiple users using the same IMAP account (not well supported natively, e.g. no shared reading states), and it opens the path to ActivityPub inboxes… But it’s a terrible news because its implementation favors email giants who turned a federated service into a largely centralized system. I understand the attraction to work with large scale providers, but I hope IMAP standard will be preferred over provider-specific extensions so that any email provider will be supported once this feature reaches beta state.

Kudos for making this possible.

sam · August 12, 2020, 8:18am

There is a big reason this feature is considered “alpha”

The trouble with IMAP from what I have observed from the sidelines is the amount of “open to interpretation” we are stuck dealing with.

We certainly want to make this work with more providers, our internal itch favors the big Google cause the reality is that we use Google at Discourse so we wanted to solve our problem first.

PRs to make this work better with more providers are certainly welcome, in absence of them as time passes and we observe more desire to use this with other providers we shall build support.

martin · August 12, 2020, 8:52am

This is a fair criticism, like @sam said we focused on Gmail first because this is what we use so it is the most relevant for us. However I believe that our base “generic” provider, while certainly not nearly 100% compliant to the RFC and not feature complete, follows the IMAP protocol quite closely…see https://github.com/discourse/discourse/blob/master/lib/imap/providers/generic.rb. The only additions I think this will need are special handling of Archiving and Deleting. Deleting we currently just flag as \Deleted and send an EXPUNGE command, but I know certain email clients will let you choose to do this or send to a trash folder first then EXPUNGE etc. I am not sure what the Archiving standard is for IMAP. We have a specific Gmail provider class that overrides some of these generic methods.

If you have any suggestions on more “generic” IMAP servers/providers that I can test against that would be a great help, as Gmail has its own special snowflake handling of things and Gmail IMAP extensions.

Also, as Sam said, any PRs for improvement are welcome, otherwise we will cater to demand, which I am predicting will be Outlook next as another big use case for organizations wanting this kind of feature (Outlook has its own special snowflake rules…like Archive being a folder/mailbox and no label support).

itsbhanusharma · August 12, 2020, 8:57am

Looks very exciting, My firm recently switched from our own mailserver to G-Suite and this integration will make it much easier for us to respond to client queries and manage our progress trackers. I have one query for @martin though. It isn’t clear to me whether it would use gmail smtp to reply to messages received in group inbox or would it use the mass smtp configured to send emails on behalf of Discourse? I see that the plugin mentions gmail’s smtp as well but I still wanted clarity on this.

martin · August 12, 2020, 8:58am

Gmail’s SMTP server is used to send the replies, as you are sending the replies on behalf of the email account used for IMAP syncing! I will add a note to this effect in the OP to clarify.

itsbhanusharma · August 12, 2020, 9:02am

That’s actually brilliant. I can think of so many use cases for this. Kudos @martin @dan @joffreyjaffeux

martin · August 17, 2020, 12:10am

Quick update here. I merged these two PRs last week. First, the fix so that tagging does not need to be enabled for archiving and deleting to work:

https://github.com/discourse/discourse/pull/10426

Second, I am now writing all IMAP logs to the database for easier inspection. A job will be run daily to delete logs > 5 days old:

https://github.com/discourse/discourse/pull/10435

StanD · August 21, 2020, 10:47pm

This is super sweet! Thanks.

I should say, I had to create an application password in my google email account for discourse to get it to accept my username/password

sunjam · August 23, 2020, 3:25pm

Are you running 2-factor authentication on the Google end? If so, this is standard practice.

StanD · August 23, 2020, 4:34pm

Yes. I believe it was required in order for me to create the app password (though I don’t remember for sure)

martin · August 23, 2020, 11:40pm

This is noted in the OP Otherwise you have to jump through crazy hoops to make your Gmail account accept “Less secure apps”. At some point we will need to support OAuth for Google as that looks like the way things are headed.

StanD · August 24, 2020, 1:19am

Oops! Did I mention that my reading comprehension skills suck?

martin · August 24, 2020, 1:25am

It’s okay! It is an important point so I moved it to the top of the Gmail list and added some warning indicators. Thank you for trying to let others know about this to save them trouble

thoka · February 6, 2021, 1:10am

I’m trying to understand why I’m failing to sync mails with a dovecot IMAP server (unfortunately without experience in ruby or rails).

Reading the imap_sync_logs the import daemon hangs in a loop:

UIDVALIDITY = 1612565899 does not match expected 0, invalidating IMAP cache and resyncing emails for mailbox INBOX

So I am wondering, if @group.imap_uid_validity shouldnt be updated in this case:

https://github.com/discourse/discourse/blob/f34fa999a2c8a4f3d0c477bef9401817c538b405/lib/imap/sync.rb#L47

After a manual update of this value, the import daemon hangs in another loop.

Dovecot does not support 'LABELS' in

https://github.com/discourse/discourse/blob/f34fa999a2c8a4f3d0c477bef9401817c538b405/lib/imap/sync.rb#L238

yielding a Net::IMAP::BadResponseError which, unfortunately, does not find its way into the logs.

Now I’m wondering

if dovecots support of custom flags is sufficient for bi directional sync with discourse
how to detect dovecot in lib/imap/providers/detector.rb
how to emulate the label feature in a dovecot provider

…

martin · February 7, 2021, 11:39pm

IMAP syncing for group inboxes is still an early stage feature that currently only supports Gmail. Theoretically you could write a dovecot provider, but I noticed a couple of weeks back that there are some places in the main code that use Gmail-only features that need to be moved into provider-specific code. So sorry about that, but you may want to wait until this is a little closer to primetime.

thoka · February 8, 2021, 2:54pm

IIUC 'LABELS'is an non-standard feature of gmail.
I would like to understand the difference of google imap and standard IMAP.

This is my summery so far:

standard IMAP	google IMAP
archived mails live in subfolders	all mails live in one folder archived massages are missing the “\INBOX”-label
custom flags are stored along system flags	stores custom flags as labels

Would it be sufficient

to emulate labels in standard IMAP by splitting flags into system and custom flags
handle email_is_archived by the provider (for example add and test a “archived” flag)

to get standard IMAP syncing going?

This would affect this lines:

github.com

discourse/discourse/blob/f34fa999a2c8a4f3d0c477bef9401817c538b405/lib/imap/sync.rb#L295


      
                update_email(incoming_email)
                incoming_email.update(imap_sync: false)
              end
            end
          end
          
          def update_topic_archived_state(email, incoming_email, opts = {})
            topic = incoming_email.topic
          
            topic_is_archived = topic.group_archived_messages.size > 0
            email_is_archived = !email['LABELS'].include?('\\Inbox') && !email['LABELS'].include?('INBOX')
          
            if topic_is_archived && !email_is_archived
              ImapSyncLog.debug("Unarchiving topic ID #{topic.id}, email was unarchived", @group)
              GroupArchivedMessage.move_to_inbox!(@group.id, topic, skip_imap_sync: true)
            elsif !topic_is_archived && email_is_archived
              ImapSyncLog.debug("Archiving topic ID #{topic.id}, email was archived", @group)
              GroupArchivedMessage.archive!(@group.id, topic, skip_imap_sync: true)
            end
          end

github.com

discourse/discourse/blob/f34fa999a2c8a4f3d0c477bef9401817c538b405/lib/imap/sync.rb#L357


      
          # don't do any of these type of updates on anything but the OP in the
          # email thread -- archiving and deleting will be handled for the whole
          # thread depending on provider
          return if post&.post_number != 1
          topic = incoming_email.topic
          
          # if email is nil, the UID does not exist in the provider, meaning....
          #
          # A) the email has been deleted/moved to a different mailbox in the provider
          # B) the UID does not belong to the provider
          email = @provider.emails(incoming_email.imap_uid, ['FLAGS', 'LABELS']).first
          return if !email.present?
          
          labels = email['LABELS']
          flags = email['FLAGS']
          new_labels = []
          new_flags = []
          
          # Topic has been deleted if it is not present from the post, so we need
          # to trash the IMAP server email
          if !topic

github.com

discourse/discourse/blob/f34fa999a2c8a4f3d0c477bef9401817c538b405/lib/imap/sync.rb#L381


      
            return @provider.trash(incoming_email.imap_uid)
          end
          
          # the topic is archived, and the archive should be reflected in the IMAP
          # server
          topic_archived = topic.group_archived_messages.any?
          if !topic_archived
            # TODO: This is needed right now so the store below does not take it
            # away again...ideally we should unarchive and store the tag-labels
            # at the same time.
            new_labels << "\\Inbox"
          
            ImapSyncLog.debug("Unarchiving UID #{incoming_email.imap_uid} (incoming ID #{incoming_email.id})", @group)
          
            # some providers need special handling for unarchiving too
            @provider.unarchive(incoming_email.imap_uid)
          else
            ImapSyncLog.debug("Archiving UID #{incoming_email.imap_uid} (incoming ID #{incoming_email.id})", @group)
          
            # some providers need special handling for archiving. this way we preserve
            # any new tag-labels, and archive, even though it may cause extra requests

This line seems strage to me:

github.com

discourse/discourse/blob/f34fa999a2c8a4f3d0c477bef9401817c538b405/lib/imap/sync.rb#L399


      
          
              # some providers need special handling for archiving. this way we preserve
              # any new tag-labels, and archive, even though it may cause extra requests
              # to the IMAP server
              @provider.archive(incoming_email.imap_uid)
            end
          
            # Sync topic status and labels with email flags and labels.
            if tagging_enabled?
              tags = topic.tags.pluck(:name)
              new_flags = tags.map { |tag| @provider.tag_to_flag(tag) }.reject(&:blank?)
              new_labels = new_labels.concat(tags.map { |tag| @provider.tag_to_label(tag) }.reject(&:blank?))
            end
          
            # regardless of whether the topic needs to be archived we still update
            # the flags and the labels
            @provider.store(incoming_email.imap_uid, 'FLAGS', flags, new_flags)
            @provider.store(incoming_email.imap_uid, 'LABELS', labels, new_labels)
          end
          
          def tagging_enabled?

Any system flags will be overriden by discourses tags.
Does this handle all IMAP flags well (`\Answered, \Deleted, \Draft, \Flagged, \Recent and \Seen / $Forwarded, $MDNSent, $SubmitPending and $Submitted)?

StanD · March 29, 2021, 2:58am

I am revisiting this after some time away.

I have not been able to figure out how to get the reply-to header to work properly. I can set up a group to sync with a particular gmail account but when I try to reply via email, the reply-to address is set to the configured notification email from discourse and not the IMAP address.

Is there something that I’m missing?

mognet · October 16, 2021, 4:15pm

Just tested this with GoDaddy email provider but no success. With Gmail works great sadly we want to use our own domain. Is there any updates for this be ready for primetime?

pfaffman · October 16, 2021, 5:58pm

You can use your own domain with Gmail (for money), fwiw.

Topic		Replies	Views
Discourse Version 2.0 Releases	5	18414	May 31, 2018
Migrate a mailing list to Discourse (mbox, Listserv, Google Groups, etc) Migrating to Discourse how-to	98	22882	August 12, 2024
Discourse Version 1.8 Releases	4	18095	May 31, 2017
Discourse Version 2.7 Releases	4	9223	May 18, 2021
MOSS Roadmap - Mailing lists Feature	53	17578	December 28, 2017

IMAP support for group inboxes

Benefits

Features

Getting started

Provider-specific information

Gmail

Limitations

Feedback & Roadmap

Special Thanks

Related topics