Preserving user sessions when migrating between hosts

TallTrees · October 6, 2020, 10:07pm

Hi!

I’m preparing to migrate some Discourse instances to new hosting. The plan is to set the old site read-only, take a backup, move that across to the new hosting and restore while updating the DNS settings (on a short TTL). This process is all taken from guides here and around.

I’ve been doing some dry-runs on this, using a /etc/hosts file hack to simulate the DNS update (and with DNS over HTTPS disabled in the browser too). So far so good.

However, despite being careful to close the browser on the ‘source’ site, then only opening it after complete migration on the ‘destination’ site, it forgets that it was logged in.

Obviously I don’t want the site members to all be logged out over the migration. Where should I look or what am I missing here?

The user I’m currently testing has Admin and is using 2FA, and the discourse containers are behind Nginx which terminates SSL, if that makes a difference to the problem.

Thanks in advance for any pointers!

ChrisBeach · October 6, 2020, 11:41pm

I’ve had this problem too and I’m keen to resolve it.

Any pointers appreciated.

TallTrees · October 7, 2020, 7:16am

I think I’m going to try creating some test accounts without 2FA in case that has anything to do with it - very few site members use 2FA, so that might be acceptable it it’s just those sessions that get lost.

Also I’ll check the source and destination servers are sync’d to NTP though they can’t be more than a few seconds off each other now I think.

Other than that, I guess there’s always the source code to try and figure out what goes into the authentication token and could be causing a problem…

IAmGav · October 7, 2020, 7:24am

Your best would be to inform your users about the change and that they would be logged out and need to sign in again.

Each server has their unique signature with the SSL in place.

It will be less of a headache to make the user sign in again, than trying to change server settings.

Just apologize for the inconvenience caused, but point out its for the better

ChrisBeach · October 7, 2020, 7:30am

Is that the case though?

How are session cookies affected by SSL “signatures” (especially when, in my case, the new and old hosts are using the same SSL certificate files, on the same domain name)?

TallTrees · October 7, 2020, 9:10am

While I have no problem apologizing to the members (), I really want to avoid that scenario as I can see a couple of unfavorable side-effects:

Members don’t log in again, and are less likely to participate in the forum in future
Members forget their password and need help recovering their passwords
Members forget their passwords and create new accounts, leaving lots of ‘zombie accounts’ on the forum and lost/confusing duplicate identities without User Notes or post history etc…

The SSL bit all works fine in front of Discourse (even with the hosts file hack), which if you think about it, must be the case otherwise load-balancers would have problems. As far as I’m aware, Discourse doesn’t even know about the SSL as it is terminated at Nginx in this setup.

I guess a different question is ‘is it possible to migrate a Discourse server to a new IP without logging everyone out?’ I had assumed it would be, but maybe that assumption was wrong…

riking · October 7, 2020, 12:00pm

I know that I’ve definitely had forum sessions survive a backup & restore. I’m really not sure what’s going on here

david · October 7, 2020, 12:33pm

Session cookies are encrypted using a randomly generated secret key which, by default, is stored in redis. Redis data is not included in a backup so a new secret key is regenerated when you restore the site onto a new server.

github.com

discourse/discourse/blob/6e69c251348aae29af9ad504e1b01d431a751423/app/models/global_setting.rb#L18-L24


# In Rails secret_key_base is used to encrypt the cookie store
# the cookie store contains session data
# Discourse also uses this secret key to digest user auth tokens
# This method will
# - use existing token if already set in ENV or discourse.conf
# - generate a token on the fly if needed and cache in redis
# - enforce rules about token format falling back to redis if needed

You can manually set the secret key using an environment variable DISCOURSE_SECRET_KEY_BASE in your app.yml file. So you could try something like this to preserve sessions:

On your existing server, enter the console and find the secret key from redis
```
pry(main)> GlobalSetting.safe_secret_key_base
=> "5fb9dc98be368599e0a..."
```
Add DISCOURSE_SECRET_KEY_BASE to the env: section of your app.yml on the old server, using the value you found in (1), and rebuild the app. In theory, if everything went right, Discourse should now use the value from the app.yml, and user sessions will persist. You can check the env variable is being used by running
```
GlobalSetting.secret_key_base
```
Make sure the app.yml on the new server has the same SECRET_KEY_BASE. When you restore the backup, the user sessions should persist

I haven’t tested that flow, so if you plan to use it for a production forum make sure to test it out first!

Side note: you absolutely should not share the secret key between multiple discourse instances. Possession of the secret key would allow someone to decrypt and modify their session cookie on a site, and could have very bad security consequences.

TallTrees · October 7, 2020, 1:01pm

Excellent - this is something I shall try out on a sandbox instance. I’ll see if I can report back if it works or not

Understood. I am now pondering whether there should be a feature request for an option to export this key in backups? To me, losing all the sessions on a site would seem like a ‘very bad thing’ to happen as there are a few thousand accounts on the site. Practicing the migration has shown this problem, but I guess if the machine or Docker instance were lost for some reason, the key would be lost too and we’d be facing the same loss of sessions.

Some note of this ‘feature’ on the migration topic might be good too: Move your Discourse Instance to a Different Server

david · October 7, 2020, 1:06pm

It is a delicate balancing act between security and convenience.

At the moment if someone manages to steal a discourse backup, they have all the data on the forum. But they do not have the ability to log in to the live forum. Passwords are hashed, api keys are hashed, and session cookies are encrypted.

If we included the secret inside the backup, then anyone in possession of a backup could gain access to the live forum, and conduct phishing/fraud/etc.

Sounds great! Totally open to having a topic here on Meta explaining how to migrate from a redis secret key to an app.yml secret key. It could be linked from the ‘move to a different server’ topic.

TallTrees · October 7, 2020, 1:21pm

The security of the default backups should also be improved too, in my humble opinion. While passwords and sessions may be hashed and protected, there could be a lot of other data in private messages and such which should still remain confidential. Particularly note that in our community forum we encourage people to use private messages to swap contact details and arrange sales or collections, rather than putting phone numbers and addresses in public places.

The approach I’m taking to backups is to enter the instance, generate a backup, then immediately encrypt it with a public key using gpg. This renders the backup useless to anyone that doesn’t have the corresponding private key, which I store off the server and protected by a passphrase.

That said, if the secret key for sessions doesn’t change, it only need to be backed up once, so a simple procedure for that is all that’s needed - and hopefully is what you’ve outlined above.

I’ll give it a go and see if I can create that topic for you

ChrisBeach · October 7, 2020, 1:43pm

Very helpful @david, thanks!

Stephen · October 7, 2020, 1:54pm

Discourse doesn’t offer private messages, the feature is called Personal Messages. The distinction is very important, there’s no suggestion of privacy.

Admins can already read every PM on the site unless encryption is in use.

TallTrees · October 7, 2020, 2:00pm

Good point. But it doesn’t change the fact that members may use PM’s to exchange personal details or things that they wish not to be in the public domain. Protecting this data as far as is reasonable is therefore important.

Yes, we also have to trust the admins of every site we use, even on sites like ProtonMail.

david · October 7, 2020, 2:04pm

Kind of off topic… but you might be interested in Discourse Encrypt (for Private Messages) for truly ‘private’ messages. Leaked/stolen backups of encrypted PMs would not be readable by an attacker.

(although, as you said, there is still an assumption that the admins are trusted)

TallTrees · October 7, 2020, 9:46pm

@david - please can you check the following post and split and move to #howto if okay? (I cannot create a topic there, oh and Akismet just hid the post so that needs fixing too )

david · October 8, 2020, 9:22am

2 posts were split to a new topic: Preserving user sessions when migrating to a new host

TallTrees · October 8, 2020, 9:30am

@david - many many thanks for sorting that and the advice, I hope it helps others too - it’s certainly a relief for us to be able to migrate and keep the sessions

david · October 8, 2020, 5:00pm

This topic was automatically closed after 7 hours. New replies are no longer allowed.