Set up a staging server

[nathank (Nathan Kershaw)]

There are several tricks that can help when you are setting up a staging server.

What is a staging server?

A staging server is essentially a clone of a production site. It also resides on a server, and functions identically. It runs inside a Docker container, just as a normal Discourse site does.

It exists to give you a place to try risky things out, or to trial things that you cannot easily hide from your users. It is very useful for trialling adverts using the Discourse Advertising Plugin (Ads), or if you want to do something funky like a forum import or merge.

This is in contrast to a development server, which typically runs in an easily accessible (and sandboxed) place so that a developer can tinker with the code safely.

What do I need?

1. Everything that you need for a standard self-hosted install

2. If you have S3 backups setup, your life will be a whole lot easier

   • otherwise you’ll need a way to move large files into and out of a server via SSH

Steps

Set up your server as you wish

Typically in a virtual Ubuntu server hosted on Digital Ocean, but you can use whatever you are comfortable with.

Install Discourse

Via this guide (or maybe via dashboard.literatecomputing.com). I recommend using ‘junk’ email credentials (you don’t need or want email to work).

discourse/docs/INSTALL-cloud.md at main · discourse/discourse · GitHub

Confirm that your install is working:

Establish an admin account (if needed)

Set yourself up an admin account from the command line. This skips the need to authenticate via email.

./launcher enter app
rake admin:create

This isn’t strictly necessary except for testing the install as you can restore from backup from the command line.

Edit app.yml and add some tweaks

1. You might like to make a copy of the original app.yml (I call mine app.vanilla.yml) which you can revert to if you stuff things up

2. At the bottom of the env section add these lines:

     ## Staging server specific settings
     DISCOURSE_AUTOMATIC_BACKUPS_ENABLED: false
     DISCOURSE_LOGIN_REQUIRED: true
     DISCOURSE_DISABLE_EMAILS: 'yes'
     DISCOURSE_S3_DISABLE_CLEANUP: true
     DISCOURSE_ALLOW_RESTORE: true
   

3. If you have S3 (or similar) backups configured then add these too (with your settings from the main site)

     ## S3 Configuration
     DISCOURSE_S3_ACCESS_KEY_ID: 'your_key'
     DISCOURSE_S3_SECRET_ACCESS_KEY: 'your_secret'
     DISCOURSE_BACKUP_LOCATION: 's3'
     DISCOURSE_S3_BACKUP_BUCKET: 'your_backups_location'
     DISCOURSE_S3_REGION: 'your_s3_region'
     DISCOURSE_S3_DISABLE_CLEANUP: true
   

   and if you also are doing S3 uploads:

     DISCOURSE_ENABLE_S3_UPLOADS: true
     DISCOURSE_S3_UPLOAD_BUCKET: 'your_uploads_location'
   

4. You might like to add the same plugins that you have on your production site while you are there.

5. Do a rebuild

   • ./launcher rebuild app

Managing the staging server

You’ve now got a staging server that is connected to your S3 backups (but won’t overwrite them), is easy to restore to, and that is unable to email anyone under any circumstances. Perfect!

You can restore a fresh backup to the staging server and go to town. If you don’t like what you have, you simply restore it again.

Turning it off or on

If you leave your staging server ‘on’ for prolonged periods, you do risk it being indexed by Google, and having your users accidentally login to it instead. As their credentials are a clone of your production site, this is very possible.

A simple way to mitigate these two things is to simply turn Discourse off:

 ./launcher stop app

And to turn it back on so you can use it:

./launcher restart app

Updates

You’ll have to make sure that you update/rebuild both it and your production site at the same time if you want to ensure that things remain aligned from the plugin and code point of view. Same goes for app.yml changes.

If you don’t use S3, you’ll have to manually move backups between servers. And they are big!

Seeding a Test Server

If you want a staging server, then you should populate it with your actual data from your actual forum via a Restore. Sometimes it is your particular data that is causing the issue, and testing your forum with some other set of data can give you a sense of false hope.

If what you want is a test server to see what Discourse is like, though, you might want to check things out with some fake data, and if you do, you can do this:

./launcher enter app
ALLOW_DEV_POPULATE=1 bundle install
ALLOW_DEV_POPULATE=1 rake dev:populate

This will seed your forum with some fake data so that you can see what things look like with whatever themes and plugins you want. If you haven’t started your forum yet, this will give you some idea what things are likely to look like.

Managing Two Factor Authentication

While your account username / password from your main site should also work just fine in the staging site, it isn’t so pretty with 2FA. If you have an issue, turn off 2FA:

./launcher enter app
rake users:disable_2fa[<USERNAME>]

[merefield (Robert)]

Nathan, a great idea to put this guide together.

Maybe I’ve missed it, but what step here disables email?

[nathank (Nathan Kershaw)]

Good question. Putting in junk SMTP credentials absolutely prevents it, but it would make sense to turn off emails as well with:

DISCOURSE_DISABLE_EMAILS = yes

Also, that is turned on automatically when you do a restore so it isn’t really necessary.

[nathank (Nathan Kershaw)]

Added some instructions for turning off the app to the OP.

[pfaffman (Jay Pfaffman)]

@nathankershaw

Also, that is turned on automatically when you do a restore so it isn’t really necessary.

Right, and, it’s often nice to be able to get a login link, so, I’d recommend

 DISCOURSE_DISABLE_EMAILS = 'non-staff'

[merefield (Robert)]

How about a section on creating fake users? Some admins may not want any personally identifiable information on staging servers. What are people using to create fake users on a bigger scale or get rid of PII?

I thought about using a production import and then running the anon job on each, but that would end up with a very dull looking staging site!

Can I suggest the OP is made into a Wiki?

Some links:

Ruby on Rails and the Complexity of Fake User Profiles, Made Simple | HackerNoon

How to populate database with fake data?

[pfaffman (Jay Pfaffman)]

I made it a wiki.

Generally, I want a staging site to be using the same data as the production site so that you can test that things will work with the actual data. But maybe people want fake data to see what discourse can do before they start to use it? (Oh, I guess those links have some more sophisticated solutions.)

I guess you could have a User.create in a loop with a list of names from somewhere.

[JammyDodger]

Obviously not my forte , but would this be a good opportunity to use rake dev:populate?

cd /var/www/discourse
ALLOW_DEV_POPULATE=1 bundle install
ALLOW_DEV_POPULATE=1 rake dev:populate

I think that would also work on a production site that is more a staging environment/test site.

[merefield (Robert)]

Apparently that’s no hindrance!

That’s a great suggestion:

Task code: discourse/populate.rake at 1472e47aae5bfdfb6fd9abfe89beb186c751f514 · discourse/discourse (github.com)

User specific actions:

discourse/discourse/blob/1472e47aae5bfdfb6fd9abfe89beb186c751f514/lib/discourse_dev/user.rb#L16

[pfaffman (Jay Pfaffman)]

@JammyDodger

Obviously not my forte , but would this be a good opportunity to use rake dev:populate

Nice! Indeed, someone already thought of this problem!

EDIT: and for kicks, I tried this out on a recently installed test site; I pasted in your bundle and rake tasks and it did this:

root@test2-app:/var/www/discourse# ALLOW_DEV_POPULATE=1 rake dev:populate
OK
I did no detect a custom `config/dev.yml` file, creating one for you where you can amend defaults.
There are 9 group records. Creating 6 more.
......
There are 3 user records. Creating 27 more.
...........................
There are 4 category records. Creating 26 more.
..........................
discourse-solved enabled on category 'Recipes' (12).
Creating 30 sample tag records
..............................
There are 6 topic records. Creating 24 more.
........................
root@test2-app:/var/www/discourse# 

[Stephen]

@merefield

How about a section on creating fake users? Some admins may not want any personally identifiable information on staging servers. What are people using to create fake users on a bigger scale or get rid of PII?

The big problem with this is that your data set no longer represents your live data.

A staging server needs to be representative of live, otherwise you can’t test everything prior to going to production with whatever changes are planned.

I’ve spectated some pretty impressive failures where non-representative tests failed to identify issues which later arose in the live environment. More often than not they occurred because of data quality issues.

Double-barreled surnames (with and without the hyphen) and accented characters for example caused tremendous amounts of hurt.

If it’s a true staging server then it needs to mimic live precisely. The copy doesn’t need to be visible to normal users and disabling non-staff email is pretty advisable, but otherwise it’s just inviting problems.

7 Reasons Why Your Staging Environment Sucks

[merefield (Robert)]

Oh I agree. My question was driven by a client who was nervous about having real client identities in staging.

Optimally we should have a script to jumble names and email addresses and leave everything else the same.

[Stephen]

It sounds like a pretty straightforward conversation. If they don’t have a representative copy, then they don’t have a staging site.

If it’s deployed and secured in the same way as live what is their perceived risk?

[pfaffman (Jay Pfaffman)]

I’m with Stephen. Is the client more nervous having real data on a staging site, or not having a staging site that’s actually testing your actual data?

If you’re not testing with your actual live data from production, then you don’t know what will happen when you do use the live data.

And this discussion is getting far afield from the OP.

[pfaffman (Jay Pfaffman)]

I think that this should be set up to delete posts after 30 days or something. I added this to the OP. There are times when fake data is handy. The more paranoid amongst us (myself included), have real world reasons not to trust a staging server if it hasn’t been tested with your actual data.

@nathankershaw

Seeding a Test Server

If you want a staging server, then you should populate it with your actual data from your actual forum. Somtimes it is your particular data that is causing the issue, and testing your forum with some other set of data can give you a sense of false hope.

If what you want is a test server to see what Discourse is like, though, you might want to check things out with some fake data, and if you do, you can do this:

cd /var/discourse
./launcher enter app
cd /var/www/discourse
ALLOW_DEV_POPULATE=1 bundle install
ALLOW_DEV_POPULATE=1 rake dev:populate
exit
exit

This will seed your forum with some fake data so that you can see what things look like with whatever themes and plugins you want. If you haven’t started your forum yet, this will give you some idea what things are lkely to look like.

[nathank (Nathan Kershaw)]

After having a few issues after implementing 2FA on our site I’ve added this:

@nathankershaw

Managing Two Factor Authentication

While your account username / password from your main site should also work just fine in the staging site, it isn’t so pretty with 2FA. If you have an issue, turn off 2FA:

./launcher enter app
rake users:disable_2fa[<USERNAME>]

[denvergeeks]

WoW – that was so awesome! Bada-bing-bada-boom!

I feel so productive after importing that dummy data… all of a sudden my testing forum automagically populated with a bunch of users and posts and tags and categories and groups… oh my!

Thank you so much @nathank and @pfaffman and @merefield and @JammyDodger and @Stephen… oh my!

[thoka (Thomas Kalka)]

I would love to read recommendations, how to disable pop polling via command line.

[pfaffman (Jay Pfaffman)]

The best way is to set an env variable DISCOURSE_pop3_polling_enabled=false

You need to upcase the entire variable name, but I can’t do it on my phone.

[Octoberon]

I recently migrted my production forum to S3 and CloudFront. I already had a staging server up-and-running but it’s now out of step with production and S3 because I’m not sure whether I need a separate bucket and CDN connection - I don’t particularly want to incur additional AWS costs just for a staging server. Presumably pointing both servers at the same S3 bucket isn’t recommended? What’s the right way to do this?