Building an Amazon Discourse pilot, planning for scale


(Eoin Ryan) #1

Having looked at a few different platforms, I think that Discourse will be the best platform for our volunteer group.

I’m not fully sure how best to build it so that it is very cheap to run for the first few months of piloting, but then can easily scale to perhaps a few hundred concurrent users.

Is this the best way, or are there any drawbacks to using pre-packaged amazon discourse?
https://aws.amazon.com/marketplace/pp/B00JDSK2T4

I don’t really understand this thread but it seems important; Document for installing on AWS ECS

I have no experience with Docker and my linux skillz are limited to googling, Crtl C Ctrl V.

In terms of add-ons/plug ins, I would like

  • a voting app that supports single transferable votes and also allows users with different roles to vote on different topics
  • a chat/status plug-in such as babble (maybe there’s a better one?)
  • a ‘suggestions/ideas’ app where members can propose ideas and vote on all ideas with a certain number of votes based on their role/contributions to the community
  • suggestions as to the best way to verify identity; we currently have a volunteer database in a CRM and we would like to only allow CRM Contacts with a certain field ticked to have access to discourse, and for voting it’s important to use to tie each discourse account to a real identity (no issues creating active directory/LDAP if there’s any benefit in that?)

A previously used check list/task list for the installation/testing that someone might be willing to share?

Thanks for all advice and any questions.


(Matt Palmer) #2

Frankly, with that level of skill you’re going to be really pushing it to get anything solid and “scalable” on EC2. There’s lots of moving parts to Discourse, and while you can tease them all out into their component parts to run on Amazon’s proprietary stack, it requires a fair amount of skill and experience to do so without causing everything to burn. It’s also not particularly cheap, when all is said and done.

The cheapest and easiest way to get a Discourse instance up and running is to just stick it on a DO droplet. Scale it up by upgrading to a bigger droplet as needed. By the time you’re on DO’s biggest droplet and still running out of steam, then you’re paying so much that one of our managed services is a good deal cheaper (and easier, since we take care of all the infrastructure management).


(Eoin Ryan) #3

Thanks Matt - is this the best/current guide to doing that on DO?

As regards Amazon, we have our existing infrastructure of 15 wordpress (multitenant) sites on Amazon (another volunteer looks after that and he’ll be coming on board with this when time allows), so I have a preference to have it hosted there if I can find a clear step by step to share with him.

That said, if it’s simple to move from DO to somewhere else (Amazon/Discourse MS) with all customisations maintained then I think a little droplet might be the quickest way to get up and running.


(Matt Palmer) #4

The best documentation is our own 30 minute install guide. The DO one-click installer forces you to use a bigger droplet than you really need. If all you want to do is run a single VM you can do that on AWS, too, but it won’t be “the AWS way”, taking advantage of their proprietary features to more easily manage and scale the setup.

It’s easy to move a Discourse instance from one hosting provider to another, provided you’re using our Docker setup – copy the container defintion YAML file out of containers onto the new host, bootstrap/start, then take a backup of the data from the old and load it into the new.


(Eoin Ryan) #5

Thanks very much, I’ll give that go over the weekend and see how I get on.


(Eoin Ryan) #6

I have been able to find a colleague with Amazon/Linux skills. With that resource, would anyone be able to point me to a resource that describes how to build/install discourse to plan for scale (hundreds of concurrent users at peak times).

What about the links in my first post?


(Jeff Atwood) #7

Hundreds of concurrent users is not a very large site. You should be fine with the general cloud install instructions, and 2 CPUs / 4 GB RAM.


(Eoin Ryan) #8

Vain to quote myself, but many thx to a co-founder for spotting my thread, and would you have any one line pointers on the above pls?


(cpradio) #9

So something similar to this?

At best, that plugin fills the void you are seeking, at worst, it gives you a starting point to create your own from.


(Eoin Ryan) #10

That plug-in seems absolutely perfect, thanks. I’ll have to note the beta version (when the thread started) that may be needed to support it. I’m really impressed with how mature and slick discourse is to use. We’ve been piloting slack for a small group, and it’s good, but discourse will really support the reflective and collaborative best of this group that we are organising.

It’s a new political party, and having visited the US lately and seen some of the Sanders activity, I think that our young supporters will be able to maximise volunteer collaboration in an organisation-supported way.


(Matt Palmer) #11

I’m not aware of anything Discourse-specific, but there’s hundreds or thousands of opinions “out there” about how to scale Rails, PostgreSQL, and Redis, and Discourse is a fairly standard Rails/PostgreSQL/Redis application. As the person who is primarily responsible for the infrastructure here at CDCK, I’ve got to say that my job pretty much never involves worrying about scaling an individual site; my time is spent working on the problems of automation and scaling for the entire customer base. Rails+PgSQL+Redis just chugs along nicely all by itself.


(Eoin Ryan) #12

Great, Matt, thanks.
It’s reassuring to learn that my initial worries about those problems can be set to the side for now.