How discourse stays online (Message Bus, Faye, Long Polling)

Dev
1

Hi, sorry for bad English. I looked at source, but it’s hard to read and understand in big project. But as i understood Discourse using own implementation of communication over (Long polling (only(?))) to represent most actual data to users with MessageBus, why? Why were not used Faye?

6 Likes
2

I too would be interested in a quick story about this architecture. I’m currently thinking how to best implement a chat (sorry, in PHP and not for Discourse), and a story about how you manage and route a large stream of messages would be very interesting. (Plus, in my case, I don’t know Ruby at all, so reading the source code would entail learning Ruby first - hence I’d like to hear a story)

1 Like
3

As i know, to use Faye as a publish server, you can’t use Nginx or Apache as web server to proxy the Faye, so you need to publicize the Faye server to publish to client (it’s bad ?), in case of long polling, it works well with many kinds of web servers. Facebook uses long polling itself, i think long polling is suitable to deliver real time information to web clients.

1 Like
4

The main feature of Faye and Socket.io etc… is distinguish the best transport according of capabilities of browser look here http://faye.jcoglan.com/architecture.html

Persistent connections using WebSocket
Long-polling via HTTP POST
Cross Origin Resource Sharing
Callback-polling via JSON-P
1 Like
6

It’s a client side https://github.com/discourse/discourse/blob/master/app/assets/javascripts/discourse/components/message_bus.js.coffee i’m interesting in server side.

7

I wrote the message bus so I should answer this. However, before I answer anything let me explain the message bus.

(in ruby code on the server)

# publish the string norris to a channel called /chuck
MessageBus.publish('/chuck', 'norris')

# publish the string 'secret' to a channel called /check, but only to these particular user_ids
MessageBus.publish('/chuck', 'norris', user_ids: [1,2,3]) 

# subscribe to the channel '/chuck' on the server 
MessageBus.subscribe('/chuck') do |msg|
   # yay, I got a message on the /chuck channel
   data = msg.data
   site_id = msg.site_id
   channel = msg.channel
   user_ids = msg.user_ids
   # a global ever increasing id for the message
   global_id = msg.global_id
   # a unique id for this message within the channel
   message_id = msg.message_id
end

# give me all the messages after local message id 10 on the /chuck channel
messages = MessageBus.backlog('/chuck', 10)

# the last local id on the chuck channel
id = MessageBus.last_id('/chuck')

On the client side you have (in JavaScript)

MessageBus.subscribe('/chuck', function(data){
   // called when server publishes a message 
});

MessageBus.unsubscribe('/chuck', fn)

Why not use Faye?

Faye is an awesome project I have nothing bad to say about it. But, my intention around message bus is both wider and narrower than Faye.

Faye supports the full Bayeux protocol, it abstract transport and storage so you can plug in redis and websockets if you wish. It has a node and a ruby port.

Faye, by design, is a lot of things to a lot of people, Message Bus on the other hand has a much more specific use case and a lot of the decisions I made reflect that.

  • Message Bus is opinionated, it only supports the protocol it needs to drive Discourse. It only supports redis for storage. Message Bus does not support web sockets. It only supports polling and long polling.

  • Message Bus is multi-host aware. We serve both http://meta.discoruse.org and http://try.discourse.org from the same pool of processes. Message Bus has smart enough routing to ensure only the correct site gets the messages targeted at it.

  • Message Bus is efficient and stores no client state. Many storage strategies will save up messages in “client” buckets (Faye does this) This means that when you are distributing messages you need to add one to each client bucket that cares about it and have to worry about expiring this bucket at some point. Message Bus on the other hand stores the backlogs on a channel backlog. This allows clients to recover from lost messages days later if they are still around in the channel bucket.

  • Message Bus is replayable: At any point you can request a backlog of all the messages on a channel (you can control how big you allow the backlog to get)

  • Message Bus is small: the entire implementation fits in a handful of files, see: https://github.com/SamSaffron/message_bus/tree/master/lib/message_bus because it only supports a limited protocol the code can be a lot smaller.

  • Message Bus is robust: not many buses can pass a test like this: https://github.com/SamSaffron/message_bus/blob/master/spec/lib/multi_process_spec.rb

  • Message Bus is used for intra-server comms. If you are running Discourse over 3 machines and need to expire a cache you can use Message Bus for that.

Historically, I originally did not use Faye cause I wanted to use em-websocket, in fact I even wrote integration bits to allow for em-websocket support in thin. em-websocket is by far the most complete socket implementation in ruby and it is far more complete than Fayes. Since then I have changed my tune.

These days I don’t really believe the complexity added by having web socket support really buys you much over long polling. Additionally, if you really must have* web sockets reliably, you must be using HTTPS and you got to have robust fallback logic, just in case. Various rewriting proxies prevalent on mobile and planes/hotels will muck with traffic cause sockets to hang.

It was also critical for me that the same process that runs our webs can serve the sockets on the same port (something I was not able to do with Faye web sockets). I wanted to ease deployment as much as I could.

30 Likes
8

How does the message bus determine whether a particular message needs to be delivered to a particular recipient (who’s asking) or not? In other words - how does it keep track of security?

Consider the situation: there’s a private conversation which you are invited to. There are 3 messages in the Message Bus. First one invites you to the conversation. Second one is a post in it. Third one kicks you from the conversation.

If a browser tab needs to get all three, you shouldn’t send any, because in the end you are kicked. If it needs something from the middle, you should just send the “kick” message. Etc.

Basically you need to keep track of the “inivitation status” for each tab individually. Which seems like a lot of overhead/complexity, when you factor in that there are a lot of potential conversations to keep track of, lot of different statuses to keep track of, etc.

1 Like
9

When you publish a message you specify who is allowed to see it:

# only user id 100 will get secret santa
MessageBus.publish('/secret', 'santa', user_ids: [100])

The middleware will not distribute messages to users who are not allowed to see them

1 Like
10

Forgive my lack of rails knowledge, but can you talk about the nuts and bolts of how this works? Does the handler proc run on a thread that is dedicated to getting MessageBus notifications? I ask because I was under the impression that trying to do multithreaded stuff in rails is kind of asking for trouble (I have run across a lot of gems that are not threadsafe and the excuse is usually, “but rails!”).

1 Like
11

The api has a blocking subscribe for the less brave, Discourse supports multi threading even in a multi tenanted case. In fact we patch Active Record to support it. Allow connection_handler to be overriden per-thread by SamSaffron · Pull Request #8368 · rails/rails · GitHub

Message bus is aware of the site it is operating on and takes care of opening the correct db connection using rails multi site. https://github.com/discourse/discourse/tree/master/vendor/gems/rails_multisite … and let me pre-empt the “why not use the apartment gem”. The apartment gem is not multi tenant / multi thread safe. It does not monkey patch AR into compliance.

1 Like
12

Doesn’t this result in some rather long lists of user id’s attached to messages? Also - how is this stored in the DB? Table allowances(message_id, user_id)? For a long list of IDs that would mean a lot of INSERT’s, no? Or is it stored as a CSV column in the messages table (which would then complicate SELECT queries)?

1 Like
13

There is no table, its stored in redis in a sorted list, and the redis message bus takes care of delivering the messages in “sort of order” so there is very little work checking permissions.

1 Like
14

Ah, I see. Thanks! :smile:

1 Like
15

A little off-topic, but I’m confused - was it you who wrote and nowdays maintains the messagebus.com service?

1 Like
16

When @sam is talking about the message bus, he’s talking about the library he wrote :wink:

2 Likes
17

Ah sorry, in the original topic it was “MessageBus” - so it looked more like a service name )

1 Like
18

@sam I’m looking to do some simple channel-based chat between ios/android clients and a server. Would message-bus be a good fit for this? I can publish (via HTTP post) from the clients. The only difficulty is the subscribe portion. On the server, I would need to get a callback from message-bus subscribe event so i can do a (apns/gcm) push to the clients ( I don’t want to do a socket connection).

Is this worth pursuing?

1 Like
19

Server side is pretty simple, definitely worth a look imo. Lots of different layers there are reusable.

1 Like
20

I got it working, check it out: YouTube
—
Thanks,
Alex Egg

3 Likes
21

With the message_bus gem?

1 Like