A tour of how the Widget (Virtual DOM) code in Discourse works

The latest builds of Discourse are much faster at rendering topics thanks to our re-written post stream. I’ve written up our new plugin API but so far haven’t explained how the code all fits together. The purpose of this topic is to allow Discourse developers to understand how the new code works.

What’s a Virtual DOM?

A Virtual DOM is a data structure that enables browsers to re-render dynamic content very quickly. The technique was pioneered by React but has since been integrated into many other frameworks.

The basic idea is that updating the HTML content of a browser is slow, so it’s much faster to update a data structure in memory, then figure out what changed (knowng as diffing), so you can instruct the browser to only update exactly what changed.

Discourse uses the virtual-dom library to perform all the hard work. However, that library is a bit too low level, so it’s been abstracted in Discourse by a series of classes called Widgets.

Anatomy of a Widget

A Widget is a class with a function called html() that produces the virtual dom necessary to render itself. Here’s an example of a simple Widget:

import { createWidget } from 'discourse/widgets/widget';

createWidget('my-widget', {
  tagName: 'div.hello',

  html() {
    return "hello world";
  }
});

The above code registers a widget called my-widget, which will be rendered in the browser as <div class='hello'>hello world</div>. It can be rendered in any of our Handlebars templates using the mount-widget helper:

{{mount-widget widget="my-widget"}}

If you wanted to return more HTML, you can include the h helper and use it to emit more formatting:

import { createWidget } from 'discourse/widgets/widget';
import { h } from 'virtual-dom';

createWidget('my-widget', {
  tagName: 'div.hello',

  html() {
    return h('span.greeting', h('b', "hello world"));
  }
});

The above would render:

<div class="hello"><span class="greeting"><b>hello world</b></span></div>

Rendering Attributes

So far our widget is 100% static, and in reality Discourse’s code is quite dynamic. When a widget is rendered, it is passed a series of attributes. In our post stream, for example, these attributes are a plain Javascript object with the attributes of a post.

Aside: you might wonder why we don’t render from our Post Ember object itself. This is done for performance reasons. In Ember, calling get on a property has overhead that, while small, adds up when you are rendering as fast as possible. In our old code we would be calling get repeatedly, frequently on the same property rather than caching it locally. In the new code, we transform a Post ember model into a plain javascript object that we don’t have to call get on to retrieve attributes.

(I should add that this is one of the uglier aspects of the new post stream code, and I’m not 100% sure we need to do it. It may change.)

You can pass attributes to be rendered by a widget using the args property. Let’s say you had a user object with {name: 'Robin'} in your template. You can pass it to be rendered in the widget like so:

{{mount-widget widget="display-name" args=user}}

Then your widget would be passed the user object as the first argument to html():

import { createWidget } from 'discourse/widgets/widget';
import { h } from 'virtual-dom';

createWidget('display-name', {
  tagName: 'div.name',

  html(attrs) { 
    return h('b', attrs.name);  
  }
});

The final HTML would be:

<div class="name"><b>Robin</b></div>

Widgets can also render other widgets. Use attach to render a widget in an html method:

html(attrs) {
  return this.attach('another-widget', attrs);
}

It’s widgets all the way down!

State and Actions

Widgets can also contain state. You can think of state as data that you want to keep around the next time your widget is rendered. (For example, if you wanted to remember if a post is collapsed or expanded, it would make sense to store that state on the widget itself, and not the the post itself.)

To add state to your widget, create a defaultState method that returns an object that will represent its default state the first time the widget is rendered.

Each widget that has state also needs a unique key, so we have a function buildKey which returns a string. If you use the widget more than once you need to make this key unique, so you could for example base it on the id of an attribute that is passed in.

Your state is basically useless without a form of user interaction, so let’s also implement a click() function. click is called whenever the user clicks on your widget. Let’s look at it together:

import { createWidget } from 'discourse/widgets/widget';

createWidget('increment-button', {
  tagName: 'button',
  buildKey: () => 'increment-button',

  defaultState() {
    return { clicks: 0 };
  },

  html(attrs, state) { 
    return `Click me! ${state.clicks} clicks`;
  },

  click() {
    this.state.clicks++;
  }
});

The above code will render a button that, when clicked, will increment a counter.

Widgets can also send actions up. This means that you can call any pre-existing Discourse action. Simply pass it into the widget the same way you would for an Ember component:

{{mount-widget widget="my-widget" deleteThing="deleteThing"}}

Then in your click handler you can call sendWidgetAction to call it:

click() {
  this.sendWidgetAction('deleteThing');
}

Bindings and Rerendering Content

If you’ve been working with Ember for a while, you might be wondering how the widgets know when they need to be rerendered since they aren’t using Ember’s bindings or observers system. In fact, this is one of the big trade offs widgets make: in Ember’s view rendering this is handled transparently for you, but with Widgets we assume the widget only needs to be re-rendered when you interact with them.

Widgets are rerendered when:

  • A user clicks on them

  • After any actions are triggered

  • If an action returns a promise, after that promise has resolved

The above cases count for the vast majority of the times when Discourse needs to rerender a widget. However, if you have an edge case, a widget can rerender itself by calling this.scheduleRerender().

Render calls are coalesced on the Ember Run Loop, so don’t be afraid to call this.scheduleRerender() multiple times in the same event loop – the widget will only render one time for all the changes you’ve queued.

Where to go from here

If all of the above makes sense, you are probably ready to jump into the codebase! In Discourse, all Widgets used to render the post stream are in the widgets folder.

There are also quite a few tests that are worth looking through to get an idea of how different parts of the application were implemented as well as how they can be tested.

Of course, I’m also around here on meta for questions, so feel free to ask if something doesn’t make sense. I probably missed something!

75 Likes

This is sweet, @eviltrout! Should really make reusable code more prevalent.

I’m working through my first use of widgets and literally copy/pasted your code.

/assets/javascripts/discourse/widgets/my-widget.js.es6

import { createWidget } from 'discourse/widgets/widget';
createWidget('my-widget', {
  tagName: 'div.hello',
  html() {
    return "hello world";
  }
});

Then I drop the following into a handlebars template:

{{mount-widget widget="my-widget"}}

But when I do that, I get a TypeError here:

https://github.com/discourse/discourse/blob/fa9943c162fa7d9fa038d29721c41b9f615ccaaf/app/assets/javascripts/discourse/components/mount-widget.js.es6#L58

If I try to reuse one of the existing widgets, it works fine. Any ideas?

Edit: I figured this out. You can’t place it in the widgets directory. The createWidget call needed to be in an initializer.

3 Likes

That’s unusual. You should be able to place the widget in the widgets directory within a plugin and it should be discovered. I just tried locally with a plugin and it worked.

Did you remember to rm -rf tmp after you added the widgets folder? That might help.

1 Like

That’s interesting. My typical routine when I make changes is:

  • Ctrl+c
  • rm -rf tmp
  • rails s
  • check for changes

The only time I don’t go through that process is when it’s on the Ember side as those don’t normally need tmp cleared. But in this case I still cleared it. I just tried it again and got the same error. I updated my local discourse right before checking. :confused:

1 Like

Is your plugin code public? If I could try it locally I could try to figure out what is incorrect.

Thanks for being willing to debug with me. I have the widget working through the initializer in the master branch. I broke it out into it’s own file in the widgets directory in the widget-breakout branch. As it is, that branch is giving me the error I mentioned above.

Aha, I see the issue! You weren’t exporting the widget from your ES6 module.

export default createWidget('who-voted', {

Thinking about it, you shouldn’t need to export something from a widget’s module to get it to load since you name the widget in createWidget. I’ve just committed a fix to autoload all widgets:

https://github.com/discourse/discourse/commit/fbf45426e411e0e38fc3d49538106fd8e08ccf4e

That should work if you’re on tests-passed, but if you want to support more versions of Discourse just export default from your widget file and it should work out.

4 Likes

Got it! That makes a lot of sense. The voting plugin is already dependent on v1.5.0.beta11. Most aren’t likely to use it until v1.5 is stable. If I leave it without the export that would make it dependent on v1.5.0.beta14?

Is there much difference performance-wise and functionally in one over the other? I’m still getting my head around Ember.

If you leave it without the export it will depend on v1.5.0.beta14 yes, or tests passed.

From your plugin’s point of view there should be no performance difference exporting versus not. In the Discourse codebase there is the question of when to parse the Javascript in the file. The fix means I’m doing it on boot versus when it is embedded in a template which might make loading discourse slower, but in practice probably makes no difference. I might revisit when widgets are loaded later to be more dynamic, but either way your plugin will be fine!

5 Likes

Is it possible to mount a widget into a post from decorateCooked? The polls plugin seems to do some funky stuff with Ember components/controllers/templates, and a set of widgets seem like they might be easier to work with.

1 Like

The polls code is definitely not something to imitate right now. When it was first created it made sense as a way to mount more ember code into something dynamically rendered, but now it is a bit of a hack. When I created the widget stuff it seemed vastly easier to maintain the old ember code than re-write it as vdom as there is quite a lot involved in rendering a poll.

If I had to do it again I’d just do it via widgets somehow. There is currently no way to render a widget in decorateCooked that I am aware of, but you could use attach other widgets before and after the cooked content to achieve a similar effect.

3 Likes

Ran into another hang up here. I have two widgets each created in separate files with export default createWidget(). In the first there is an action testAction(). I’m attempting to call that action from the second widget using this.sendWidgetAction("testAction") but I’m continually getting an error saying that the action testAction cannot be found. Is it possible to call an action across widgets?

Actions are meant to bubble up. If a widget embeds another widget using this.attach, then the attached widget will send the action up to the parent, its parent and so on.

There is no way to call a widget action for siblings. Like if B and C are attached to A, B cannot send an action to C.

4 Likes

@eviltrout, I’m hoping you can bear with me another time on this. I ran my development environment a couple hours after pulling from master to upgrade my local discourse instance and something happened to the voting plugin. It ran fine yesterday and as far as I know there have been no changes to it. Now none of the widgets render and all I can find for errors is this:

I’m not even sure where to start on this one but it seems as though something changed in the widget creation process. But they aren’t being mounted. It errors out in the attempt to mount them in the handlebars templates. Any guesses on what to look for?

Edit: It looks like this may be the source of the issue:

https://github.com/discourse/discourse/blob/514c3976f05785f487d275af1e96c43533e7a478/app/assets/javascripts/discourse/components/mount-widget.js.es6#L95

Why is it that when you post a question or a bug that you find the issue shortly afterwards?

Sorry for the trouble. Turns out this line led to the issue:

https://github.com/discourse/discourse/blob/514c3976f05785f487d275af1e96c43533e7a478/app/assets/javascripts/discourse/widgets/widget.js.es6#L129

If you define defaultState for a widget, but then leave it empty it will cause the plugin to fail. Makes sense but took a bit to find it. ¯\_(ツ)_/¯

2 Likes

Ah I see! That was me trying to be smart and tell people always to use a key when using state for a widget. defaultState should never return null though, so I’ve added another check to warn about that if that’s the case:

https://github.com/discourse/discourse/commit/d89fb6e83ef8b5f37b25673e80c964867f3c20fd

4 Likes

So building off of those new warnings, what keys are needed?

1 Like

Whenever a widget has state it needs a key so that the virtual dom can apply the state to the same element the next time it is repainted.

It’s pretty easy to do:

createWidget('my-widget', {
  buildKey: attrs => `my-widget-${attrs.id}`
});

That will build a key using the id of the thing being rendered. If it’s a thing attached to a post you’ll want to use the post’s id for example.

If there is only ever one of your widget, feel free to not even include an id and just return a string, like I do for the header: buildKey: () => 'header'

6 Likes

Thanx for the great explanation. One question though. How du I pass the model to a click action?

Solved it by ‘model=post’

2 Likes