Upgrading our front end models to use a store


(Robin Ward) #1

The long battle of upgrading Discourse’s Javascript from constants to ES6 modules finally has an end in sight. The vast majority of Javascript files we use to serve up Discourse have been converted to ES6 modules and are working great in production.

Having said that, there are some holdouts in our code base that have not been converted yet, mainly because they are difficult. In particular, our Model/data fetching infrastructure hasn’t been upgraded in years and has a lot of room for improvement.

The Old Way to fetch data

Right now, the standard way to fetch data is to perform a Discourse.ajax call, then to wrap the result in an instance of the model you want. For example:

Widget = Ember.Object.extend();

Widget.reopenClass({
  find(id) {
    return Discourse.ajax("/widget/" + id + ".json").then((result) => {
      return Widget.create(result);
    });
  }
});

With the above code, you could say Widget.find(123) and get a promise that will resolve into a Widget.

This approach is not bad, it has obviously scaled up to our needs so far and kept our data layer simple. However, there is a lot of room for improvement:

  • We repeat ourselves a lot. On the server side we use a REST convention but the client side ends up making the same kinds of calls a lot.

  • For more complex objects (that contain side loaded or embedded objects) the finders become more complex.

  • It has no concept of an identity map, introducing potential bugs as the same object can be loaded twice in memory.

The New way to fetch Data

The new Discourse betas have a store injected into routes, controllers and models. If you’ve used ember-data in the past this concept is familiar to you and that is no coincidence. The new API is similar to ember data but is based off Discourse’s needs and is considerably simpler. It can be seen as us stealing many of the ideas of ember data but without going full hog down that road.

I’ve been slowly building up the store over the last few features I’ve built and the API is finally stable and good enough to use so I recommend it for all new features.

The core idea is you ask the store for data. For example, to find the widget in the example about you’d do this instead:

this.store.find('pet', 123);

The above code will automatically make a GET request to /pets/123.json. You don’t have to write any more code in Discourse to do this. You don’t have to define a pet model or hardcode the pets path or anything like that. The store will make the request, find the pet in the returned JSON and return it to you as an instance of RestModel.

RestModel is the new base class for our models. Since we didn’t define a Pet model, we’ll just get an instance of that. However, if we did want to return an instace of Pet instead we could define a class like this.

// discourse/models/pet.js.es6
import RestModel from 'discourse/models/rest';

export default RestModel.extend({
  bark() { 
    console.log('woof');
  }
});

If a Pet model is defined, the store will automatically instantiate it for you. If not, no bother.

RestModel instances come with some handy methods. For example, let’s say you want to update a Pet’s name. You could do this:


pet.update({ name: 'rover' });

That will automatically make a PUT request to /pets/:pet_id with the name attribute. Again this is based on Discoruse’s conventions. (If you write your endpoints in our conventional way, you have to add zero client side code to handle this update.)

Result Sets

To find more than one of a thing, you can call this.store.findAll('pet'). This will make a GET to /pets.json and instantiate records for each Pet. You can also call this.store.find('pet', {dog: true}) and it will call /pets.json?dog=true as a filter.

Any time you get back multiple records from the store, it will look like a array, but it’s actually a ResultSet. For almost all your code the difference doesn’t matter, you can still loop through it and use all the Ember array functions. However, it has some added functionality to make our lives easier.

One thing we do a LOT is loading more data with scrolling. We used to have to code this over and over, so I made a common API for it. If a JSON result includes total_rows_pets and load_more_pets (the later is a URL to load more), the ResultSet will remember those attributes.

You can then just call model.loadMore() to load more results and it will call the remote URL and fetch more records. This means you have to write a lot less code.

RestModel States

We have a lot of repeated code that handles the cases of whether a model is currently saving. You’ll see a lot of code that sets a saving property while a promise is resolving, then updating it to false when it finishes. With RestModel this is done automatically. While you are making the above update call, it will automatically set isSaving to true on the model. So if your template looks for that, it will be true while saving and false when it isn’t. This should cut out a LOT of code for us.

Creating and Destroying Records

If you want to make a new instance of something that’s easy:

const pet = this.store.createRecord('pet', {name: 'woofie'});

At that point, it will have isNew set to true. You can bind it to a form or do whatever you need to input data for the new model. When you are ready to save it, just call pet.save() and it will be sent across the wire. When a record is created, it makes a POST to /pets with the data, just like our conventions.

To destroy a record, just call pet.destroyRecord() and you’re good to go. It will make a DELETE to /pets/:pet_id.

Loading Sideloaded Objects

Often you want to return a list of items that include associated data. For example our QueuedPostSerializer embeds a user and a topic. By default, ActiveModel::Serializers will sideload this data, which means that instead of every queued_post in the json having an embedded user and topic property, it will have a user_id and topic_id, and then separate users and topics collections on the root of the JSON that contains those objects. This can cut down on the JSON size quite a bit, but is difficult to instantiate on the client side.

With the new store code, there is a way to do this automatically! When serializing, add the rest_serializer: true option:

render_serialized(@queued_posts, QueuedPostSerializer, root: :queued_posts, rest_serializer: true)

If you do this, this.store.find('queued-posts') will automatically load and instantiate the sideloaded data! Any key that begins with something_id will look for a somethings collection in the root and instantiate Something models if they exist.

(As a bonus, I’ve added it so that if you include category_id anywhere in a rest_serializer: true dump, the category will automatically be found. We always have the categories loaded client side so there is no need to serialize anything but the id).

Identity Map

Anything found via the store will use an identity map. So if you retrieve the same user by id twice, you are guaranteed to be pointing at the same instance in memory. Updating it in one place will update it everywhere that it is bound. Note this is true even if the user comes back in a collection or side loaded as explained above.

Munging Data

Sometimes the data that comes back from the server isn’t right for you. I’d highly suggest trying to use the rest_serializer: true option but if that can’t work for you, you can just implement the munge() method. For example:

const Pet = RestModel.extend();

Pet.reopenClass({
  munge(json) {
     // weight comes back as lbs but we want kgs
     json.weight = json.weight * 0.453592;
     return json;
  }
});

If a munge method is present JSON retrieved from the server will be passed through it before instantiated so you can modify it. Again, try not to do this unless dealing with old models or code.

Adapters

What if your JSON endpoints don’t follow our RESTful conventions? You should definitely try to make everything follow the discourse conventions, but if you can’t do that there is a way to modify where data is sent/retrieved.

If you define an adapter in discourse/adapters, it will be used to contact the server instead of the default REST method. For example, I implemented one of these for topic-list because it uses the PreloadStore to fetch data quickly. It will first look for the model in the PreloadStore and if it doesn’t exist will use a custom finder.

Adapters are useful for when you don’t control the JSON endpoint or the JSON endpoint doesn’t work with our conventions. Most of the time going forward you shouldn’t need to use them, but they are there if you need to do something different.

Conclusion

I’ve been using this store and RestModel in the last few features I’ve written and it has really made a huge difference in the amount of code I’ve written. The API is pretty good now, but I would love suggestions/tweaks if people have them.


Plugin Development: record.save() returns 404
(Vinoth Kannan) #2

I recently created two plugins using the REST model. Both plugins will read exactly same data using store api like below.

this.store.findAll('menu-link').then(function(rs) {

});

When I run both plugins in same website fetching GET /menu_links.json twice in a same page view. The store calling server twice instead of getting existing data from the local cache or something. Is there any option to do?

Edit: one plugin calling it inside a widget and another one calling it from initializers.


(Robin Ward) #3

Calling findAll will always make a call to the server, however the API uses an identity map to make sure the same objects are used and updated. So if you call it twice, all the templates that are bound to it will update.

Instead you could wrap it in a service or another function that retains the result, and call that from both places.

Also I should note that widgets aren’t really meant to retrieve data. They can do it, but it usually makes a lot more sense to retrieve the data in a route or controller and pass it down to the widgets.


Are there any examples in Discourse of decorating a widget after an ajax request?
(Vinoth Kannan) #4

thanks for the information.

My plugin is to display a custom menu. In this case can I create new route which respond in all pages (Since I should display the menu in all pages). Also is this possible to link controllers with widgets?

I will look into the ember documentations in meantime.


(Robin Ward) #5

The easiest way is probably to create a function you can call that will only retrieve the store data once. You can consider it like a service object. The Widget stuff is not ember though, it is our own home grown view layer for our performance hot path. But it can call any regular Javascript functions or libraries you create.


#6

Hello. I’m not sure if this is the right place, but I noticed some small inconsistencies with the store.

  • The method _saveNew will return an object with the target property referencing the record, the method update won’t. It would be convenient to have the record object be part of both results. In general, both result objects are very different. Is this on purpose?

  • The following will throw an error, but in the console, it will just read undefined.

    rest.js.es6#L62:

    throw "You must overwrite `createProperties()` before saving a record";
    

    Something like this will produce a more helpful stacktrace:

    throw new Error("You must overwrite `createProperties()` before saving a record");
    

    There is several places in the code with this issue.

  • The method createProperties needs to be implemented in one’s own RestModel extension, as hinted by the error it throws if one doesn’t. The same should be true for updateProperties. Something along the lines of:

    updateProperties() {
      throw new Error("You must overwrite `updateProperties()` before updating a record");
    }
    

(Robin Ward) #7

These are all good suggestions. If you’d like to make PRs for these (with tests) I’d gladly merge them!


#8

(I submitted Prefer throwing a new Error object instead of just a string expression by kleinfreund · Pull Request #5928 · discourse/discourse · GitHub tackling the first part of the mentioned issues.)


I have a question. Calling createRecord like this is different than the example in the original post where no id property is provided.

const record = this.store.createRecord('pet', { id: 1, name: 'Relojero Pajaro' });
record.save()
  .then(result => {
    console.log(result);
  });

Providing the id property leads to a PUT request to /pets/:pet_id being made. Without the ID, that would be a POST request to /pets. That’s intentional, right? And it’s expected to always return the created resource?

Now, if I always provide the id property when calling createRecord, I won’t need a route for post requests in my plugin at all, right?


(Robin Ward) #9

Yes, the Rails convention for CRUD is that “create” does a POST and “update” does a PUT. Our store makes the assumption that if a record has an id already it is an “update” when you call save().