Non-discovery API Pagination

Dev
1

I’ve recently been doing some work with the groups api and have some thoughts on pagination, specifically in the non-discovery context.

The context

For clarity, by the “discovery context” I mean the pagination in list_controller.rb and tags_controller.rb which use similar construct_url_with methods to generate previous and next page urls.

On that subject, these construct_url_with methods could perhaps be combined in an included module given they are quite similar, however this pagination is quite integral to topic list generation it is something of a seperate bucket.

The pattern

In the non-discovery context, the pagination uses some version of this:

page = params[:page].to_i
page_size = 38
items = items.offset(page * page_size).limit(page_size)

You can see a version of this pattern in various controllers, for example (not exhaustive):

  • groups
  • web hooks
  • emails
  • notifications
  • directory items
  • staff action logs
    …

There are a few issues with the pagination when using these endpoints from a third party client.

The issues

Specific: “load more” url inclusion

Some of these endpoints will return a “load more” url (with the “next” page), even if there are no more pages.

For example, there are 9 visible groups (to me) on meta. However will get a “load_more_groups” entry with a non-existant second page (page “1” is the second page as page numbering effectively starts at 0) in the .json for that page:

https://meta.discourse.org/g.json

...
"total_rows_groups":9,
"load_more_groups":"/groups?page=1"

This could perhaps be described as a “bug” of the endpoint as an API client will often look to use a “load more” url when iterating over pages, instead of attempting to handle page counting itself.

General: different nomenclature, page size handling and pagination data returned

More generally, there’s a fair bit of variation in pagination nomenclature, page sizes and what pagination data is returned in non-discovery contexts. For example

  • “page” is called either :page or :offset.
  • “total number of items” is sometimes included as ``total_rows_*" however it’s not always included and is sometimes called something else.
  • most page sizes are assigned to variables locally in their relevant action, and differ somewhat, meaning you need to go digging for them.

Suggestion

Now, there are various reasons for a number of the things I’ve raised. Many of the non-discovery endpoints are primarily (or only) consumed by the Discourse app client and are tailored to that client. That makes sense.

That said, there may be a move here that starts to incrementally move non-discovery pagination, particularly for endpoints that are often consumed by external clients (such as groups) toward more standardisation, which would also make it easier to address issues like the “load more” url inclusion.

Essentially what I have in mind is a “Pagination” module that codifies the pattern and could handle pagination in some selected endpoints, perhaps starting with Groups.

I could possibly make a PR for that, but thought I’d raise it for discussion first.

5 Likes