# Bot schrijftip: Elk bericht verwerken

**URL:** https://meta.discourse.org/t/bot-writers-tip-processing-every-post/172697
**Category:** Extras
**Created:** [9 december 2020 om 00:37 UTC](https://meta.discourse.org/t/bot-writers-tip-processing-every-post/172697 "2020-12-09T00:37:52Z")
**Posts on this page:** 1
**Page:** 1

<div class="post-metadata">

### Author: ![riking](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/riking/32/170938_2.png) [@riking](https://meta.discourse.org/u/riking)
#### Post date: [9 december 2020 om 00:37 UTC](https://meta.discourse.org/t/bot-writers-tip-processing-every-post/172697/1 "2020-12-09T00:37:52Z")

</div>

> ℹ Note: This guide assumes you are operating an authorized bot on a Discourse forum, potentially using the [User API](https://meta.discourse.org/t/user-api-keys-specification/48536) or an [Admin API Key](https://meta.discourse.org/t/discourse-api-documentation/22706). If your bot is blocked by the admins, discuss the purpose of your bot with them and do not attempt to circumvent that block.

> ℹ Would your bot be better if it was run on the server? Consider creating a plugin instead: [Developing Discourse Plugins - Part 1 - Create a basic plugin](https://meta.discourse.org/t/beginners-guide-to-creating-discourse-plugins-part-1/30515)

## Introduction

This guide will present an algorithm for a bot user to inspect and process every post made on a Discourse forum that the bot’s user is allowed to access (aside from private messages).

You will need durable storage for a single integer, the highest successfully processed post ID. For example, you could write this to Redis or to a plaintext file. Using Redis will allow you to persist message bus subscriptions across process restarts on your end.

It is highly recommended to **create a brand-new user account for the bot** , so that it can be added to groups and private messages as necessary. Avoid using the @system account.

The following set of algorithms is written in an imitation of the WHATWG specification style, and gradually builds up to the algorithm to _continuously monitor for new posts_.

## Algorithm Specifications

Let the **forum base URL** be the URL to the site with no trailing slash - e.g. `https://meta.discourse.org` or `https://www.contoso.com/forum` for subfolder installations.

### Fetch the next recent posts

To **fetch the next recent posts** given an integer _highest seen post ID_, and a flag _triggered by message bus_, run these steps:

- Let **maximum response post ID** be the result of adding fifty (50) to the _highest seen post ID_.
- Let **request uri** be the concatenation of the _forum base URL_, `/posts.json`, `?before=`, and the _maximum response post ID_.
- Let **response** be the result of _🛰 fetch JSON respecting rate limits_ with the _request uri_ and the _credentials_.
- If _response_ is an HTTP error, abort these steps with an error.
- Let **posts** be the JSON array at path `latest_posts` inside _response_.
- Let the **new posts seen** flag be unset.
- For each JSON object _post_ in _posts_ in reverse order, execute these steps:
  - Let **post ID** be the JSON number at path `id` inside _post_.
  - Set _highest seen post ID_ to _post ID_.
  - Set the _new posts seen_ flag.
  - ✅ Emit _post_. (ℹ In other words: Send the post to whatever custom processing you want to perform.)
  - If _emit_ returned a backpressure signal, break this loop.

- 

> ℹ The above loop executes in reverse order so that your code sees the oldest posts first and the newest posts last.

- If the _new posts seen_ flag is set:
  - Execute the steps for _💾 persisting state to storage_ with _highest seen post ID_.

- End these steps returning the _highest seen post ID_.

### Probe a high existing post ID

To **probe a high existing post ID** , execute these steps:

- Let **latest probe request uri** be the concatenation of the _forum base URL_ and `/posts.json`.
- Let **latest probe response** be the result of _🛰 fetch JSON respecting ratelimits_ with the _latest probe request uri_ and the _credentials_.
- If _latest probe response_ is an HTTP error, abort these steps with an error.
- Let **probe posts** be the JSON array at path `latest posts` inside _latest probe response_.
- For each JSON object _post_ in _probe posts_:
  - Let **post id** be the JSON number at path `id` inside _post_.
  - End these steps, returning _post id_.

- Abort these steps with an error.

### Backfill from latest

To **backfill from latest** given an optional integer _highest seen post ID_, execute these steps:

- Let **minimum post ID** be the _highest seen post ID_ if present, and zero (0) otherwise.
- Let **high existing post ID** be the result of _probe a high existing post ID_.
- If _maximum post ID_ is an error, abort these steps with an error.
- Execute the steps to _backfill_ given the _minimum post ID_ and the _high existing post ID_.

### Backfill

To **backfill** given two integers _minimum post ID_ and _high existing post ID_:

- Let **current minimum post ID** be _minimum post ID_.
- Repeat these steps:
  - Execute the steps to _fetch the next recent posts_ given the _current minimum post ID_ and an unset _triggered by message bus_ flag.
  - If the steps to _fetch the next recent posts_ did not complete successfully:
    - Update the exponential backoff algorithm with a failure signal, and wait the specified amount of time.
    - Continue to the next loop iteration (without updating the the _current minimum post ID_).

  - Let **candidate maximum response post ID** be the result of adding fifty (50) to the _current minimum post ID_.
  - If the _candidate maximum response post ID_ is greater than or equal to the _high existing post ID_, ✅ end these steps.
  - Set the _current minimum post ID_ to the _candidate maximum response post ID_.

### Continuously monitor for new posts

To **continuously monitor for new posts** , execute these steps:

- Let **highest seen post ID** be an unset optional integer.
- Set _highest seen post ID_ to the result of **▶ restoring state from storage**.
- If _highest seen post ID_ is unset:
  - Set **initial post ID** to the result of _probing a high existing post ID_.
  - Execute the steps for _💾 persisting state to storage_ with _initial post ID_.
  - Set _highest seen post ID_ to _initial post ID_.

- Set **notifications** to the result of executing the steps to _🛰 subscribe to the message bus_, with a channel of `/latest`.
- Execute the following steps repeatedly:
  - Set **new highest seen post ID** to the result of _fetching the next recent posts_, with the _triggered by message bus_ flag set if a message bus update occurred, and the _highest seen post ID_.
  - If the steps to _fetch the next recent posts_ did not complete successfully:
    - Update the exponential backoff algorithm with a failure signal, and wait the specified amount of time.
    - Continue to the next loop iteration.

  - If the _new highest seen post ID_ is different from the _highest seen post ID_:
    - Update the exponential backoff algorithm with a success signal.
    - Set the _highest seen post ID_ to the _new highest seen post ID_.

  - Wait for a message on _notifications_ or for an implementation-defined timeout to occur. This timeout must be no shorter than 10 minutes and may reasonably range up to 24 hours or slightly higher.

Algorithms you need to provide:

- 🛰 fetch JSON respecting rate limits, taking a request uri and optional credentials.
  - This must automatically back off and retry using the exponential backoff algorithm and/or the server-provided `Retry-After` information when presented with a 429 error.

- ▶ restoring state from storage
- 💾 persisting state to storage, taking an integer
- 🛰 subscribe to the message bus

* * *