The main problem with the AI “Related Topics” thing is that you have to generate embeddings for all your existing topics. In large forums that takes a while, and is the “expensive” part of the operation. However, you only need to run this once, so you can leverage hourly instances to pay just the bare minimum here.
Generating embeddings for Meta took me around 1h. So you can, for example, get a VPS with a full NVIDIA A100 for one hour to run it and pay less than $3 for it!
After you have all those embeddings already generated, you only need to generate new ones for new and edited topics, and there you can probably fly with CPU-based inference.
So let’s say you now have:
- One droplet at Digital Ocean running Discourse
During the backfill you can have:
- One droplet at Digital Ocean running Discourse
- One droplet at Digital Ocean running PostgreSQL for storing the embeddings
- One VPS at Vultr for computing embeddings fast
After the backfill you change it to:
- One droplet at Digital Ocean running Discourse
- One droplet at Digital Ocean running PostgreSQL for storing the embeddings and now also the embeddings service
As for the droplet size for the 2, a small one with 4GB RAM may be enough, gotta double check how much RAM that embeddings service container is using.