I’m running several different things. For Discourse stuff, I will run a 7B model based off Mistral and fine-tuned for the tasks. I’m looking at various BERT-like models for classification tasks and still undecided on the embeddings yet. This runs on a 2nd hand 3090 Ti which I bought for $700.
I would love to have an A100, but instead, I built a separate 4 GPU system ‘on the cheap’ for only $1,000 that runs Llama 3 70Bq4 at over 20 tok/s.
For sure in many/most cases it would make sense to just go with a provider, however, it might make sense to DIY if:
- You want to learn
- You want to have control certainty over your models (so you don’t lose access to them, or are beholden to a company to use their non-public embeddings)
- You have a lot of bulk processing to do which would be cheaper to do in-house
- You want reserved and reliable capacity (there are limits on both requests and tokens available from providers) for bulk processing