I made an interesting discovery: the web server that I’m hosting my forum on has sufficient grunt to run a small LLM at modest speeds (6 tok/s without batching) even without a GPU. This will be useful for offline/background tasks.
I made an interesting discovery: the web server that I’m hosting my forum on has sufficient grunt to run a small LLM at modest speeds (6 tok/s without batching) even without a GPU. This will be useful for offline/background tasks.