I’m trying to add a custom LLM to Discourse AI plugin. When I press the ‘test’ button I get “Internal Server Error”. Is there a way of debugging this or getting a better error message? When I go into the docker image and curl the /v1/models, I’m able to fetch this correctly. The model name is “models/Meta-Llama-3-8B-Instruct.Q6_K.gguf” and I’m not sure whether there could be any issue with special characters.

Trying another one gives: Trying to contact the model returned this error: {"error":{"code":404,"message":"File Not Found","type":"not_found_error"}} But it doesn’t display what URL/model it is trying to fetch which might help to debug. The same settings were pasted into Open WebUI which was able to contact both LLM endpoints and inference correctly.

What inference server are you using? vLLM? When configuring the URL, add the path /v1/chat/completions at the end.

[image] Falco: When configuring the URL, add the path /v1/chat/completions at the end This was the issue. Note that in LLM software, it is customary to include only upto the /v1 as the endpoint URL. Selection of /chat/completion etc. is then normally added by the software.

I’m trying to get one running on the localhost to test so put the URL as: “http://172.17.0.1:8081/v1/chat/completions” and get internal server error. I’m able to curl “http://172.17.0.1:8081/v1/models” from the discourse docker container so the connectivity is working. Are there any other gotchas (e.g. does Discourse allow non-https and arbitrary ports for the LLM endpoint)?

[image] Isambard: Are there any other gotchas (e.g. does Discourse allow non-https and arbitrary ports for the LLM endpoint)? Both should work. [image] Isambard: I’m trying to get one running on the localhost to test so put the URL as: “http://172.17.0.1:8081/v1/chat/completions” and get internal server error What is the error you see on /logs ?

[image] Falco: What is the error you see on /logs ? Ah. I didn’t know about /logs! NameError (undefined local variable or method 'tokenizer' for an instance of DiscourseAi::Completions::Dialects::ChatGpt) app/controllers/application_controller.rb:424:in 'block in with_resolved_local Hmm. The one that works is for a model that I quantized myself. I’ll try to quantize the others to see if it is a model format issue.

There’s a topic here about it [image] Can I add other LLM? Support It works with VLLM, you can can just configure a VLLM endpoint and it should work out of the box.

Debugging adding new LLM

Product Support

Isambard (Isambard) August 23, 2024, 11:43am 8

Anyone managed to get DeepSeek API working? I’m trying to figure out the right incantation to get it to work with Discourse.

I have it working in Open WebUI and other clients.

Topic		Replies	Views
Internal Server Error 500- Manual configuration [ Discourse AI] Product Support ai	7	511	September 5, 2025
How to use internal AI endpoints? Product Support ai	8	538	July 6, 2026
How do I use hugging face paid inference endpoints as Discourse custom LLMs Product Support ai , ai-bot	2	289	December 9, 2024
Discourse AI with local ollama Internal Server Error Bug ai	1	271	December 27, 2025
Self-Hosting an OpenSource LLM for DiscourseAI Self-Hosting ai	12	4590	July 6, 2026

Debugging adding new LLM

Related topics