# Debugging adding new LLM
**URL:** https://meta.discourse.org/t/debugging-adding-new-llm/321362
**Category:** Support
**Tags:** ai
**Created:** [August 13, 2024, 12:01am UTC](https://meta.discourse.org/t/debugging-adding-new-llm/321362 "2024-08-13T00:01:54Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Isambard](https://meta.discourse.org/u/Isambard)
#### Post date: [August 13, 2024, 12:01am UTC](https://meta.discourse.org/t/debugging-adding-new-llm/321362/1 "2024-08-13T00:01:54Z")
I’m trying to add a custom LLM to Discourse AI plugin. When I press the ‘test’ button I get “Internal Server Error”.
Is there a way of debugging this or getting a better error message? When I go into the docker image and curl the /v1/models, I’m able to fetch this correctly.
The model name is “models/Meta-Llama-3-8B-Instruct.Q6\_K.gguf” and I’m not sure whether there could be any issue with special characters.
---
### Author:  [@Isambard](https://meta.discourse.org/u/Isambard)
#### Post date: [August 13, 2024, 12:15am UTC](https://meta.discourse.org/t/debugging-adding-new-llm/321362/2 "2024-08-13T00:15:06Z")
Trying another one gives: `Trying to contact the model returned this error: {"error":{"code":404,"message":"File Not Found","type":"not_found_error"}}`
But it doesn’t display what URL/model it is trying to fetch which might help to debug.
The same settings were pasted into Open WebUI which was able to contact both LLM endpoints and inference correctly.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [August 13, 2024, 12:23am UTC](https://meta.discourse.org/t/debugging-adding-new-llm/321362/3 "2024-08-13T00:23:32Z")
What inference server are you using? vLLM?
When configuring the URL, add the path `/v1/chat/completions` at the end.
---
### Author:  [@Isambard](https://meta.discourse.org/u/Isambard)
#### Post date: [August 13, 2024, 7:13am UTC](https://meta.discourse.org/t/debugging-adding-new-llm/321362/4 "2024-08-13T07:13:51Z")
> [@Falco](#):
>
> When configuring the URL, add the path `/v1/chat/completions` at the end
This was the issue. Note that in LLM software, it is customary to include only upto the /v1 as the endpoint URL. Selection of /chat/completion etc. is then normally added by the software.
---
### Author:  [@Isambard](https://meta.discourse.org/u/Isambard)
#### Post date: [August 13, 2024, 7:26am UTC](https://meta.discourse.org/t/debugging-adding-new-llm/321362/5 "2024-08-13T07:26:31Z")
I’m trying to get one running on the localhost to test so put the URL as: “[http://172.17.0.1:8081/v1/chat/completions](http://172.17.0.1:8081/v1/chat/completions)” and get internal server error. I’m able to curl “[http://172.17.0.1:8081/v1/models](http://172.17.0.1:8081/v1/models)” from the discourse docker container so the connectivity is working.
Are there any other gotchas (e.g. does Discourse allow non-https and arbitrary ports for the LLM endpoint)?
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [August 13, 2024, 1:51pm UTC](https://meta.discourse.org/t/debugging-adding-new-llm/321362/6 "2024-08-13T13:51:19Z")
> [@Isambard](#):
>
> Are there any other gotchas (e.g. does Discourse allow non-https and arbitrary ports for the LLM endpoint)?
Both should work.
> [@Isambard](#):
>
> I’m trying to get one running on the localhost to test so put the URL as: “[http://172.17.0.1:8081/v1/chat/completions”](http://172.17.0.1:8081/v1/chat/completions%E2%80%9D) and get internal server error
What is the error you see on `/logs` ?
---
### Author:  [@Isambard](https://meta.discourse.org/u/Isambard)
#### Post date: [August 13, 2024, 5:36pm UTC](https://meta.discourse.org/t/debugging-adding-new-llm/321362/7 "2024-08-13T17:36:15Z")
> [@Falco](#):
>
> What is the error you see on `/logs` ?
Ah. I didn’t know about /logs!
`NameError (undefined local variable or method 'tokenizer' for an instance of DiscourseAi::Completions::Dialects::ChatGpt) app/controllers/application_controller.rb:424:in 'block in with_resolved_local`
Hmm. The one that works is for a model that I quantized myself. I’ll try to quantize the others to see if it is a model format issue.
---
### Author:  [@Isambard](https://meta.discourse.org/u/Isambard)
#### Post date: [August 23, 2024, 11:43am UTC](https://meta.discourse.org/t/debugging-adding-new-llm/321362/8 "2024-08-23T11:43:31Z")
Anyone managed to get [DeepSeek](https://platform.deepseek.com/api-docs/api/create-chat-completion) API working? I’m trying to figure out the right incantation to get it to work with Discourse.
I have it working in Open WebUI and other clients.
---
### Author:  [@Lilly](https://meta.discourse.org/u/Lilly)
#### Post date: [August 23, 2024, 2:11pm UTC](https://meta.discourse.org/t/debugging-adding-new-llm/321362/9 "2024-08-23T14:11:45Z")
There’s a topic here about it
> [@Can I add other LLM?](https://meta.discourse.org/t/can-i-add-other-llm/319540/3):
>
> It works with VLLM, you can can just configure a VLLM endpoint and it should work out of the box.