Manual configuration For VLLM

That is the hostname of the machine where you are running vLLM. It may also work with an IP in the form of http://1.1.1.1:1111 but I have not tested it as such.

We have a few tokenizers to helps us limit the size of the prompts before we send it to the LLM. Pick whatever produces closest results to the tokenizer used by the model you are running in vLLM, it doesn’t really need to be a perfect match.

When in doubt leave it as the OpenAI or Lllama3 one.