OpenAI Azure endoint support for gpt-4-32k?

Product Support Self-hosting

AI-generated summary

In a discussion titled ‘OpenAI Azure endpoint support for gpt-4-32k?’, user peyrusse reports issues with the Azure OpenAI endpoint when using gpt-4-32k, specifically noting that the model gpt-4-32k-2023-07-01-preview is not handled in lib/summarization/entry_point.rb. Logs reveal a 400 error: ‘Unrecognized request argument supplied: tools’ peyrusse. Falco suggests the issue stems from needing an updated endpoint with tool support, confirming Azure can handle it internally. Zak argues it’s not Azure-specific, pointing to the plugin’s abstraction layer and tracing the error to lib/completions/endpoints/open_ai.rb, where payload[:tools] = dialect.tools is invoked. Falco recommends using api-version=2023-12-01-preview, which peyrusse agrees to test. Ultimately, peyrusse confirms the issue is resolved after updating the Discourse AI plugin. The thread also includes unrelated queries from peyrusse about forking, testing, and activating Discourse plugins, though no responses address these.

1

Hello,

I’m encountering an issue with the OpenAI Azure endpoint. It seems not to be working as expected for the following URL:

https://***.openai.azure.com/openai/deployments/DEPLOYMENT_NAME/chat/completions?api-version=2023-07-01-preview

Where DEPLOYMENT_NAME is set to gpt-4-32k.

Could you please assist in resolving this issue? Any guidance or suggestions would be greatly appreciated.

Additionally, I have some queries regarding Discourse plugins:

  1. How to Fork a Discourse Plugin: Could you provide detailed instructions or a guide on how to fork an existing Discourse plugin?
  2. Testing the Forked Plugin: Once I have forked a plugin, what are the steps to test this forked version on my own Discourse instance?
  3. Uploading and Activating the Plugin: After testing, how can I upload and activate this forked plugin on my Discourse instance?

Your assistance with these queries would be extremely helpful. Thank you for your time and support.

Best regards,

2

Did you set that in the setting ai openai gpt4 32k url ?

3

Yes i did, but the problem seems to be that in the file lib/summarization/entry_point.rb the gpt-4-32k-2023-07-01-preview is not handled at all.

module DiscourseAi
  module Summarization
    class EntryPoint
      def inject_into(plugin)
        foldable_models = [
          Models::OpenAi.new("gpt-4", max_tokens: 8192),
          Models::OpenAi.new("gpt-4-32k", max_tokens: 32_768),
          Models::OpenAi.new("gpt-4-1106-preview", max_tokens: 100_000),
          Models::OpenAi.new("gpt-3.5-turbo", max_tokens: 4096),
          Models::OpenAi.new("gpt-3.5-turbo-16k", max_tokens: 16_384),
          Models::Anthropic.new("claude-2", max_tokens: 200_000),
          Models::Anthropic.new("claude-instant-1", max_tokens: 100_000),
          Models::Llama2.new("Llama2-chat-hf", max_tokens: SiteSetting.ai_hugging_face_token_limit),
          Models::Llama2FineTunedOrcaStyle.new(
            "StableBeluga2",
            max_tokens: SiteSetting.ai_hugging_face_token_limit,
          ),
          Models::Gemini.new("gemini-pro", max_tokens: 32_768),
          Models::Mixtral.new("mistralai/Mixtral-8x7B-Instruct-v0.1", max_tokens: 32_000),
        ]

        foldable_models.each do |model|
          plugin.register_summarization_strategy(Strategies::FoldContent.new(model))
        end

        truncable_models = [
          Models::Discourse.new("long-t5-tglobal-base-16384-book-summary", max_tokens: 16_384),
          Models::Discourse.new("bart-large-cnn-samsum", max_tokens: 1024),
          Models::Discourse.new("flan-t5-base-samsum", max_tokens: 512),
        ]

        truncable_models.each do |model|
          plugin.register_summarization_strategy(Strategies::TruncateContent.new(model))
        end
      end
    end
  end
end
4

We can see this error in the logs:

DiscourseAi::Completions::Endpoints::OpenAi: status: 400 - body: {
  "error": {
    "message": "Unrecognized request argument supplied: tools",
    "type": "invalid_request_error",
    "param": null,
    "code": null
  }
}


 
Préparation du payload avec prompt: [{:role=>"system", :content=>"You are a helpful Discourse assistant.\nYou _understand_ and **generate** Discourse Markdown.\nYou live in a Discourse Forum Message.\n\nYou live in the forum with the URL: https://<URL>\nThe title of your site: Discourse\nThe description is: \nThe participants in this conversation are: gpt4_bot, Chris\nThe date now is: 2024-01-19 10:10:05 UTC, much has changed since you were trained.\n\nYou were trained on OLD data, lean on search to get up to date information about this forum\nWhen searching try to SIMPLIFY search terms\nDiscourse search joins all terms with AND. Reduce and simplify terms to find more results."}, {:role=>"user", :content=>"comment faire une boucle en dart ?", :name=>"Chris"}], model_params: {}, dialect: #<DiscourseAi::Completions::Dialects::ChatGpt:0x00007f230513e6e0>
5

Oh I see, that means you need an updated endpoint with tool support. Azure can handle it, as we are using it internally.

6

No, I don’t think it’s a specificity of Microsoft Azure, I’ve tested the requests in pure JavaScript and it works without any problems, without any notion of “tools”. Here, I have the impression that it’s a problem with the plugin (I specify that I’m not a Ruby developer) but it seems that the abstraction layer used makes things more complicated. In doubt, we forked that plugin and put debuggers everywhere, the URL, the headers that go to Azure are 100% compliant.

DiscourseAi::Completions::Endpoints::OpenAi: status: 400 - body: { “error”: { “message”: “Unrecognized request argument supplied: tools”, “type”: “invalid_request_error”, “param”: null, “code”: null } }

It seems that the only place where we call a “tools” in ./lib/completions/endpoints/open_ai.rb (DiscourseAi::Completions::Endpoints::OpenAi) is:

    def prepare_payload(prompt, model_params, dialect)
      Rails.logger.warn("Preparing payload with prompt: #{prompt}, model_params: #{model_params}, dialect: #{dialect}")
      default_options
        .merge(model_params)
        .merge(messages: prompt)
        .tap do |payload|
          payload[:stream] = true if @streaming_mode
          payload[:tools] = dialect.tools if dialect.tools.present?
        end
    end

which seems to lead us to ./lib/completions/dialects/chat_gpt.rb (payload[:tools] = dialect.tools if dialect.tools.present?)

In the message from PEYRUSSE Christian, we realize that the “dialect” is “dialect: #DiscourseAi::Completions::Dialects::ChatGpt:0x00007f230513e6e0”

If that can help…

Thank you.

7

It is not, as long as you are using an up to date endpoint.

We are using Azure endpoints on this site, but with the parameter being api-version=2023-12-01-preview. Can you try using an endpoint with that API version?

9

We check this out and let you know surely next week, regards.

10

Hello,

This morning i have updated the plugins Discourse AI and it works now … that’s great thanks for your help.