Max consecutive tool usage in the answer

Don · January 27, 2025, 11:10am

Hello

I created two custom tools for collecting information from webshops. Such information is the name of the web store, the product link, the price and the stock information. After these informations it lists the results in a table.

For this, I created a custom Google search tool that is similar to the built-in Google search tool, but I can limit the number of results and I was able to create a separate programmable search that only use by this tool. This was necessary so that the results only contain relevant webshops.
The next custom tool is similar to the Discourse web browsing tool, but I optimized it specifically for web stores. So, first of all, it searches for the JSON-based schema.org format “Offer or Product” located in <script type="application/ld+json"> and runs within it and searches for the necessary data (price, stock information). If this does not yield results, it starts searching for the same based on the itemtype. If this does not yield results either, then what the web browser tool does by default comes up, which is to start reviewing the main contents of the page and search for the necessary information. Another important thing that has been built into this is the token limit. Where I can set the maximum token limit while browsing. It is currently 3000.

But the reason I am writing this topic is that is there any kind of limit in tool usage? Because I noticed that after 5 tool usages the process is interrupted. This is the other reason why I built the results limitation in the custom search. I set it to 4 because then browser tool can run success. If I set it to 5 the browser tool will fail on the 5th tool usage.

For example: in an answer after 1 google search and 5 web browsers in a row I get a warning in /logs.

Message (3 copies reported)

DiscourseAi: Tool not found: tool-name

Backtrace

/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/broadcast_logger.rb:130:in `block in warn'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/broadcast_logger.rb:231:in `block in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/broadcast_logger.rb:231:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/broadcast_logger.rb:231:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/broadcast_logger.rb:130:in `warn'
/var/www/discourse/plugins/discourse-ai/lib/ai_bot/bot.rb:158:in `block in reply'
/var/www/discourse/plugins/discourse-ai/lib/completions/endpoints/base.rb:190:in `block (5 levels) in perform_completion!'
/var/www/discourse/plugins/discourse-ai/lib/completions/endpoints/base.rb:190:in `each'
/var/www/discourse/plugins/discourse-ai/lib/completions/endpoints/base.rb:190:in `block (4 levels) in perform_completion!'
/var/www/discourse/plugins/discourse-ai/lib/completions/endpoints/base.rb:178:in `each'
/var/www/discourse/plugins/discourse-ai/lib/completions/endpoints/base.rb:178:in `block (3 levels) in perform_completion!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-protocol-0.2.2/lib/net/protocol.rb:535:in `call_block'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-protocol-0.2.2/lib/net/protocol.rb:526:in `<<'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-protocol-0.2.2/lib/net/protocol.rb:162:in `read'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-http-0.6.0/lib/net/http/response.rb:631:in `read_chunked'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-http-0.6.0/lib/net/http/response.rb:595:in `block in read_body_0'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-http-0.6.0/lib/net/http/response.rb:588:in `inflater'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-http-0.6.0/lib/net/http/response.rb:593:in `read_body_0'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-http-0.6.0/lib/net/http/response.rb:363:in `read_body'
/var/www/discourse/plugins/discourse-ai/lib/completions/endpoints/base.rb:173:in `block (2 levels) in perform_completion!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-http-0.6.0/lib/net/http.rb:2433:in `block in transport_request'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-http-0.6.0/lib/net/http/response.rb:320:in `reading_body'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-http-0.6.0/lib/net/http.rb:2430:in `transport_request'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-http-0.6.0/lib/net/http.rb:2384:in `request'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-3.3.1/lib/patches/net_patches.rb:19:in `block in request_with_mini_profiler'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-3.3.1/lib/mini_profiler/profiling_methods.rb:50:in `step'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-3.3.1/lib/patches/net_patches.rb:18:in `request_with_mini_profiler'
/var/www/discourse/plugins/discourse-ai/lib/completions/endpoints/base.rb:116:in `block in perform_completion!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-http-0.6.0/lib/net/http.rb:1632:in `start'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-http-0.6.0/lib/net/http.rb:1070:in `start'
/var/www/discourse/plugins/discourse-ai/lib/completions/endpoints/base.rb:99:in `perform_completion!'
/var/www/discourse/plugins/discourse-ai/lib/completions/endpoints/open_ai.rb:40:in `perform_completion!'
/var/www/discourse/plugins/discourse-ai/lib/completions/llm.rb:279:in `generate'
/var/www/discourse/plugins/discourse-ai/lib/ai_bot/bot.rb:115:in `reply'
/var/www/discourse/plugins/discourse-ai/lib/ai_bot/playground.rb:477:in `reply_to'
/var/www/discourse/plugins/discourse-ai/app/jobs/regular/create_ai_reply.rb:18:in `execute'
/var/www/discourse/app/jobs/base.rb:316:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management.rb:21:in `with_connection'
/var/www/discourse/app/jobs/base.rb:303:in `block in perform'
/var/www/discourse/app/jobs/base.rb:299:in `each'
/var/www/discourse/app/jobs/base.rb:299:in `perform'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:202:in `execute_job'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:170:in `block (2 levels) in process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:177:in `block in invoke'
/var/www/discourse/lib/sidekiq/pausable.rb:132:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:182:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:169:in `block in process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:113:in `local'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq.rb:44:in `block in <module:Sidekiq>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:263:in `stats'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:13:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:80:in `global'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:124:in `block in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:39:in `prepare'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:123:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:168:in `process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:78:in `process_one'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:68:in `run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:8:in `watchdog'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:17:in `block in safe_thread'

I don’t know exactly what cause this, maybe it’s a limitation. If yes, is there any chance to add a setting to the persona or to the custom tool to set up the tool consecutive usage limit in an answer.

Edit: I found this which is probably the limit. It would be great to have the ability to override with a setting I mentioned above.

github.com/discourse/discourse-ai

lib/ai_bot/bot.rb

fe44d7815


      
          # frozen_string_literal: true
          
          module DiscourseAi
            module AiBot
              class Bot
                attr_reader :model
          
                BOT_NOT_FOUND = Class.new(StandardError)
                MAX_COMPLETIONS = 5
                MAX_TOOLS = 5
          
                def self.as(bot_user, persona: DiscourseAi::AiBot::Personas::General.new, model: nil)
                  new(bot_user, persona, model)
                end
          
                def initialize(bot_user, persona, model = nil)
                  @bot_user = bot_user
                  @persona = persona
                  @model = model || self.class.guess_model(bot_user) || @persona.class.default_llm
                end

Thank you

denvergeeks · January 27, 2025, 5:33pm

What is the Discourse Web Browsing Tool, and where can I find/get it?!

Jagster · January 27, 2025, 5:39pm

I guess it refers to tools that an AI can use. So, these can enable when editing AI-personas.

MachineScholar · January 27, 2025, 8:37pm

I haven’t encountered this issue yet, but if it is indeed a hardcoded limitation then it’d indeed be cool to be able to override it!

In the personas provided by the plugin by default there is the “Researcher” one. You can check it out to see how it works

Falco · January 27, 2025, 8:52pm

What LLM are you using?

Is your LLM returning over 5 function calls in a single response, as in they are parallel function calls?

Can you share that LLM response?

Don · January 27, 2025, 10:15pm

GPT4o, GPT4o Mini, Gemini Flash 2. Honestly I tried it with a lot LLM.

Yes, This is happening when I set Forced Tools and add a system message to the Persona to check things. For example: “visit all website founded on google search” This is necessary for a use case that I have.

I’ll send you an example in PM because that’s could be an inappropriate content for public. In that example the bot is using the Google Search and Web Browser with Gemini Flash 2.

sam · January 27, 2025, 11:28pm

I think I see the bug here… generally max_tools was there to avoid huge chains with tool calls, but there are probably 2 different types of max:

Max times we round trip with the LLM prior to replying
Maximum amount of tool calling (which can be much higher)
Maximum amount of context tools can inject. (eg: if you blow the context in 3 tool calls and then pile on another 2 that cause the model to forget about 2 tool replies)

Going to have a think about this, it is solvable.

Topic		Replies	Views
Discourse AI - Self-Hosted Guide Self-Hosting ai	60	9859	May 20, 2024
Building a technical support chatbot Extras chat , ai	1	403	October 4, 2024
Discourse DB Error When Creating New Users via SSO Login Bug	6	1411	October 6, 2021
Could a List of Recommended Topics Be Added to the Top Menu? Feature ai	21	1263	June 26, 2024
Discourse AI - AI bot Site Management how-to , ai , ai-bot	23	13513	October 30, 2024

Max consecutive tool usage in the answer

Related topics