Error in any AI Tool with no parameters, e.g. "tags"

[markschmucker (Mark S)]

EDIT- Now I understand this happens with the built-in tags AI Tool also, not just my custom tool. Please skip ahead to my second post.

I have two custom AI Tools- one with parameters and one with no parameters. Both pass with the Run Test button. But when a bot tries to use the tools, the first one works, and the second fails with:

I see the troubleshooting tips at AI bot - Custom tools.

@Discourse

• Use the Test interface to ensure it behaves as expected for your inputs.
• Ensure your group is in ai_bot_debugging_allowed_groups. Members of this group have full access to the bot transcripts; you can view the AI logs there.
• If anything unexpected is happening, visit https://SITENAME/logs to check for errors.

• The Test interface passes with Success.
• My group is in ai_bot_debugging_allowed_groups, but I don’t know where the bot transcripts are.
• /logs just has the same error shown above.

I tried adding a dummy parameter, same result.

Up-to-date with 3.5.0.beta1-dev. Using Gemini 2.0 Flash.

function invoke(params) {
    const url = `https://example.com/investments/sponsors`;
    let result = http.get(url);
    return result;
}

function details() {
    return "Returns sponsors.";
}

[markschmucker (Mark S)]

After a lot of experiments, I get the same errors with the built-in tags tool, so there’s nothing wrong with my custom tool. All tools that take parameters work as expected. The common thing between tags and my custom tool is that neither takes any parameters.

I believe the bot has to be pretty bright to call a function with no parameters. gpt-4o is able to use those tools with no problem. But flash 1.5 and 2.0 and haiku always fail. Sonnet usually fails but got it right once.

The errors are always one of the following:

or:

An empty string is not a valid JSON string.

I’m under the impression the bots aren’t sure what to pass to a function that takes no parameters- null? An empty string? An empty object? Empty json {}?

[sam (Sam Saffron)]

will write an eval here, thank for raising!

[sam (Sam Saffron)]

I have an eval for no param tools:

tool_calls/no_params_native.yml

And it passes with:

sam@arch evals % ./run -e no_params_tool
Running evaluation 'no_params_tool'
GPT-4o: Passed 🟢
GPT-4o-mini: Passed 🟢
Claude 3.5 Haiku: Passed 🟢
Claude 3.5 Sonnet: Passed 🟢
Gemini 2.0 Flash: Passed 🟢
Gemini 2.0 pro: Passed 🟢

Did you test on latest?

[markschmucker (Mark S)]

I am on 3.5.0.beta1-dev. Adding a dummy param to my custom tool “fixes” it, so I still think there’s an issue with no params.

@Discourse

Ensure your group is in ai_bot_debugging_allowed_groups. Members of this group have full access to the bot transcripts; you can view the AI logs there.

Where are these logs?

[sam (Sam Saffron)]

If you enable there is an (i) in the post actions at the bottom of a post