I’m getting the “Mondays” conversation in my summaries:
I’ve seen that in my logs- it seems to be intended to help the model interpret the format? But it shouldn’t end up visible to users.
This behavior started after changing two things:
AiSummary.destroy_all. Maybe this isn’t sufficient?Is every summary like that after changing to Claude Haiku 3.5?
No, about half- about 5 out of 10 summaries generated. I’ve tried switching back to sonnet to see if that fixes it, but am running into rate limits so this may take a while to characterize.
Can you check the entire API request and response from a topic where this happened and share with me? It will be on the ai_api_audit_logs table.
Not at the moment unfortunately. But I did look at that earlier and found the response for that topic was a valid summary with no mention of Mondays. Is there any way it could get confused post-response? If nothing comes to mind I’ll give you a more definitive example in the next couple days.
Our logging into ai_api_audit_logs is guaranteed, any successful responses are logged there and you should see the entire prompt. Is there any way you can run data explorer and look at the contents of the table?
I get the Mondays when I hit a rate limit at Anthropic, at least with haiku. I can try to fix the rate limit but it would be nice to fail and not display Mondays.
/logs:
Job exception: {"type":"error","error":{"type":"rate_limit_error","message":"Number of request tokens has exceeded your per-minute rate limit (https://docs.anthropic.com/en/api/rate-limits); see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase."}}
Request from ai_api_audit_logs. I’ve redacted the real user content:
{
"model": "claude-3-haiku-20240307",
"max_tokens": 3000,
"messages": [
{
"role": "user",
"content": "Here are the posts inside <input></input> XML tags:\n\n<input>1) user1 said: I love Mondays 2) user2 said: I hate Mondays</input>\n\nGenerate a concise, coherent summary of the text above maintaining the original language."
},
{
"role": "assistant",
"content": "Two users are sharing their feelings toward Mondays. [user1](/t/-/11772/1) hates them, while [user2](/t/-/11772/2) loves them."
},
{
"role": "user",
"content": "he discussion title is: [redacted].\n\nHere are the posts, inside <input>[redacted]</input> XML tags:\n\n<input></input>\n\nGenerate a concise, coherent summary of the text above maintaining the original language."
}
],
"system": "You are an advanced summarization bot that generates concise, coherent summaries of provided text.\n\n- Only include the summary, without any additional commentary.\n- You understand and generate Discourse forum Markdown; including links, _italics_, **bold**.\n- Maintain the original language of the text being summarized.\n- Aim for summaries to be 400 words or less.\n- Each post is formatted as \"<POST_NUMBER>) <USERNAME> <MESSAGE>\"\n- Cite specific noteworthy posts using the format [NAME](/t/-/11772/POST_NUMBER)\n - Example: link to the 3rd post by sam: [sam](/t/-/11772/3)\n - Example: link to the 6th post by jane: [agreed with](/t/-/11772/6)\n - Example: link to the 13th post by joe: [#13](/t/-/11772/13)\n- When formatting usernames either use @USERNMAE OR [USERNAME](/t/-/11772/POST_NUMBER)"
}
Response from ai_api_audit_logs:
{
"id": "msg_01QPVSmg93tu21WtPUvoNZ33",
"type": "message",
"role": "assistant",
"model": "claude-3-haiku-20240307",
"content": [
{
"type": "text",
"text": "Summary:\n\nTwo users are sharing their feelings toward Mondays. [user1](/t/-/11772/1) loves them, while [user2](/t/-/11772/2) hates them."
}
],
"stop_reason": "end_turn",
"stop_sequence": null,
"usage": {
"input_tokens": 7298,
"output_tokens": 49
}
}
Clicking on the Summarize button displays this:
That would mean that they are returning a 200 response when you hit a rate limit instead of a 429.
This would be a bug on their API. Can you open a bug report there?
They want the network traffic including headers. I don’t see how I can get that from my browser, either by pushing the Summarize button or by triggering the sidekiq job.
We don’t log that, you would need to reproduce issuing a cURL from your server and pass that forward.
You can also supply them with msg_01QPVSmg93tu21WtPUvoNZ33 they can look that up directly in their logs.
Any news from Anthropic given the request id provided by Sam @markschmucker ?
If you are still rate-limited, maybe try halving the backfill batch size.
They don’t have a typical support ticket system- instead, you talk to a bot. I reported the issue to the bot, including the message id that @sam provided. The bot was convinced and promised to pass it along to a human, and email me when there was new status. No emails yet.
Today I received a chat message from Anthropic, simply asking if it’s still happening. It is. No further feedback. Frustrating experience.
Response from Anthropic:
Hi there,
Unfortunately I am not reproducing this error at this time.
Best,
Ryan
I don’t see how I can push this forward any further. I could create my own test with cURL but I suspect that wouldn’t be an accurate replication of what Discourse is doing.
Discourse does the same HTTP calls that cURL will do, so it is possible to reproduce with 100% fidelity.
That said, it’s well know that Anthropic is having a hard time keeping up with the increasing demand, so it may be easier to shop around for either using Claude via AWS or Google Cloud, or even try Gemini or other LLM APIs.