AI bot stopped responding after last update

Contribute Bug
,

AI-generated summary

After updating to Discourse v3.3.0.beta6 and Discourse AI 9d887ad4, the AI bot stopped responding after starting a conversation, only updating titles without generating replies. Logs revealed a DiscourseAi::Completions::Llm::UNKNOWN_MODEL error. Testing LLMs via the admin panel showed no issues, but the bot failed to make LLM calls. The issue was traced to a specific persona with a user association, which was not working, while another similar persona without a user worked fine. The problem persisted after updating to v3.4.0.beta1. Ultimately, deleting and recreating the problematic persona resolved the issue, confirming that deleting a persona removes related embeddings from the database aas.

1

Discourse version: v3.3.0.beta6
AI version: 9d887ad4

  • It doesn’t matter which model I choose when starting a convo with the bot
  • I can see successful requests in the ai_api_audit_logs table.
    • when a user starts a convo, the only LLM call made is title bot call (“. You are titlebot. Given a topic, you will figure out a title.\nYou will never respond with anything but 7 word topic title.”)
    • the bot updates the title but doesn’t respond
  • Requests to our providers work fine - Discourse AI isn’t even attempting to make a call

Here’s a screenshot of the convo:

CleanShot 2024-08-12 at 09.23.47
CleanShot 2024-08-12 at 09.23.47629×603 32 KB

Notice the empty bot response.

From the logs you can see there is only a bot_title call, and not bot log.

CleanShot 2024-08-12 at 09.31.24
CleanShot 2024-08-12 at 09.31.24865×542 72.9 KB

Here’s the persona’s settings

CleanShot 2024-08-12 at 09.33.54
CleanShot 2024-08-12 at 09.33.54931×2239 250 KB

2

What happens when you go to /admin/plugins/discourse-ai/ai-llms and hit Test on this LLM ?

3

Assuming you mean to click Edit next to an LLM and then scroll down and click Run Test, I get a success message for all enabled LLMs (anthropic and openai).

CleanShot 2024-08-12 at 09.56.58
CleanShot 2024-08-12 at 09.56.58339×686 30 KB

4

Is there anything in /logs ?

5

By golly, there is! Job exception: DiscourseAi::Completions::Llm::UNKNOWN_MODEL

Message (30 copies reported)

Job exception: DiscourseAi::Completions::Llm::UNKNOWN_MODEL

Backtrace

/var/www/discourse/plugins/discourse-ai/lib/completions/llm.rb:129:in `proxy'
/var/www/discourse/plugins/discourse-ai/lib/ai_bot/personas/persona.rb:155:in `craft_prompt'
/var/www/discourse/plugins/discourse-ai/lib/ai_bot/bot.rb:54:in `reply'
/var/www/discourse/plugins/discourse-ai/lib/ai_bot/playground.rb:426:in `reply_to'
/var/www/discourse/plugins/discourse-ai/app/jobs/regular/create_ai_reply.rb:18:in `execute'
/var/www/discourse/app/jobs/base.rb:305:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.0.0/lib/rails_multisite/connection_management.rb:21:in `with_connection'
/var/www/discourse/app/jobs/base.rb:292:in `block in perform'
/var/www/discourse/app/jobs/base.rb:288:in `each'
/var/www/discourse/app/jobs/base.rb:288:in `perform'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:202:in `execute_job'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:170:in `block (2 levels) in process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:177:in `block in invoke'
/var/www/discourse/lib/sidekiq/pausable.rb:132:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:182:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:169:in `block in process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:113:in `local'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq.rb:44:in `block in <module:Sidekiq>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:263:in `stats'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:13:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:80:in `global'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:124:in `block in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:39:in `prepare'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:123:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:168:in `process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:78:in `process_one'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:68:in `run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:8:in `watchdog'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:17:in `block in safe_thread'

env:


hostname	forum-app
process_id	2575726
application_version	a4bfe8adb4aae3e8049ce5e7122ba01f4db1bdb5
current_db	default
current_hostname	community.ankihub.net
job	Jobs::CreateAiReply
problem_db	default
time	5:15 pm
opts	
post_id	334872
bot_user_id	-121
persona_id	1
current_site_id	default

CleanShot 2024-08-12 at 17.19.06@2x
CleanShot 2024-08-12 at 17.19.06@2x1816×1030 109 KB

6

try selecting a different default llm on the persona, saving, reloading and then picking claude 3.5 sonnet again

This may be migration related.

7

I changed the default model for the persona to gpt-4o. I tried starting a PM with the bot and it didn’t work. Switched the default model back to claude 3.5 sonnet and still doesn’t work. I also tried disabling and re-enabling the persona.

I tried a PM with a different persona that uses claude 3.5 sonnet, and it works fine. One difference between the two personas is that the one that isn’t working has a user associated with it. The other persona doesn’t and responds with the auto-created claude-3-5-sonnet user.

I suppose I could delete and recreate the broken persona? However, the broken persona uses uploaded docs. If I delete the broken persona, will the associated embeddings from the uploaded docs be deleted from postgres? I don’t mind regenerating the embeddings as that is quite cheap.

8

I just updated to Discourse 3.4.0.beta1 and Discourse AI 36ce88f3, but it still doesn’t work. I suppose I will delete and recreate the broken persona.

9

For anyone who might stumble upon this, I can confirm that deleting and recreating the persona fixed the issue. Also, deleting a person indeed removes the related items in the rag_document_fragments and ai_document_fragment_embeddings tables.

4 Likes
10

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.