Discourse Chatbot 🤖 (supporting ChatGPT)

Can you test the others too please and conclude if you can repro it yes or no? I’m unable to repro any of this in a repeatable way but I have seen ... sometimes.

Btw, you can edit your post to add additional info instead of double replying.

1 Like

It appears to work for public messages.
Edit: It’s now working for me in private messages. :person_shrugging:

1 Like

Did you change any setting?

May sound as good news but we still don’t know why and when this is happening. Please get back whenever this happens again.

1 Like

OK, I had it ignore a private message now. I asked again (this time invoking @AIbot), and got this message: “Sorry, I’m not well right now. Let’s talk some other time. Meanwhile, please ask the admin to check the logs, thank you!”

OpenAIBot: There was a problem: Net::ReadTimeout

/var/www/discourse/plugins/discourse-chatbot/app/jobs/regular/chatbot_reply_job.rb:60:in `rescue in execute'  
/var/www/discourse/plugins/discourse-chatbot/app/jobs/regular/chatbot_reply_job.rb:55:in `execute'  
/var/www/discourse/app/jobs/base.rb:249:in `block (2 levels) in perform'  
rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:80:in `with_connection'
/var/www/discourse/app/jobs/base.rb:236:in `block in perform'  
/var/www/discourse/app/jobs/base.rb:232:in `each'  
/var/www/discourse/app/jobs/base.rb:232:in `perform'  
sidekiq-6.5.8/lib/sidekiq/processor.rb:202:in `execute_job'  
sidekiq-6.5.8/lib/sidekiq/processor.rb:170:in `block (2 levels) in process'  
sidekiq-6.5.8/lib/sidekiq/middleware/chain.rb:177:in `block in invoke'  
/var/www/discourse/lib/sidekiq/pausable.rb:134:in `call'  
sidekiq-6.5.8/lib/sidekiq/middleware/chain.rb:179:in `block in invoke'  
sidekiq-6.5.8/lib/sidekiq/middleware/chain.rb:182:in `invoke'  
sidekiq-6.5.8/lib/sidekiq/processor.rb:169:in `block in process'  
sidekiq-6.5.8/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'  
sidekiq-6.5.8/lib/sidekiq/job_retry.rb:113:in `local'  
sidekiq-6.5.8/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'  
sidekiq-6.5.8/lib/sidekiq.rb:44:in `block in <module:Sidekiq>'  
sidekiq-6.5.8/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'  
sidekiq-6.5.8/lib/sidekiq/processor.rb:263:in `stats'  
sidekiq-6.5.8/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'  
sidekiq-6.5.8/lib/sidekiq/job_logger.rb:13:in `call'  
sidekiq-6.5.8/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'  
sidekiq-6.5.8/lib/sidekiq/job_retry.rb:80:in `global'  
sidekiq-6.5.8/lib/sidekiq/processor.rb:124:in `block in dispatch'  
sidekiq-6.5.8/lib/sidekiq/job_logger.rb:39:in `prepare'  
sidekiq-6.5.8/lib/sidekiq/processor.rb:123:in `dispatch'  
sidekiq-6.5.8/lib/sidekiq/processor.rb:168:in `process'  
sidekiq-6.5.8/lib/sidekiq/processor.rb:78:in `process_one'  
sidekiq-6.5.8/lib/sidekiq/processor.rb:68:in `run'  
sidekiq-6.5.8/lib/sidekiq/component.rb:8:in `watchdog'  
sidekiq-6.5.8/lib/sidekiq/component.rb:17:in `block in safe_thread'  
2 Likes

I got the same error in a PM now. It happened at after the 15th message where lookback value is set to 5 posts.

Nothing strange is in this PM, just plain text-useless chats with the AI.

EDIT: Can repro in another PM. This time happened after post #13.

EDIT2: The same is happening in topics, same error after post #13.

EDIT3: Can repro again, in PM on another instance, after post #9.

2 Likes

I’ve done about a dozen chats with it now. Sometimes it dies after the second message - other times it makes it to 6 or 7 messages.

2 Likes

Yes. This is definitely an issue, it can happen anytime. Im real ruby noob so I can’t really see where it’s going wrong.

From the error I am assuming something in the prompt to OpenAI is wrong, because the bot is failing to get the response and then returns an error.

This is different then the ... error, which does not return an error if I recall correctly.

1 Like

The chat gpt API is timing out. That’s a shame. Open AI need to improve response times.

There was no timeout extension feature I could configure before. I’ll take another look soon.

Edit: just took a quick look, we need this to be merged:

Or this one! :thinking:

But that won’t solve the ultimate problem which is that Open AI need to improve their response times.

It does look like it is getting worse :frowning:

Guess it’s popular!! :sweat_smile:

3 Likes

For me the … happens in PMs, topics, and chats. I’ve never seen a different response.

Edit: in the rails log I can see all the numbered steps logged too.

3 Likes

Can you share some rails output? This might be helpful for @merefield

1 Like

The Open AI response retrieve Job currently doesn’t retry on failure (usually because of timeout).

I’m about to remedy that, but it just needs a little more testing:

(retries currently set to 0 on this PR so we can test the exhausted scenario, will update to 5 before merge).

1 Like

That is merged, give that a whirl.

It will now retry 5 times before giving up and responding with a sorrowful message :sweat_smile:

@DiscourseMetrics @gjsman

(@DiscourseMetrics I’m not sure this will fix the attack of the 3 dots but we shall see if things improve for you).

4 Likes

The three dots …

I’d missed this but this (old code I moved over from FrotzBot) happens when the response is blank. I’m not yet sure if this is due to a bug with Open AI or a local issue. I haven’t had time to look into it. It seems extremely rare for me.

(Thanks to @MarcP for pointing out the code I’d forgotten about!)

3 Likes

It did not seem to help the “…” issue, unfortunately :slight_smile:

Here’s what I see in unicorn.stdout.log after making a new post:

1. trigger
2. evaluation
humans: 1
3. invocation
4. Retrieving new reply message...
5. Creating a new Post...
1. trigger
6. The message has been created successfully
2 Likes

Well I have an update for you:

There was an issue with the error handling. There’s some problem with your config, but you are not able to see the error.

Previously, if the ChatGPT response errored (e.g. missing or invalid token), the code would return a blank message which was then converted elsewhere in the code to ‘…’ (unhelpfully!).

This was a bug and didn’t affect the other model choices, e.g. Text Davinci 003.

But I’ve fixed all that with this PR I just merged:

You should now be able to work out what the issue is.

Unfortunately this will mean updating again, sorry! I would have shared this sooner but wanted to test it out in Production myself which I have just done.

Make sure you are seeing at least version 0.16 in the plugin list.

These last two PRs should massively improve the robustness & error feedback of the solution.

@DiscourseMetrics @gjsman

1 Like

Brilliant! I think I found the solution independently :slight_smile: I tried the playground on the openAI site, and it said I was over my usage limit. I had not actually had any usage, so that was weird, but the second I added a credit card the bot started working :slight_smile:

1 Like

Haha, great! Well sorry my code was not able to surface that issue until now.

They must be limiting free usage now to avoid becoming over-burdened. There are more than enough people willing to pay!

Please update at your convenience in any case, just in case there are any other Open AI bot issues you need to receive feedback on.

4 Likes

Will do! Thanks for the help :slight_smile:

1 Like

I recommend a “puts” helper here @merefield

def debug_message(message)
   if SiteSetting.chatbot_enable_verbose_console_logging
      puts message
   end
end

IMO this should be a hidden site setting as well, no need to trigger it from the UI.

Technically we usually use Rails.logger.debug for this kind of stuff and rely on logster, but if you really need this in the console, then it should be hidden by default.

6 Likes

Nice, I’ll use that! Thanks!

2 Likes