Balancing Costs and Functionality in AI-Powered Forums

[Aizada_M]

Continuing the discussion from Limit # topic posts / user / month:

@Saif

I would be curious to know what other parameters (and why) you would like to see to help control costs or for fine-tuning functionality

Hello Thank you for your interest

For more context: In 2021, my first project with a forum in Kazakhstan was unsuccessful because most residents prefer using Telegram or WhatsApp. The introduction of plugins for the Discourse AI bot and Chatbot
gave my website a second chance, but now it is focused solely on communication with artificial intelligence (categorized query templates, bot characters, etc.).

1. Regarding token expenditure control: It would be desirable to have a separate statistics/settings panel to track and manage token expenditure depending on the bot character (GPT3, 3.5, 4, 4.5t, and/or Composer Assistant) being interacted with. As an administrator, within a month of trial usage, I have already burned tokens for ChatGPT queries amounting to over $70, which is a significant expense for me. Now, as I want to provide access to bots for regular users, I start worrying about my budget, which is challenging to control.

Let’s imagine a situation where I grant access to AI for a specific user group and say, “Use it.” Suppose one user generates enough queries in a day to deplete my budget. Now, suppose another user tries to make a query to AI and… receives no response (nothing). The second user may not understand why the bot didn’t reply, assume the service is malfunctioning, and opt for other services.

For this, it would be appropriate to have a separate panel in the user tab where each user (admin/moderator/regular user) can track and control token expenditure, bot temperature, top (although this can be placed in the post editor, see the diagram below), and other fine-tuning settings.

For example, I would like to set a threshold amount for myself, and when reached, receive a notification to replenish the budget/tokens. Since different AI models may differ in the cost of consumed tokens, I would like the ability to limit tokens for each bot for myself and other user groups. Each user should be able to independently manage the allocated token limit at their discretion, similar to what an administrator can do. It would also be helpful to provide some user groups (moderators, TL4) with the ability to finely tune generation settings (temperature, TOP , etc.).

For example, instead of defining the maximum value of embeddings semantic related topics for all users, it would be practical to provide these limits depending on the user group. Thus, the Staff group might be given a maximum of 7, regular users 3, and so on. Each user should have the ability to set these values in the user panel in their account. This approach would democratize the use of AI and the ability to control the token limits allocated to each user.

For instance, ai helper automatic chat thread title could also be determined based on the user group, giving each user the choice to enable/disable this function in the user panel. ai helper model could also be left to the user’s choice based on the group. If I give Group-A the option to choose between GPT4t and GPT3.5t, each of them could make the choice independently.

It would also be possible to add the ability for privileged groups to have their queries prioritized and sent to LLM at the front of the queue.

I tried to illustrate this in more detail (I quickly made the illustration, please don’t judge it harshly):

Note: In the above image, I tried to reflect possible features proposed for regular users. These features may be locked, and to make it clear to the user, it would be appropriate to have buttons for activating features/increasing limits/adding a bot. These buttons are highlighted in blue, and clicking on any of these buttons would redirect the user to a page with an invitation to join a privileged group for more functionality in interacting with AI.

2) In the editor, I suggest:

• Categorizing bots by types (Working with images, text, audio, etc.) and additional query settings (see point 1 above) within the Composer interface.

• Adding the ability to limit the number of characters for a query based on the bot character (as one of the levers to reduce server load) or user group. I discussed something related here.

• The ability to insert a query template using the existing Discourse Templates plugin or possibly a future modification (for use in personal messages) currently under development: Experimental Form Templates.

• The ability to insert a hint template in the text input area (similar to category theme templates in category settings).

Here is an example illustration:

Note: It would be advisable to reflect the character input limit at the bottom of the editor (as shown) in the image.

Additional options (aside from API settings) in AI Persona Editor for Discourse, which will then be displayed in the message editor:

PS. In these days, I’ve been a bit under the weather (I’m sick), and some of my suggestions might be a bit scattered and not quite clear. I’m a newcomer to Discourse, lack programming knowledge, and find it challenging to grasp information on this English-language forum, where posts often contain specific terms. Thus, I acknowledge that my ideas (proposals) might be somewhat absurd at times, not align with some technical constraints of Discourse. I also understand that the team may have its own project roadmap for the plugin, which may not necessarily align with my views. However, I decided to write this post because I believe the AI revolution will attract many users to such services, and Discourse already has all the technological capabilities to interact with AI ahead of most projects emerging in the market (the fact that Open AI uses Discourse for its forum speaks volumes). So, it’s better to say than not say. In this regard, consider my proposal as an outsider’s perspective, a suggestion from an ordinary user (who is often used to social networks and messengers) wanting clarity and interaction functionality, which social networks and messengers often lack.

Edit. I understand that implementing such functionality may require large labor and financial costs (which not every sponsor can handle). It might be worthwhile to put such proposals to a vote and/or organize crowdfunding.

[sam (Sam Saffron)]

Thank you for your carefully thought out topic!

I hear you, at the heart one thing that is clearly needed in the medium term is a concept of “quotas”.

Pick LLM → set quota for group (how many tokens are they allowed per day, since everything is billed by tokens)

This let’s you open up AI features, safely, to larger groups without worrying about going bust.

I think we will get there, but it is not quite scheduled.

On the upside though we have done a lot of work recently to normalize our LLM interfaces so this is going to be much more practical and straightforward to build than it was in the past.

[Aizada_M]

Thank you for your answer . I’m very glad that you heard me

@sam

On the upside though we have done a lot of work recently to normalize our LLM interfaces so this is going to be much more practical and straightforward to build than it was in the past.

It’s really true. Your team is doing a great job. What seemed like something distant just a couple of months ago is already working today. It can’t help but be inspiring.