Google Search for Discourse AI
To set up a Google search bot with the Discourse AI plugin, you will need to configure a Google Programmable Search Engine and Custom Search API key. Whichever AI model you use will need to use this search tool to return Google search results. When you set up an AI service for your Discourse Plugin, for example if you are using OpenAI API GPT or Anthropic Claude, and you want to use a helper bot to do Google searches, you will need to configure a Programmable Search Engine to search Google with and an API key for your researcher bot to access the search engine and return results.
Introduction: Google Cloud, Programmable Search Engine and Custom Search API
The Programmable Search Engine (or Custom Search Engine) is a free search engine service that anyone can set up on their website and is identified by a search engine ID (or CX number). A Custom Search API key is used to make API requests to the search engine, and API keys are managed from the Google Cloud console. Thus, you will need to set up a Google Cloud account - a credit card is required to register your account, but the Custom Search API provides 100 search queries per day for free before you pay (additional requests cost $5 per 1,000 queries, up to 10,000 queries per day, but you will have to buy more usage). See how to manage billing.
You will also want to set up your key under a Google Cloud project, preferably a new one designated for the search key (for easier tracking). Note that Google Cloud has quotas on both the number of projects and the number of projects attached to a billing account, and projects can be added to billing accounts after they are set up and API keys are created.
Checklist
Set up Google Cloud account
Create a project for your custom search API key
Connect project to billing account
Create API key
Configure API key for Custom Search
Set up a Google Programmable Search Engine
Save custom search API key and custom search ID in Discourse AI settings
Step 1: Set up a Project in Google Cloud
Before you set up your custom search engine and API key, set up a project in Google Cloud.
Go to Google Cloud admin for APIs for you (or your organization), and make sure you are in the parent organization (you can see it in the field at the top next to the Google Cloud logo).
You will have to create a project for your search engine if you haven’t got one already that you want to use - the link to select or create a new project is near the top of the page.
Complete the create new project page (make sure your organization is selected in the appropriate dropdown fields).
For managing billing for your project, you will need to set up a billing account for the project in Google Cloud billing, assuming you have billing configured for your organization. You (or your billing account manager) should be able to do this later after the key is created. Note that there are quotas on projects and billing projects depending on your organization’s Google Cloud account.
To enable billing on your project, go to the Billing tab in the Google Cloud navigation menu and link a billing account, or enable billing for the project from the billing projects page.
Step 2: Create an API Key
To create an API key from Google Cloud Console, go to the API & Services tab in your Google Cloud console and the Credential settings for your new project. (Don’t worry about the “configure consent screen” message). Click the “+ create credentials” link near the top of the page and select “API key” from the drop-down options.
A new API key will be created for you. Copy and secure it in a safe place like a password vault application. Do not share keys.
This is the Google Custom Search API key you will need to enter in your Discourse AI settings for ai google_custom_search_api_key
You should now see your new key listed in the API keys list. Click the 3 dots in the Actions column on the far right and select “Edit API key”.
This will bring up the API key credentials configuration page, and here you can edit the name of the API key to something more appropriate. (You can also view/copy your API key or regenerate it if necessary). Then go to the “API restrictions” section at the bottom and select “Restrict key” then choose “Custom Search API”. Click OK and then Save your API key.
If you do not see any restrictions listed, you may have to enable the API service in the API Library. Search for “google custom search api” then select it. Here you will see the Custom Search API details, including pricing and links to more documentation - click the link to enable the API.
The new API key will now be listed with its new name and a green checkmark to indicate that it is restricted to Custom Search API JSON requests.
Step 3: Create a Programmable Search Engine
To create a search engine for your API key to use, go to the Google Programmable Search Engine page and click “Get started” if you aren’t already logged in (log in or create your account). Once logged in, go to the search engine console page. All your existing programmable search engines will be listed here.
Click the “Add” button (or the “Create your first search engine!” link) to open the new search engine page. Give it a suitable name and fill out the fields as applicable.
Click the “Create” button and your new search engine will be created. The page will give you an option to copy the search script if you want to use the search box in a web interface somewhere, but that isn’t necessary here, since we are only interested in using it for our API key credentials to access.
Select the customize button to go to the search engine overview page. Here we can now see the Search Engine ID (CX number) needed for our Discourse AI Google search settings, as well as some other options it’s possible to configure. (Note that if you scroll down to the bottom, there is a link to create API keys - you do not need to do this since you did so in step 2 above.)
Step 4: Configure Google Custom Search in Discourse AI Plugin
Copy the Search Engine ID number and store it with the API key you made in step 2. Go to Admin - Site-settings and in the All Site Settings filter, type “google custom search” to bring up the two settings you need. Paste the API key in ai_google_custom_search_api_key and the Search Engine ID (CX number) into ai_google_custom_search_cx of your Discourse instance and save the new settings. Configure your other AI settings as needed.
Step 5: Tracking API Search Keys
You can see your Custom Search API key’s activity in the Google Cloud Console API and Services Details and also set up quotas in the Quota and Systems Limit tab. Cost reports can be viewed in the Cost tab.
Examples of Discourse AI Searches
Here is an example of OpenAI GPT-4 Turbo using Google Custom Search to explain what it does and how to set itself up.
Here is an example of Anthropic Claude Opus using Google Custom Search to explain the meaning of “reverse centaur”.
Relevant Links
- Official Discourse AI Plugin
- Google Developer Custom Search Engine Documentation
- Google Custom Search API
- Google Cloud Billing
- Google Programmable Search Engine
