Why doesn't Discourse support IndexNow?

Feature
12

After reviewing the IndexNow capability, I agree this should be one of the core features/plugins. I also understand that developer resources are limited.

Here are my thoughts on the required plugin to assist the core team. Please feel free to add additional comments.

Assumptions:

  1. IndexNow Plugin will use bulk notifications on a scheduled time model - See Design Consideration #1
  2. Bulk notifications will be set on a time interval
  3. Notifications will only use public topics
  4. Notifications will only be for new/changed/deleted topics when the plugin is enabled.
  5. Plugin will not retroactively notify historical changes/events.

Instructions for Users:

  1. Sign up with the IndexNow search engine of choice.
    • Obtain your API Key
    • Obtain the search engine endpoint URL
  2. Install Plugin
  3. Setup admin

Use Case - Admin Settings

  1. Allow user to turn on/off auto submission capabilities
  2. Allow user to enter the IndexNow search engine endpoint. See Design Consideration #3.
    • Input field is a text parameter
    • Input field must be a valid URL
    • default to Bing URL at https://www.bing.com/indexnow
  3. Allow user to input and store API key
    • Input string field to store API key
    • Input field is alphanumeric
    • Default value will be “”
  4. Allow user to define scheduled time parameters for bulk notifications
    • Time parameter will be set by interval hours
    • Input string to store hour value
    • Valid inputs will be integers
    • Valid inputs can vary from 1 to 24
    • Default value will be 12

Use Case - Text Key File

  1. The system will generate a file called indexnowkey.txt
  2. The key file must be stored at the root level.
  3. The system will populate the file with the API key
  4. The file will be accessible via any remote users/system via http/https

Use Case - Scheduling of Bulk Notification process

  1. The system will schedule the jobs to process on an interval basis based on the setting defined in the admin settings.
  2. The interval value defines the delay between jobs in hours. For example, an input value of 2 would indicate the job should run every 2 hours. A value of 4 indicates the job should run every 4 hours. A value of 24 would indicate that the job should run once a day.

Use Case - Bulk Notification process

  1. The system will determine if the notification process is activated via the site setting defined in the admin settings.
  2. The system will determine if an API key is valid in site settings - not “”.
  3. The system will create a list of topics based on the defined time interval setting. See Design Consideration #2 on query time frames. The topic parameters for inclusion are:
    • Topics must be Public View only
    • New Topics
    • Topics with new posts
    • Topics with posted edited
    • Deleted topics
    • Topic list must be distinct - no duplicates
  4. The system will create the JSON packet using the following format.
{
  "host": "current_site",
  "key": "api_key",
  "keyLocation": "https://current_site/indexnowkey.txt",
  "urlList": [
      "https://www.example.com/url1",
      "https://www.example.com/folder/url2",
      "https://www.example.com/url3"
      ]
}
  1. The JSON packet will be sent to the following
    • URL: sitesettings.search_engine_indexnow_endpoint
  2. The JSON packet will be sent with the following headers
    • Content-Type: application/json; charset=utf-8
    • Http/1.1
    • Host: bing
  3. Validate submission receipt of HTTP request
    • http 200 - successful submission - end process
    • Http 429 - Too many submission attempts - Send notification to the administrator to increase interval timing

Design considerations:

  1. Bulk Notifications vs. Single Notifications—A single notification would be acceptable for small domains, but for larger boards, adding a notification for every new/updated post could create many event processes. From a search engine indexing performance perspective, bulk notifications on an hourly basis would be acceptable for 80% of the forums.
  2. Bulk notifications query timing - SideKiq controls interval timings. If SideKiq is in a heavy process status, the bulk notification process could be delayed. The bulk notification process could miss new/updated topics if the query timeframe equals the scheduling interval. Should a time parameter extend the query to cover delayed processes? Or is it possible to have the Scheduler pass initiated timestamps to control query time intervals? Or do we need to create a database table/value for submitted topics with a time stamp?
  3. Should we build an internal table with each search engine and the defined IndexNow URL endpoint? The user can select the choices from a drop-down menu instead of entering a URL. This removes potential human error.

What is missing? What would you add?