Discourse AI Persona, upload support

[tpetrov (Toni Petrov)]

@sam

Additionally, Discourse gives you extra control over how your text will be split up.

The [[metadata YOUR METADATA HERE]] separator can be used to split up large bodies of text and properly highlight what each section covers.

For example:

[[metadata about cats]]
a long story about cats
[[metadata about dogs]]
a long story about dogs

This allows a single text document to cover a large variety of content and protects you from “chunk contamination”. You are guaranteed that only data about cats will be included in cats chunks and dogs in dog chunks.

Hey Sam, I wonder how this works exactly. It will tell the AI that this is data about cats or dogs, but how will it affect chunks if they are already set to a set number of tokens (say 2000). Will it cut off a chunk when it sees a line like [[metadata about dogs]] and start a new chunk?