Automatically generated llms.txt

I think the template should be good for 99% of sites, really all the critical points are hit.

But… if for some reason it is not fitting your exact rules, then after enabling the upcoming change, I would copy the text, edit to taste and upload a custom one.

I strongly recommend though being very careful with llms.txt, you want an optional map, not a manifest.

Every token costs, so if you have a 120k context window and then say “check out X on site” and robot ends up with a 60k of tokens it does not need cause you created a giant llms.txt it only leads to confusion and bad results.

So if I was customising … it would probably be minimal.

Robots stay off my site

OR

Never search more than 3 times per session.

Of course llms.txt is not a ratified spec and very few llms will proactively follow it. Generally llms these days just sit on a “search” and “fetch” tool, so they tend to find deep links and don’t care about maps.

1 Like