Why was external AI chosen over an internal system?

Note: I am not a Discourse employee and not a OpenAI employee but a category moderator on the OpenAI forum (creators of ChatGPT and GPT 4) and have been using many of the Discourse AI features even during development.

The means you note are syntactic searches, this is semantic search and uses embeddings instead of keywords.

A Google for how does semantic search work reveals many articles, here is one I think many here may like

Yes it is part of the Discourse AI plugin specifically Semantic Related Topics.