I enabled “AI embeddings enabled” last night and the backfill was happening in the background. Today I enabled “AI embeddings semantic search enabled” and the CPU shot upto 100 constantly with a huge LOW queue sidekiq backlog. Search stopped working completely on the server.
I disabled the semantic search and the server went back down to normal CPU within minutes with all the queue cleared.
So is there a way to do this differently or is this expected?
I’d love to know more about your “hardware” profile if you selfhost; I’ve thought of doing the same on my servers, but am not sure of the impact. This anecdotally tells me I should hold off or get a better plan together.
That time Jobs::GenerateEmbeddings is expected to take seconds a hit, cause you are making an API call to an embedding service to get the data. That graph is normal, it just shows something it taking a while, another GPU in the cloud is actually doing the work.
CPU going high is very unexpected, maybe your PG setup started extremely resource strained and inserting 1000 or so numbers in a row is enough to hurt it.
We need a lot more data here to help diagnose, what are the specs on the server? What are actual CPU graphs showing, which process is consuming CPU, what embedding service are you using, etc…
I see you only shared a sidekick job duration graph, and it shows that after enabling embeddings Discourse was correctly backfilling those.
Can you share more information about your setup? What is the embeddings provider? Is it a third party API or are you self hosting it too? How many topics your have? What is the DB specs?
