Is there is any open API for getting similar topic in discourse?

Use the web inspector as detailed in that topic to watch what happens as you write in the composer and trigger the suggested similar topics dialog.

I believe that it is materially different from “normal” site search, as it marks the search as “give me anything that matches any of these, and use frequency / multiple matches etc to order the results”.