Is there is any open API for getting similar topic in discourse?

You will see that if you request the JSON for a topic, it has a section called suggested_topics.

https://meta.discourse.org/t/144806/2.json:

From title to topic ID can be done using search:

https://meta.discourse.org/search.json?q=title%3A%22Is%20there%20is%20any%20open%20API%20for%20getting%20similar%20topic%20in%20discourse%22&page=1