Duplicate topics constantly created

Support
1

Hello,
We have had a recurring issue where duplicate topics are created, seemingly at random. We are using the following embed code on our WordPress website:

 <script type="text/javascript">
            var discourseUrl = "https://discourse.sitename.com/",
            discourseEmbedUrl = "https://sitename.com/' . $post-category . '/' . $post-url . '";

    (function() {
      var d = document.createElement('script');
      d.type = 'text/javascript';
      d.async = true;
      d.src = DiscourseEmbed.discourseUrl + 'javascripts/embed.js';
      (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(d);
    })();
</script>

To my knowledge, Discourse is supposed to create topics based on Canonical URL of a post.

I enabled the Embed set canonical URL option, but it looks like we’re still getting duplicate topics created. We did set them to be unlisted though and have been manually unhiding on new legitimate topic creation.

The web development company we’re working with, since migrating from Drupal to WordPress, has asked a few questions about this:

  • If we were to add a excerpt that would be a repeat of the og:meta description (so, the specified excerpt of the articles) to each article page and wrap that excerpt in a custom CSS selector that we would provide to Discourse AND a CSS tag that would also make this excerpt hidden, would Discourse successfully crawl the code and pick up the excerpt even with it being hidden?

  • It looks like Discourse first checks for an RSS/ATOM feed, and I have verified that there are actually two of those: https://sitename.com/feed AND https://sitename.com/category/news/feed. There is also a feed for individual pages.

  • This URL is not provided in the code of the page itself, though. If we were to add this feed URL to each post programmatically, do you think this may solve the problem? What specific code/meta tag/CSS selector should be used to make this explicit to Discourse scrapers?

Would anyone have experience with this sort of thing? We have been getting duplicate topics created multiple times per hour now, and have since resorted to automatically hiding every embed topic upon creation, then manually unhiding topics that are confirmed to not be duplicates.

Any help would be greatly appreciated!