Duplicate topics constantly created

Support

AI-generated summary

The user spectrum is experiencing an issue with duplicate topics being created on their Discourse platform, despite using the Canonical URL option to prevent this. The problem started after migrating from Drupal to WordPress. To address this, the user’s web development company suggests two potential solutions:

  1. Adding a hidden excerpt to each article page with a custom CSS selector, which would allow Discourse to crawl and pick up the excerpt.
  2. Adding the RSS/ATOM feed URL to each post programmatically, making it explicit to Discourse scrapers.

The user is seeking help and experience with these solutions to resolve the issue of duplicate topics being created multiple times per hour.

1

Hello,
We have had a recurring issue where duplicate topics are created, seemingly at random. We are using the following embed code on our WordPress website:

 <script type="text/javascript">
            var discourseUrl = "https://discourse.sitename.com/",
            discourseEmbedUrl = "https://sitename.com/' . $post-category . '/' . $post-url . '";

    (function() {
      var d = document.createElement('script');
      d.type = 'text/javascript';
      d.async = true;
      d.src = DiscourseEmbed.discourseUrl + 'javascripts/embed.js';
      (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(d);
    })();
</script>

To my knowledge, Discourse is supposed to create topics based on Canonical URL of a post.

I enabled the Embed set canonical URL option, but it looks like we’re still getting duplicate topics created. We did set them to be unlisted though and have been manually unhiding on new legitimate topic creation.

The web development company we’re working with, since migrating from Drupal to WordPress, has asked a few questions about this:

  • If we were to add a excerpt that would be a repeat of the og:meta description (so, the specified excerpt of the articles) to each article page and wrap that excerpt in a custom CSS selector that we would provide to Discourse AND a CSS tag that would also make this excerpt hidden, would Discourse successfully crawl the code and pick up the excerpt even with it being hidden?

  • It looks like Discourse first checks for an RSS/ATOM feed, and I have verified that there are actually two of those: https://sitename.com/feed AND https://sitename.com/category/news/feed. There is also a feed for individual pages.

  • This URL is not provided in the code of the page itself, though. If we were to add this feed URL to each post programmatically, do you think this may solve the problem? What specific code/meta tag/CSS selector should be used to make this explicit to Discourse scrapers?

Would anyone have experience with this sort of thing? We have been getting duplicate topics created multiple times per hour now, and have since resorted to automatically hiding every embed topic upon creation, then manually unhiding topics that are confirmed to not be duplicates.

Any help would be greatly appreciated!