# Autobot plugin - Duplicate posts problem

**URL:** <https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806>\
**Category:** Support\
**Created:** [November 12, 2017, 3:27pm UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806 "2017-11-12T15:27:09Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![kuyashi](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/kuyashi/32/82068_2.png) [@kuyashi](https://meta.discourse.org/u/kuyashi)\
**Post date:** [November 12, 2017, 3:27pm UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/1 "2017-11-12T15:27:09Z")

</div>

When importing via an RSS feed into a category:

`https://medium.freecodecamp.org/feed`

Each time the source is polled, the same article is pulled in and duplicated as a topic each time.

Here is the logs:

`Job exception: undefined method `acting_user=' for nil:NilClass`

```
/var/www/discourse/lib/post_revisor.rb:133:in `revise!'
/var/www/discourse/app/models/post.rb:465:in `revise'
/var/www/discourse/app/jobs/regular/pull_hotlinked_images.rb:112:in `execute'
/var/www/discourse/app/jobs/base.rb:134:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rails_multisite-1.1.1/lib/rails_multisite/connection_management.rb:73:in `with_connection'
/var/www/discourse/app/jobs/base.rb:129:in `block in perform'
/var/www/discourse/app/jobs/base.rb:125:in `each'
/var/www/discourse/app/jobs/base.rb:125:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:188:in `execute_job'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:170:in `block (2 levels) in process'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/var/www/discourse/lib/sidekiq/pausable.rb:80:in `call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:169:in `block in process'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:141:in `block (6 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/job_retry.rb:97:in `local'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:140:in `block (5 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq.rb:36:in `block in <module:Sidekiq>'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:136:in `block (4 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:204:in `stats'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:131:in `block (3 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/job_logger.rb:7:in `call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:130:in `block (2 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/job_retry.rb:72:in `global'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:129:in `block in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/logging.rb:44:in `with_context'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/logging.rb:38:in `with_job_hash_context'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:128:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:168:in `process'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:85:in `process_one'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'

```

---

<div class="post-metadata">

**Author:** ![vinothkannans](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/vinothkannans/32/86465_2.png) [@vinothkannans](https://meta.discourse.org/u/vinothkannans)\
**Post date:** [November 18, 2017, 2:55pm UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/2 "2017-11-18T14:55:09Z")

</div>

@schungx, Thanks for your PR 👍. I merged it. It seemed good to me and I didn’t tested well. Let me know if any issues raised.

@kuyashi, Since currently I am working in official plugins I am unable to check your report soon. I will try to have a look at this weekend.

---

<div class="post-metadata">

**Author:** ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)\
**Post date:** [November 21, 2017, 3:55pm UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/3 "2017-11-21T15:55:39Z")

</div>

I am wondering, if a post created by the autobot is _DELETED_, then what happens?

> <https://github.com/vinkashq/discourse-autobot/blob/master/lib/autobot/post_creator.rb#L111>

Notice the use of `first` here – it finds the _first_ post with the correct feed item url. Why the first one? Shouldn’t it be the _LAST_ one at least? And shouldn’t the list be sorted (say `order(:post_id)`) before taking `first` or `last` so it is at least predictable?

Since the post custom fields are never deleted together with the post, this almost means that the _deleted_ first post will be found every time. Mostly like meaning that `try(:post)` will return `nil` and then a new post gets created EVERY TIME the RSS feed is scanned.

This, I believe, is the reason behind all those duplicated posts.

Shouldn’t the correct behavior be to:

1. First sort the posts by post\_id (same as by date)
2. Filter out all the deleted posts
3. Take the last one

I would have made a PR if I know how to… but I’m not too comfortable with all those ActiveRecord stuff.

---

<div class="post-metadata">

**Author:** ![vinothkannans](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/vinothkannans/32/86465_2.png) [@vinothkannans](https://meta.discourse.org/u/vinothkannans)\
**Post date:** [November 21, 2017, 5:48pm UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/4 "2017-11-21T17:48:11Z")

</div>

First of all you can’t delete a post completely. For more details [Search results for 'how delete post completely' - Discourse Meta](https://meta.discourse.org/search?q=how%20delete%20post%20completely)

> [@schungx](#):
>
> 1.First sort the posts by post\_id (same as by date)  
> 3.Take the last one

As per this [line](https://github.com/vinkashq/discourse-autobot/blob/master/lib/autobot/post_creator.rb#L105) of code duplicate posts won’t created. So no need for sort. In this case `first` and `last` are same.

I guess @kuyashi’s duplicate post problem is special case. I need some time to check that issue.

> [@schungx](#):
>
> I would have made a PR if I know how to…

If you like to contribute then please create PR for the features you already suggested in previous [post](https://meta.discourse.org/t/autobot-automatic-content-creator/71756/36).

---

<div class="post-metadata">

**Author:** ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)\
**Post date:** [November 22, 2017, 2:29am UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/5 "2017-11-22T02:29:49Z")

</div>

It is not just @kuyashi, I am also getting duplicate posts _every single time_ that the RSS feed is scanned. If it is scanned three times, then three duplicated posts.

So this is not an isolated problem.

---

<div class="post-metadata">

**Author:** ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)\
**Post date:** [November 22, 2017, 2:35am UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/6 "2017-11-22T02:35:15Z")

</div>

> [@vinothkannans](#):
>
> As per this line of code duplicate posts won’t created. So no need for sort. In this case first and last are same.

Not so sure about this. For example:

```plaintext
post = get_existing_post || post_creator.create!

```

However:

```plaintext
def get_existing_post
      return PostCustomField.where(name: "autobot_source_url", value: source_url).first.try(:post)
end

```

Now, assume there are _multiple_ records in `post_custom_fields` with the same `autobot_source_url`, all but the last one of them point to posts that are deleted.

I am not sure if `PostCustomField.where(name: "autobot_source_url", value: source_url)` will also return all records (included those pointing to deleted posts) or just the _live_ one.

If it returns only the _live_ one, then you’re probably correct that `first == last`. However, I’m not so sure about that assumption. AFAICT, `PostCustomField` will return all the records matching the `autobot_source_url`, including those pointing to deleted posts.

So, when you take the `first` of that, and access the `:post`, I’m not sure what will result. My suspicion is that it returns `nil`, so that the clause falls into `post_creator.create!` and a new post is created. Every single time, because you’ll always find the deleted post first.

---

<div class="post-metadata">

**Author:** ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)\
**Post date:** [November 22, 2017, 4:01am UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/7 "2017-11-22T04:01:29Z")

</div>

1. Deleting all the old posts
2. Changing the RSS feed for the items to have new URL’s
3. New posts are created
4. New posts are no longer duplicated _as long as none is deleted_

Therefore, I’d heavily suspect that it is _deleted_ RSS posts that cause the duplicated posts.

---

<div class="post-metadata">

**Author:** ![Jon\_Hawkins](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jon_hawkins/32/64130_2.png) [@Jon\_Hawkins](https://meta.discourse.org/u/Jon_Hawkins)\
**Post date:** [November 22, 2017, 5:30am UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/8 "2017-11-22T05:30:05Z")

</div>

@schungx , I have the exact same issue. Have been following along on this thread, thought i might have an older version, or maybe an issue with the feed and it’s not the case. Every refresh = duplicate post created.

---

<div class="post-metadata">

**Author:** ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)\
**Post date:** [November 22, 2017, 5:46am UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/9 "2017-11-22T05:46:40Z")

</div>

@Jon_Hawkins, make sure you pull the latest one. It should solve a few problems.

On the duplicated posts, how _many_ posts are duplicated each scan of the feed? I find that only posts that were ever _deleted_ before will continue to add new duplicated posts indefinitely. However posts that were never deleted are never duplicated.

Are any of your feed posts deleted?

A good way to find out is to:

1. Open [Data Explorer](https://meta.discourse.org/t/32566?silent=true)
2. `select * from post_custom_fields where name='autobot_source_url' and value='`_your duplicated feed item URL here_`'`
3. See if how many posts got returned
4. If \> 1, then you have some deleted posts and that’s creating the duplication problem

---

<div class="post-metadata">

**Author:** ![Jon\_Hawkins](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jon_hawkins/32/64130_2.png) [@Jon\_Hawkins](https://meta.discourse.org/u/Jon_Hawkins)\
**Post date:** [November 22, 2017, 5:48am UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/10 "2017-11-22T05:48:46Z")

</div>

I think i didi a rebuild of the app 4 days ago and have the latest build. This is my first plugin though, is there anything specific i need to do besides ./launcher rebuild app?

---

<div class="post-metadata">

**Author:** ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)\
**Post date:** [November 22, 2017, 5:50am UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/11 "2017-11-22T05:50:13Z")

</div>

> [@Jon\_Hawkins](#):
>
> is there anything specific i need to do besides ./launcher rebuild app?

Not that I know of. How many duplicates do you get?

---

<div class="post-metadata">

**Author:** ![Jon\_Hawkins](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jon_hawkins/32/64130_2.png) [@Jon\_Hawkins](https://meta.discourse.org/u/Jon_Hawkins)\
**Post date:** [November 22, 2017, 5:51am UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/12 "2017-11-22T05:51:59Z")

</div>

Im using zapier to build a custom RSS feed of only 1 item. When autobot is live on that zapier feed it will create the same post over and over again, as each interval is hit. Let me rebuild the app now and just confirm I have the latest greatest.

---

<div class="post-metadata">

**Author:** ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)\
**Post date:** [November 22, 2017, 6:00am UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/13 "2017-11-22T06:00:19Z")

</div>

1. Did you _ever_ delete the first post created by that feed?

2. What happens if you change your feed item’s URL to a different one?

---

<div class="post-metadata">

**Author:** ![Jon\_Hawkins](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jon_hawkins/32/64130_2.png) [@Jon\_Hawkins](https://meta.discourse.org/u/Jon_Hawkins)\
**Post date:** [November 22, 2017, 6:04am UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/14 "2017-11-22T06:04:00Z")

</div>

I’m fairly certain I’ve only deleted the first post created by that feed after noticing a duplicate but let me test again and let you know.

I have never changed the feed URL to another feed. I just deleted the entire record and created a new one from scratch. I think I went to edit the existing feed one time and noticed something weird when saving so figured I’d just remove it and start fresh.

---

<div class="post-metadata">

**Author:** ![schungx](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schungx/32/70989_2.png) [@schungx](https://meta.discourse.org/u/schungx)\
**Post date:** [November 22, 2017, 6:32am UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/15 "2017-11-22T06:32:20Z")

</div>

If you deleted the first post, then you’ll get duplicated posts _forever_. You have to change the URL.

Basically the first post is the most important. If it is deleted, a new post will be created during every refresh.

Deleting the campaign won’t help because the plugin won’t distinguish between different campaigns. This is probably also a potential problem, if an item is ever shared between two different feeds in two different topics/categories. They’ll never duplicate in this case.

Source:

> <https://github.com/vinkashq/discourse-autobot/blob/master/lib/autobot/post_creator.rb#L111>

It probably should be something like:

```plaintext
PostCustomField
    .where(name: "autobot_source_url", value: source_url)
    .where(name: "autobot_campaign_id", value: campaign["id"])
    .first.try(:post)

```

---

<div class="post-metadata">

**Author:** ![Jon\_Hawkins](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jon_hawkins/32/64130_2.png) [@Jon\_Hawkins](https://meta.discourse.org/u/Jon_Hawkins)\
**Post date:** [November 22, 2017, 1:09pm UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/16 "2017-11-22T13:09:09Z")

</div>

THanks @schungx! This all makes sense based on what I’m experiencing.

---

<div class="post-metadata">

**Author:** ![vinothkannans](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/vinothkannans/32/86465_2.png) [@vinothkannans](https://meta.discourse.org/u/vinothkannans)\
**Post date:** [November 22, 2017, 8:25pm UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/17 "2017-11-22T20:25:44Z")

</div>

Sorry currently I don’t have much time to look into this plugin code. I will check it all and will do the improvements as soon as possible. May be in next couple of weeks.

@schungx thanks for your PR. I merged it.

---

<div class="post-metadata">

**Author:** ![vinothkannans](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/vinothkannans/32/86465_2.png) [@vinothkannans](https://meta.discourse.org/u/vinothkannans)\
**Post date:** [November 25, 2017, 6:17pm UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/18 "2017-11-25T18:17:39Z")

</div>

Hey, I fixed this problem now. @schungx is correct. Deleted posts are causing the issue. But using the `last` method not resolving it (anyway as per his wish I added it too 😉). Instead I used `unscoped` option to include soft deleted posts while finding records. Anyway thanks for Stephan 👍.

@kuyashi @Jon_Hawkins

---

<div class="post-metadata">

**Author:** ![Jon\_Hawkins](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jon_hawkins/32/64130_2.png) [@Jon\_Hawkins](https://meta.discourse.org/u/Jon_Hawkins)\
**Post date:** [November 25, 2017, 10:50pm UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/19 "2017-11-25T22:50:12Z")

</div>

Wahoo! I’ll try it out again tonight. Thanks @vinothkannans

---

<div class="post-metadata">

**Author:** ![system](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/system/32/443519_2.png) [@system](https://meta.discourse.org/u/system)\
**Post date:** [January 20, 2020, 11:07pm UTC](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806/20 "2020-01-20T23:07:29Z")

</div>



[Next page](https://meta.discourse.org/t/autobot-plugin-duplicate-posts-problem/74806.md?page=2)