Bug when importing


(Michael - DiscourseHosting.com) #1

I get an error when importing posts to a forum, seems like a really weird bug. Occurs in about 0.01% of the posts being imported.

    #<ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint "cat_featured_threads"
    DETAIL:  Key (category_id, topic_id)=(6, 524) already exists.
    : INSERT INTO "category_featured_topics" ("category_id", "created_at", "rank", "topic_id", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id">
    
    /var/www/discourse/app/models/category_featured_topic.rb:25:in `block (2 levels) in feature_topics_for'
    /var/www/discourse/app/models/category_featured_topic.rb:24:in `each'
    /var/www/discourse/app/models/category_featured_topic.rb:24:in `each_with_index'
    /var/www/discourse/app/models/category_featured_topic.rb:24:in `block in feature_topics_for'
...
/var/www/discourse/lib/topic_creator.rb:102:in `save_topic'
/var/www/discourse/lib/topic_creator.rb:22:in `create'
/var/www/discourse/lib/post_creator.rb:182:in `setup_topic'
/var/www/discourse/lib/post_creator.rb:57:in `block in create'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:202:in `block in transaction'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:210:in `within_new_transaction'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:202:in `transaction'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.2/lib/active_record/transactions.rb:209:in `transaction'
/var/www/discourse/lib/post_creator.rb:56:in `create'

....

(Régis Hanol) #2

I’m in the process of rebuilding our backup/restore features. We’ll be using pg_dump & pg_restore for awesome speed and none of that non-sense :wink:


(Michael - DiscourseHosting.com) #3

Ok, I wasn’t really being clear, my apologies.

First of all, I’m not doing an import of a previously exported forum, but I’m running a conversion and creating posts using PostCreator.

Second I forgot to mention that I was running with this patch and since then this error is being exposed. Which doesn’t make sense, but ah well.

About your pg_dump remark (feel free to move this to a different thread), I think we have

  • importing from another kind of forum. Different brand to Discourse. What I’m doing right now. Custom script.

  • the current rake import / export tools. As far as I understood this should eventually be able to move between different Discourse versions and database names as well. Database names is mostly working (except uploaded files), different Discourse versions is still a problem.

  • pg_dump / pg_restore which can only move between same Discourse version and same database name.

So while faster, it’s much more limited.


(Régis Hanol) #4

My fault. I did not have my morning coffee and thought you were restoring a Discourse backup.

Conversion is a totally different subject :smile:


(Michael - DiscourseHosting.com) #5

No mine, I wasn’t explaining well enough…
And I was lacking coffee as well :stuck_out_tongue:


(Roberto_Pezzali) #6

I think that the problem is related to the welcome category post that Discourse create when you migrate a post with a category that doesn’t exist.


(Michael - DiscourseHosting.com) #7

I don’t think that’s the issue, we’re creating those categories nicely and that works well.

I’ve got two theories:

  1. just above the error the table is cleared of all entries containing this specific category. Maybe the database is slow and this is asynchronous?
  2. the query returning the threads to be featured is returning duplicate ID’s

It’s hard to debug, since it requires running the conversion for over 20-30 minutes in order to make it happen…


(Roberto_Pezzali) #8

I have this issue when importing post.
I create the category, but Discourse autocreate a post with every category created. When I try to import a post that has the same category ID of the “category post” I have that error.


(Michael - DiscourseHosting.com) #9

Why don’t you let Discourse decide the post id ?


(Roberto_Pezzali) #10

Because I must keep the original post ID of my previous forum, so I can create a 301 redirect in htaccess to keep Google SEO


(Michael - DiscourseHosting.com) #11

We’re doing that differently, we create tables containing the old ID and new ID, then use that database table to perform the redirect. To avoid such problems as you’re having :smile: