Importing Xenforo to Discourse

hi, can u help me with converting xenforo to discourse? tnx

What are the list of XenForo Data covered on this import?
I hope to import this list:

XenForo Page Nodes

  • Pages

XenForo

  • Profile Fields
  • Member Groups
  • Members
  • Status Updates / Profile Comments
  • Status Comments
  • Private Messages
  • Private Message Replies
  • Attachments
  • Member Ranks

XenForo Forums

  • Forums
  • Topics
  • Posts
  • Attachments

Got this error:

# bundle install --no-deployment
Don't run Bundler as root. Bundler can ask for sudo if it is needed, and installing your bundle as root will break this application for all non-root users on this machine.
Your Gemfile has no gem server sources. If you need gems that are not already on your machine, add a line like this to your Gemfile:
source 'https://rubygems.org'
Could not find gem 'mysql2' in any of the gem sources listed in your Gemfile.

Different Error now.

root@discourse-app:/var/www/discourse# RAILS_ENV=production bundle exec ruby script/import_scripts/xenforo.rb
No connection to db, unable to retrieve site settings! (normal when running db:create)
No connection to db, unable to retrieve site settings! (normal when running db:create)
No connection to db, unable to retrieve site settings! (normal when running db:create)
No connection to db, unable to retrieve site settings! (normal when running db:create)
No connection to db, unable to retrieve site settings! (normal when running db:create)
No connection to db, unable to retrieve site settings! (normal when running db:create)
Traceback (most recent call last):
	31: from script/import_scripts/xenforo.rb:3:in `<main>'
	30: from script/import_scripts/xenforo.rb:3:in `require'
	29: from /var/www/discourse/script/import_scripts/base.rb:20:in `<top (required)>'
	28: from /var/www/discourse/script/import_scripts/base.rb:544:in `<class:Base>'
	27: from /var/www/discourse/lib/discourse.rb:411:in `system_user'
	26: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/core.rb:196:in `find_by'
	25: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/core.rb:196:in `all?'
	24: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/core.rb:196:in `each'
	23: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/core.rb:196:in `block in find_by'
	22: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/model_schema.rb:336:in `columns_hash'
	21: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/model_schema.rb:456:in `load_schema'
	20: from /usr/local/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
	19: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/model_schema.rb:459:in `block in load_schema'
	18: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/attribute_decorators.rb:51:in `load_schema!'
	17: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/attributes.rb:234:in `load_schema!'
	16: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/model_schema.rb:466:in `load_schema!'
	15: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_handling.rb:90:in `connection'
	14: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_handling.rb:118:in `retrieve_connection'
	13: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:1008:in `retrieve_connection'
	12: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:380:in `connection'
	11: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:521:in `checkout'
	10: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:793:in `acquire_connection'
	 9: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:832:in `try_to_checkout_new_connection'
	 8: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:853:in `checkout_new_connection'
	 7: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:809:in `new_connection'
	 6: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb:40:in `postgresql_connection'
	 5: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb:40:in `new'
	 4: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb:215:in `initialize'
	 3: from /usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb:684:in `connect'
	 2: from /usr/local/lib/ruby/gems/2.5.0/gems/pg-1.0.0/lib/pg.rb:56:in `connect'
	 1: from /usr/local/lib/ruby/gems/2.5.0/gems/pg-1.0.0/lib/pg.rb:56:in `new'
/usr/local/lib/ruby/gems/2.5.0/gems/pg-1.0.0/lib/pg.rb:56:in `initialize': FATAL:  Peer authentication failed for user "discourse" (PG::ConnectionBad)

Yay no idea how long this will take :smiley:

2 Likes

This is correct right?

Guide needs an update I think.
Before running this, directory must be within /var/www/discourse/.
cd /var/www/discourse/

3 Likes

I used this script today and it worked well. The only issue I had was that it imported all of the banned users and soft deleted (hidden) posts/threads from Xenforo. That made hundreds of spam posts/threads visible in Discourse after the import.

If you have a lot of spam post/threads that you don’t want to import, then here are a few SQL changes you should consider making to the “xenforo.rb” script:

  1. Add the WHERE clause below to ensure only active, unbanned users are imported.

    def import_users
     puts '', "creating users"
    
     total_count = mysql_query("SELECT count(*) count FROM #{TABLE_PREFIX}user;").first['count']
    
     batches(BATCH_SIZE) do |offset|
       results = mysql_query(
         "SELECT user_id id, username, email, custom_title title, register_date created_at,
                 last_activity last_visit_time, user_group_id, is_moderator, is_admin, is_staff
          FROM #{TABLE_PREFIX}user
          WHERE user_state = 'valid' AND is_banned = 0
          LIMIT #{BATCH_SIZE}
          OFFSET #{offset};")
    

    2. Modify the WHERE clause below to ensure only visible threads/posts are imported.
    def import_posts
     puts "", "creating topics and posts"
    
     total_count = mysql_query("SELECT count(*) count from #{TABLE_PREFIX}post").first["count"]
    
     posts_sql = "
         SELECT p.post_id id,
                t.thread_id topic_id,
                #{@prefix_as_category ? 't.prefix_id' : 't.node_id'} category_id,
                t.title title,
                t.first_post_id first_post_id,
                p.user_id user_id,
                p.message raw,
                p.post_date created_at
         FROM #{TABLE_PREFIX}post p,
              #{TABLE_PREFIX}thread t
         WHERE p.thread_id = t.thread_id AND p.message_state = 'visible' AND t.discussion_state = 'visible'
         ORDER BY p.post_date
         LIMIT #{BATCH_SIZE}" # needs OFFSET
5 Likes

Sure thanks for that, can we fold those changes into our script @techAPJ?

4 Likes

Done via:

Thanks for the tips @msinger. :+1:

5 Likes

8 posts were split to a new topic: Help migrating Xenforo

Hi guy’s,

I sucssesufuly migrated Xenforo DB with 1500 memebr and ~75.000 posts to new servere where is Discourse.

One question about attachment… from script I can see that if attachments folder exist then script will do import, my question is where I need to put attac from old server in:

a) In Docker part of server (somewhere)
b) In origin location /var/discourse/shared/standalone/tmp/attachments

?

1 Like

Anyone know path???

1 Like

You put the files wherever you want that is accessible from the machine doing the import and put that path here:

  ATTACHMENT_DIR = '/tmp/attachments'

If you’re running inside a docker container, and put it under var/discourse/shared/standalone/tmp/attachments then you’d use /shared/tmp/attachments for the path. You can check by looking for the files when you’re inside the container.

1 Like

Okay Jay thank you for information, I’m just doing clean install this time with attachments folder setup, I will put then in /shared/tmp/attachments and get back here with results.

One quick question… is there anyway to import password for account’s too?

Can someone look where is problem?

root@my-app:/var/www/discourse# RAILS_ENV=production bundle exec ruby script/import_scripts/xenforo.rb
Loading existing groups...
Loading existing users...
Loading existing categories...
Loading existing posts...
Loading existing topics...

creating users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 4 already imported users

importing categories...
       23 / 23 (100.0%)  [326682 items/min]
creating topics and posts
       74 / 74752 (  0.1%)  [99817 items/min]  Traceback (most recent call last):
        18: from script/import_scripts/xenforo.rb:396:in `<main>'
        17: from /var/www/discourse/script/import_scripts/base.rb:47:in `perform'
        16: from script/import_scripts/xenforo.rb:32:in `execute'
        15: from script/import_scripts/xenforo.rb:174:in `import_posts'
        14: from /var/www/discourse/script/import_scripts/base.rb:869:in `batches'
        13: from /var/www/discourse/script/import_scripts/base.rb:869:in `loop'
        12: from /var/www/discourse/script/import_scripts/base.rb:870:in `block in batches'
        11: from script/import_scripts/xenforo.rb:180:in `block in import_posts'
        10: from /var/www/discourse/script/import_scripts/base.rb:490:in `create_posts'
         9: from /var/www/discourse/script/import_scripts/base.rb:490:in `each'
         8: from /var/www/discourse/script/import_scripts/base.rb:491:in `block in create_posts'
         7: from script/import_scripts/xenforo.rb:186:in `block (2 levels) in import_posts'
         6: from script/import_scripts/xenforo.rb:315:in `process_xenforo_post'
         5: from script/import_scripts/xenforo.rb:324:in `process_xf_attachments'
         4: from /usr/local/lib/ruby/2.6.0/set.rb:338:in `each'
         3: from /usr/local/lib/ruby/2.6.0/set.rb:338:in `each_key'
         2: from script/import_scripts/xenforo.rb:326:in `block in process_xf_attachments'
         1: from /usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.1/lib/active_support/core_ext/string/filters.rb:22:in `squish!'
/usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.1/lib/active_support/core_ext/string/filters.rb:22:in `gsub!': can't modify frozen String (FrozenError)
root@my-app:/var/www/discourse#

I’ve put this time attachments to: var/discourse/shared/standalone/tmp/attachments

I’ve found it… problem is that a folder containing attachments for import MUST be inside Docker app in patch /var/www/discourse/…/tmp/attachments

Now it’s OK script is going on, it will finish in ~1.5h

edit: It seems that again import attachments not working… im currently at 78% of import, and in every post where we had some picture or attachments stay [ATTACH]9788[/ATTACH] and number between tags and number of uploaded so next one is [ATTACH]9789[/ATTACH]…

Any help with this will be glad :confused:

1 Like

If you imported any posts before you properly set the attachment_dir, you’ll need to delete all posts and start the import again.

Unfortunately import is done right now and I dont see in log any information regardings attachments?

If you ran the import script and the attachment directory didn’t exist it skipped the attachments.