Issues while importing attachments from XenForo

Hi guy’s,

I sucssesufuly migrated Xenforo DB with 1500 memebr and ~75.000 posts to new servere where is Discourse.

One question about attachment… from script I can see that if attachments folder exist then script will do import, my question is where I need to put attac from old server in:

a) In Docker part of server (somewhere)
b) In origin location /var/discourse/shared/standalone/tmp/attachments

?

1 Like

Anyone know path???

1 Like

You put the files wherever you want that is accessible from the machine doing the import and put that path here:

  ATTACHMENT_DIR = '/tmp/attachments'

If you’re running inside a docker container, and put it under var/discourse/shared/standalone/tmp/attachments then you’d use /shared/tmp/attachments for the path. You can check by looking for the files when you’re inside the container.

2 Likes

Okay Jay thank you for information, I’m just doing clean install this time with attachments folder setup, I will put then in /shared/tmp/attachments and get back here with results.

One quick question… is there anyway to import password for account’s too?

Can someone look where is problem?

root@my-app:/var/www/discourse# RAILS_ENV=production bundle exec ruby script/import_scripts/xenforo.rb
Loading existing groups...
Loading existing users...
Loading existing categories...
Loading existing posts...
Loading existing topics...

creating users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 4 already imported users

importing categories...
       23 / 23 (100.0%)  [326682 items/min]
creating topics and posts
       74 / 74752 (  0.1%)  [99817 items/min]  Traceback (most recent call last):
        18: from script/import_scripts/xenforo.rb:396:in `<main>'
        17: from /var/www/discourse/script/import_scripts/base.rb:47:in `perform'
        16: from script/import_scripts/xenforo.rb:32:in `execute'
        15: from script/import_scripts/xenforo.rb:174:in `import_posts'
        14: from /var/www/discourse/script/import_scripts/base.rb:869:in `batches'
        13: from /var/www/discourse/script/import_scripts/base.rb:869:in `loop'
        12: from /var/www/discourse/script/import_scripts/base.rb:870:in `block in batches'
        11: from script/import_scripts/xenforo.rb:180:in `block in import_posts'
        10: from /var/www/discourse/script/import_scripts/base.rb:490:in `create_posts'
         9: from /var/www/discourse/script/import_scripts/base.rb:490:in `each'
         8: from /var/www/discourse/script/import_scripts/base.rb:491:in `block in create_posts'
         7: from script/import_scripts/xenforo.rb:186:in `block (2 levels) in import_posts'
         6: from script/import_scripts/xenforo.rb:315:in `process_xenforo_post'
         5: from script/import_scripts/xenforo.rb:324:in `process_xf_attachments'
         4: from /usr/local/lib/ruby/2.6.0/set.rb:338:in `each'
         3: from /usr/local/lib/ruby/2.6.0/set.rb:338:in `each_key'
         2: from script/import_scripts/xenforo.rb:326:in `block in process_xf_attachments'
         1: from /usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.1/lib/active_support/core_ext/string/filters.rb:22:in `squish!'
/usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.1/lib/active_support/core_ext/string/filters.rb:22:in `gsub!': can't modify frozen String (FrozenError)
root@my-app:/var/www/discourse#

I’ve put this time attachments to: var/discourse/shared/standalone/tmp/attachments

I’ve found it… problem is that a folder containing attachments for import MUST be inside Docker app in patch /var/www/discourse/…/tmp/attachments

Now it’s OK script is going on, it will finish in ~1.5h

edit: It seems that again import attachments not working… im currently at 78% of import, and in every post where we had some picture or attachments stay [ATTACH]9788[/ATTACH] and number between tags and number of uploaded so next one is [ATTACH]9789[/ATTACH]…

Any help with this will be glad :confused:

1 Like

If you imported any posts before you properly set the attachment_dir, you’ll need to delete all posts and start the import again.

Unfortunately import is done right now and I dont see in log any information regardings attachments?

If you ran the import script and the attachment directory didn’t exist it skipped the attachments.

I didn’t do that… I’ve runned fresh import with attachments in place.

I don’t think that I can help you here. If you have a budget, please see https://www.literatecomputing.com/discourse-migration/

Here is my last piece of advice: I’m fairly certain that the attachments are not where the script could find them and that you will need to delete everything and start again.

Before you run the script again, from inside the folder, you should type

 ls /tmp/attachments

where /tmp/attachments is whatever you put in the

  ATTACHMENT_DIR = '/tmp/attachments'

line. If it doesn’t list the attachments, you need to keep trying to figure out where they are.

2 Likes

Okay could you tell me exectly where to put attachments from Xenforo into new servere so i can for sure know that script will find it?

Anyone?

I’m doing now fresh import (5th time)… I’m tired… :roll_eyes:

Do I need to sort in some order attachments fro Xenforo or just to Copy/Paste structure and /tmp/attachments ?

I’am now doing installation on new VM with snapshot so I can quickly chnage details and start again with migration.

Yesterday i put attachments folder from Xenforo which has structure like, main folder with name attachements and inside 9 subfolder with name from 0 to 9 and inside of each of them are around 600-700 attachments (.jpg) I’ve put folder in var/discourse/shared/standalone/tmp/attachments and chnage xenforo.rb script and put ```
ATTACHMENT_DIR = ‘/shared/tmp/attachments’

I’ve test it with “ls /tmp/attachments” and I’ve got true structure, but i face same error like first time:

creating topics and posts
74 / 74752 ( 0.1%) [99817 items/min] Traceback (most recent call last):
18: from script/import_scripts/xenforo.rb:396:in <main>' 17: from /var/www/discourse/script/import_scripts/base.rb:47:in perform’
16: from script/import_scripts/xenforo.rb:32:in execute' 15: from script/import_scripts/xenforo.rb:174:in import_posts’
14: from /var/www/discourse/script/import_scripts/base.rb:869:in batches' 13: from /var/www/discourse/script/import_scripts/base.rb:869:in loop’
12: from /var/www/discourse/script/import_scripts/base.rb:870:in block in batches' 11: from script/import_scripts/xenforo.rb:180:in block in import_posts’
10: from /var/www/discourse/script/import_scripts/base.rb:490:in create_posts' 9: from /var/www/discourse/script/import_scripts/base.rb:490:in each’
8: from /var/www/discourse/script/import_scripts/base.rb:491:in block in create_posts' 7: from script/import_scripts/xenforo.rb:186:in block (2 levels) in import_posts’
6: from script/import_scripts/xenforo.rb:315:in process_xenforo_post' 5: from script/import_scripts/xenforo.rb:324:in process_xf_attachments’
4: from /usr/local/lib/ruby/2.6.0/set.rb:338:in each' 3: from /usr/local/lib/ruby/2.6.0/set.rb:338:in each_key’
2: from script/import_scripts/xenforo.rb:326:in block in process_xf_attachments' 1: from /usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.1/lib/active_support/core_ext/string/filters.rb:22:in squish!’
/usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.1/lib/active_support/core_ext/string/filters.rb:22:in gsub!': can't modify frozen String (FrozenError) root@my-app:/var/www/discourse#Preformatted text`

2019-11-15T23:00:00Z

I’ve remove ! from gsub! string at location:

/usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.1/lib/active_support/core_ext/string/filters.rb

Now, import script does not stop but continiue and show this information about attachments:

Anyway for example last string:

Could not find file /shared/tmp/attachments//2/2485-c3da9f6ee1e33d3ebf418c5dea1f9e8a.data. Skipping attachment id 2485

…it IS on that path, but there is no .data. instend data there is .JPG

Any help is welcome.

2019-11-15T23:00:00Z

Script has a bug I’ve find it and fix it now i manage to import attachments.

1 Like