Issues while importing attachments from XenForo

Product Support Migration

AI-generated summary

Issues while importing attachments from XenForo

A user, vladak, is experiencing issues importing attachments from XenForo to Discourse. The user successfully migrated the XenForo database but is unsure where to place the attachments for the import script to find them.

vladak tried placing the attachments in different locations, including var/discourse/shared/standalone/tmp/attachments, but encountered errors. pfaffman advised that the attachments should be placed in a location accessible from the machine doing the import, and provided an example path, /shared/tmp/attachments.

However, vladak continued to encounter issues, including a frozen string error. pfaffman suggested that if any posts were imported before the attachment directory was properly set, all posts would need to be deleted and the import started again.

After multiple attempts, vladak discovered that the issue was due to a bug in the script, which expected attachment files to have a .data extension instead of the actual .JPG extension. The user was able to fix the bug and successfully import the attachments.

Key takeaways:

  • Attachments should be placed in a location accessible from the machine doing the import, such as /shared/tmp/attachments.
  • If any posts are imported before the attachment directory is properly set, all posts will need to be deleted and the import started again.
  • The script may have bugs that need to be fixed, such as incorrect file extension expectations.
1

Hi guy’s,

I sucssesufuly migrated Xenforo DB with 1500 memebr and ~75.000 posts to new servere where is Discourse.

One question about attachment… from script I can see that if attachments folder exist then script will do import, my question is where I need to put attac from old server in:

a) In Docker part of server (somewhere)
b) In origin location /var/discourse/shared/standalone/tmp/attachments

?

2

Anyone know path???

3

You put the files wherever you want that is accessible from the machine doing the import and put that path here:

  ATTACHMENT_DIR = '/tmp/attachments'

If you’re running inside a docker container, and put it under var/discourse/shared/standalone/tmp/attachments then you’d use /shared/tmp/attachments for the path. You can check by looking for the files when you’re inside the container.

4

Okay Jay thank you for information, I’m just doing clean install this time with attachments folder setup, I will put then in /shared/tmp/attachments and get back here with results.

One quick question… is there anyway to import password for account’s too?

5

Can someone look where is problem?

root@my-app:/var/www/discourse# RAILS_ENV=production bundle exec ruby script/import_scripts/xenforo.rb
Loading existing groups...
Loading existing users...
Loading existing categories...
Loading existing posts...
Loading existing topics...

creating users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 100 already imported users
Skipping 4 already imported users

importing categories...
       23 / 23 (100.0%)  [326682 items/min]
creating topics and posts
       74 / 74752 (  0.1%)  [99817 items/min]  Traceback (most recent call last):
        18: from script/import_scripts/xenforo.rb:396:in `<main>'
        17: from /var/www/discourse/script/import_scripts/base.rb:47:in `perform'
        16: from script/import_scripts/xenforo.rb:32:in `execute'
        15: from script/import_scripts/xenforo.rb:174:in `import_posts'
        14: from /var/www/discourse/script/import_scripts/base.rb:869:in `batches'
        13: from /var/www/discourse/script/import_scripts/base.rb:869:in `loop'
        12: from /var/www/discourse/script/import_scripts/base.rb:870:in `block in batches'
        11: from script/import_scripts/xenforo.rb:180:in `block in import_posts'
        10: from /var/www/discourse/script/import_scripts/base.rb:490:in `create_posts'
         9: from /var/www/discourse/script/import_scripts/base.rb:490:in `each'
         8: from /var/www/discourse/script/import_scripts/base.rb:491:in `block in create_posts'
         7: from script/import_scripts/xenforo.rb:186:in `block (2 levels) in import_posts'
         6: from script/import_scripts/xenforo.rb:315:in `process_xenforo_post'
         5: from script/import_scripts/xenforo.rb:324:in `process_xf_attachments'
         4: from /usr/local/lib/ruby/2.6.0/set.rb:338:in `each'
         3: from /usr/local/lib/ruby/2.6.0/set.rb:338:in `each_key'
         2: from script/import_scripts/xenforo.rb:326:in `block in process_xf_attachments'
         1: from /usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.1/lib/active_support/core_ext/string/filters.rb:22:in `squish!'
/usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.1/lib/active_support/core_ext/string/filters.rb:22:in `gsub!': can't modify frozen String (FrozenError)
root@my-app:/var/www/discourse#

I’ve put this time attachments to: var/discourse/shared/standalone/tmp/attachments

6

I’ve found it… problem is that a folder containing attachments for import MUST be inside Docker app in patch /var/www/discourse/…/tmp/attachments

Now it’s OK script is going on, it will finish in ~1.5h

edit: It seems that again import attachments not working… im currently at 78% of import, and in every post where we had some picture or attachments stay [ATTACH]9788[/ATTACH] and number between tags and number of uploaded so next one is [ATTACH]9789[/ATTACH]…

Any help with this will be glad :confused:

7

If you imported any posts before you properly set the attachment_dir, you’ll need to delete all posts and start the import again.

8

Unfortunately import is done right now and I dont see in log any information regardings attachments?

2019-11-11%2022_58_28-mc%20%5Broot%40foss-app%5D__
2019-11-11%2022_58_28-mc%20%5Broot%40foss-app%5D__402×879 79.7 KB

9

If you ran the import script and the attachment directory didn’t exist it skipped the attachments.

10

I didn’t do that… I’ve runned fresh import with attachments in place.

11

I don’t think that I can help you here. If you have a budget, please see Redirecting…

Here is my last piece of advice: I’m fairly certain that the attachments are not where the script could find them and that you will need to delete everything and start again.

Before you run the script again, from inside the folder, you should type

 ls /tmp/attachments

where /tmp/attachments is whatever you put in the

  ATTACHMENT_DIR = '/tmp/attachments'

line. If it doesn’t list the attachments, you need to keep trying to figure out where they are.

12

Okay could you tell me exectly where to put attachments from Xenforo into new servere so i can for sure know that script will find it?

13

Anyone?

I’m doing now fresh import (5th time)… I’m tired… :roll_eyes:

14

Do I need to sort in some order attachments fro Xenforo or just to Copy/Paste structure and /tmp/attachments ?

I’am now doing installation on new VM with snapshot so I can quickly chnage details and start again with migration.

Yesterday i put attachments folder from Xenforo which has structure like, main folder with name attachements and inside 9 subfolder with name from 0 to 9 and inside of each of them are around 600-700 attachments (.jpg) I’ve put folder in var/discourse/shared/standalone/tmp/attachments and chnage xenforo.rb script and put ```
ATTACHMENT_DIR = ‘/shared/tmp/attachments’

I’ve test it with “ls /tmp/attachments” and I’ve got true structure, but i face same error like first time:

creating topics and posts
74 / 74752 ( 0.1%) [99817 items/min] Traceback (most recent call last):
18: from script/import_scripts/xenforo.rb:396:in <main>' 17: from /var/www/discourse/script/import_scripts/base.rb:47:in perform’
16: from script/import_scripts/xenforo.rb:32:in execute' 15: from script/import_scripts/xenforo.rb:174:in import_posts’
14: from /var/www/discourse/script/import_scripts/base.rb:869:in batches' 13: from /var/www/discourse/script/import_scripts/base.rb:869:in loop’
12: from /var/www/discourse/script/import_scripts/base.rb:870:in block in batches' 11: from script/import_scripts/xenforo.rb:180:in block in import_posts’
10: from /var/www/discourse/script/import_scripts/base.rb:490:in create_posts' 9: from /var/www/discourse/script/import_scripts/base.rb:490:in each’
8: from /var/www/discourse/script/import_scripts/base.rb:491:in block in create_posts' 7: from script/import_scripts/xenforo.rb:186:in block (2 levels) in import_posts’
6: from script/import_scripts/xenforo.rb:315:in process_xenforo_post' 5: from script/import_scripts/xenforo.rb:324:in process_xf_attachments’
4: from /usr/local/lib/ruby/2.6.0/set.rb:338:in each' 3: from /usr/local/lib/ruby/2.6.0/set.rb:338:in each_key’
2: from script/import_scripts/xenforo.rb:326:in block in process_xf_attachments' 1: from /usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.1/lib/active_support/core_ext/string/filters.rb:22:in squish!’
/usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.1/lib/active_support/core_ext/string/filters.rb:22:in gsub!': can't modify frozen String (FrozenError) root@my-app:/var/www/discourse#Preformatted text`

2019-11-15T23:00:00Z

I’ve remove ! from gsub! string at location:

/usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.1/lib/active_support/core_ext/string/filters.rb

Now, import script does not stop but continiue and show this information about attachments:

2019-11-16%2011_26_06-Window
2019-11-16%2011_26_06-Window1313×511 259 KB

Anyway for example last string:

Could not find file /shared/tmp/attachments//2/2485-c3da9f6ee1e33d3ebf418c5dea1f9e8a.data. Skipping attachment id 2485

…it IS on that path, but there is no .data. instend data there is .JPG

Any help is welcome.

2019-11-15T23:00:00Z

Script has a bug I’ve find it and fix it now i manage to import attachments.