I’ve successfully imported my vBulletin 5 forum into Discourse. While the overall import script works fine, I’m encountering errors when importing attachments. The attachments are stored in my database and include the following extensions: java, html, jpg, png, txt, rtf, zip, js, and xml. Upon debugging the import_attachment action, I discovered that only attachments with the java extension are being imported correctly. The script fails for attachments with other extensions. Has anyone else in the community faced issues when importing attachments with these file extensions? Does anyone have insights on why the script might be failing with these particular file types? Here’s a brief overview of the issue: The first three files in my database have the java extension and are imported without problems. The script fails when it encounters a file with the jpg extension. [image] @pfaffman Any advice or solutions would be greatly appreciated!

I added debugging statement on upl_obj begin upl_obj = create_upload(post.user.id, filename, real_filename) if upl_obj&.persisted? html = html_for_upload(upl_obj, real_filename) if !post.raw[html] post.raw += "\n\n#{html}\n\n" post.save! UploadReference.ensure_exist!(upload_ids: [upl_obj.id], target: post) end else puts "Failed to create upload for #{filename}: #{upl_obj.errors.full_messages.join(", ")}" next end rescue => e puts "Error processing file #{filename}: #{e.message}" next end and see these errors [image] these errors are on extension jpg, jpeg, png, PNG, gif any idea about these errors? @pfaffman

The only files that worked are ascii. My guess is an encoding error.

Is there an issue with files in database or import script encoding issue? @pfaffman

[image] See these files if others are encoded then why these extensions e.g jpg, png etc are not?

My guess, which could be wrong, is that a problem with newline encoding makes the data in the binary files wrong because a newline character is encoded as data. If the only files that work are ascii, it’s a good bet. So it’s not a Discourse issue, but a mysql issue.

[image] Jay Pfaffman: that a problem with newline encoding makes the data in the binary files wrong Almost - not newline encoding, but treated as text, and therefore corrupted. EF BF BD is the UTF-8 byte sequence for ‘REPLACEMENT CHARACTER’ (U+FFFD). This is indicative of a file being treated as text instead of binary. A JPEG image starts with ff d8 ff e0 xx xx 4a 46 49 46 00 You can see that the first four bytes have each been replaced with EF BF BD. So your images are indeed corrupted. This is not a problem with the importer, this is a problem with the database, as @pfaffman already said. If you have copied this database from another server, you might want to check if this is already an issue in the original database. This could also only be happening on the oldest images (if this happened a long time ago). Just remove the exit line and see what happens.

@RGJ Thanks for the help. I tried importing a new database with the correct images, and while it imported, not all attachments were fully imported. I’m encountering errors like this: [image] Any idea why this is happening? @RGJ @pfaffman

Posts are appearing like this, and there are many instances: [image]

I don’t think the importer can handle those. If I recall correctly, all [ATTACH] tags are removed from the posts since they are superfluous. That probably doesn’t work here because it does not expect JSON data in it. It would be matter of looking up the place where they are being removed and modifying that code to account for the JSON data inside of the tag.

Migrating vBulletin 5 database - Import attachments errors about attachment extension

Product Support Migration

pfaffman (Jay Pfaffman) July 6, 2024, 6:34pm 2

You’ll need to add some debugging put statements to see what it’s happening.

Topic		Replies	Views
vBulletin 5 import error (invalid character in website field) and a quick question about attachments Migration vbulletin5	3	845	August 17, 2020
vBulletin import script is incompatible with vBulletin5 Feature	11	2861	June 16, 2016
Import scripts vbulletin5.rb Self-hosting	9	1386	April 20, 2020
Migrating vB5 to Discourse Product Support	3	2562	May 16, 2015
Bulk file uploads (for migration) Self-hosting	2	1191	May 8, 2015

Migrating vBulletin 5 database - Import attachments errors about attachment extension

Related topics