Importing from phpBB3

import

(pushrbx) #194

Hi all.
I was trying to import from phpbb 3.0.2 and I got an error message during the import. Please see the output in the following link:
https://pastebin.com/Lbzd8z13

What am I doing wrong?


(Gerhard Schlager) #195

That’s caused by a recent change in core:

@techAPJ I think 20000 is quite a low value for max_image_size_kb and max_attachment_size_kb. That’s not even 20MB.

@pushrbx You can either wait until this issue is fixed or use the beta channel for the import. You can change that in app.yml.


(Arpit Jalan) #196

I agree. Just pushed a fix that increases max value for max_image_size_kb to 100MB & max_attachment_size_kb to 1GB.

Thanks for bringing this in my notice!


(pushrbx) #197

Thanks. I’ll try it with beta channel then. :slight_smile:


(Gerhard Schlager) #198

No need to use the beta channel. It’s already fixed :wink:


(pushrbx) #199

Oh, that’s great. :smile:


(pushrbx) #200

I’ve tried again, and it didn’t work. :confused:
I’ve removed the docker image, in the launcher script I’ve changed the docker discourse image version from 1.3.10 to 1.4.0 (I guess this is the latest on docker hub), I’ve run again the bootstrap for the import container, and when I try to run the import script for phpbb it fails with the same message. I tried to change the “version” config item in the import.yml file, but that one didn’t cure the issue either.
How do I get the fixed version to run in the docker image? Or should I wait for the update?
Here is my output again:
https://pastebin.com/DPGzxQrZ


(Gerhard Schlager) #201

Unfortunately changing the max_image_size_kb to 100MB and max_attachment_size_kb to 1GB wasn’t enough. The phpBB3 importer knows only a max attachment size. I just submitted a PR that should fix this problem once and for all.


(pushrbx) #202

I see now what that error message means. I have no experience so far with discourse, I just want to see how it looks with data from my phpbb forum.

Thanks again for the quick resolution.

Out of curiosity: what was the reason behind the validation of attachment file sizes?


(Opus K) #203

Thanks for your quick resolution.

I have confirmed that the size error is resolved by using the updated link below :slight_smile:

Thanks again for your quick support.


#204

Hi there,

my discourse installation is using a postgresql-database installed on a remote server. Is there any way to configure these remote settings (serveradress, username, password etc.) in the phpbb3 importer? I looked around but I didn’t find the right place.


(Gerhard Schlager) #205

I’m sorry, but PostgreSQL isn’t supported at the moment. Only MySQL and MariaDB.


#206

Maybe I wasn’t quit clear; my discourse-db runs on a remote server (PostgreSQL) outside the docker container. My phpbb3 runs on a MySQL-Server. Is there any way to configure these settings?


(Gerhard Schlager) #207

Ah, sorry, I should have read your post more carefully. :blush:

I don’t think there’s anything special you need to do. If it works with the normal Docker container than it should work with the one used for importing as well. It’s just a copy of the Docker config file with a few adjustments.

Just make sure you copy your working Docker configuration file(s) and add the phpbb3.template.yml template as is described in step 1.2


(Todd Taylor) #208

Ok I’m really stuck here now…I’ve gotten to finally getting the importer to run, but I’m getting this error:

importing from phpBB 3.1.10

creating users
Skipping 1000 already imported users
     2000 / 4031 ( 49.6%)  Skipping 1000 already imported users
     4031 / 4031 (100.0%)  
creating anonymous users
Skipping 35 already imported users

creating categories
       26 / 26 (100.0%)  
creating topics and posts
Skipping 1000 already imported posts
Skipping 1000 already imported posts
Skipping 1000 already imported posts
Skipping 1000 already imported posts
Skipping 1000 already imported posts
     5061 / 36836 ( 13.7%)  [1060160 items/min]  /usr/local/lib/ruby/2.3.0/cgi/util.rb:79:in `chr': invalid codepoint 0xDBC0 in UTF-8 (RangeError)
	from /usr/local/lib/ruby/2.3.0/cgi/util.rb:79:in `block in unescapeHTML'
	from /usr/local/lib/ruby/2.3.0/cgi/util.rb:66:in `gsub'
	from /usr/local/lib/ruby/2.3.0/cgi/util.rb:66:in `unescapeHTML'
	from /var/www/discourse/script/import_scripts/phpbb3/support/text_processor.rb:19:in `process_raw_text'
	from /var/www/discourse/script/import_scripts/phpbb3/support/text_processor.rb:33:in `process_post'
	from /var/www/discourse/script/import_scripts/phpbb3/importers/post_importer.rb:33:in `map_post'
	from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:138:in `block (2 levels) in import_posts'
	from /var/www/discourse/script/import_scripts/base.rb:432:in `block in create_posts'
	from /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-mini-profiler-0.10.2/lib/patches/db/mysql2.rb:6:in `each'
	from /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-mini-profiler-0.10.2/lib/patches/db/mysql2.rb:6:in `each'
	from /var/www/discourse/script/import_scripts/base.rb:431:in `create_posts'
	from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:137:in `block in import_posts'
	from /var/www/discourse/script/import_scripts/base.rb:784:in `block in batches'
	from /var/www/discourse/script/import_scripts/base.rb:783:in `loop'
	from /var/www/discourse/script/import_scripts/base.rb:783:in `batches'
	from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:187:in `batches'
	from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:131:in `import_posts'
	from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:31:in `execute'
	from /var/www/discourse/script/import_scripts/base.rb:45:in `perform'
	from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:20:in `perform'
	from script/import_scripts/phpbb3.rb:31:in `<module:PhpBB3>'
	from script/import_scripts/phpbb3.rb:12:in `<module:ImportScripts>'
	from script/import_scripts/phpbb3.rb:11:in `<main>'

Any idea how to get around this? adding a debug output and deleting the post? Please help :frowning:


(Gerhard Schlager) #209

What’s the charset used by your MySQL database / tables?
You could try adding the encoding parameter to the database connection if it isn’t UTF-8. E.g. add something like :encoding => 'utf8mb4' to the following function.

If the encoding is correct or changing it doesn’t help I’d add an exception handler to the map_post function and output the post’s id: puts row[:post_id]
Then you’ll be able to fix or the delete the offending post in the database.


(Todd Taylor) #210

I’ve changed the encoding, both through phpmyadmin, and using a script, but neither solved the problem. So looks like I’ll need to edit the map_post function.

Forgive me though, which files would I need to edit to change the script? I’m in the dark with docker


(Gerhard Schlager) #211

Changing encoding in the database isn’t always the best idea… If the original database (before any encoding changes with phpmyadmin or scripts) wasn’t utf8 I’d still try adding the encoding parameter first. Of course with the unchanged database.

You’ll find the import script and all it’s classes at /var/www/discourse/script/import_scripts/phpbb3 within the docker container.


(Todd Taylor) #212

setting the encoding parameter didn’t seem to help anything, but spitting out the post ID’s is working just fine :slight_smile: Thanks you for the assistance!


(Todd Taylor) #213

So my import has completed, and I stopped the import container, and started up the app container. Now I’m getting a 502 gateway error when trying to view the forum through HTML. Is this just sidekiq finishing up the post-processing, or did I mess something up?