Importing from phpBB3

import

(Jeff Atwood) #23

I think there is a rake task to replace text in post bodies. Then run the rake task to rebake all posts.


(Adrian D'atri Guiran) #24

This is an excellent solution! thank you! any idea where i would find the “find and replace” rake task? is this part of core? i took a look in /lib/tasks but i’m not seeing anything about find and replace.

Is it one of these?

adrian@li1286-134:~/discourse$ ls -1 lib/tasks/
add_topic_to_quotes.rake
admin.rake
api.rake
assets.rake
auto_annotate_models.rake
autospec.rake
avatars.rake
backfill.thor
build.rake
build_test_topic.rake
cdn.rake
db.rake
docker.rake
emoji.rake
highlight.rake
i18n_stats.rake
images.rake
integration.rake
plugin.rake
populate.thor
posts.rake
qunit.rake
release_note.rake
rspec.rake
scheduler.rake
search.rake
smoke_test.rake
typepad.thor
uploads.rake
user_actions.rake

(Erlend Sogge Heggen) #25

As an alternative to the rake task there’s also this:


(Adrian D'atri Guiran) #26

Interesting approach, i gave it a shot. It took me a while to figure out that i was crushing my API with requests and hitting rate limits. After editing some nginx config to temporarily remove the rate limits its running again now. But i don’t think it will be done anytime in 2017 at this rate. I have waaaaay too many posts to be doing this abstracted through a web API.

I would be interested in the find and replace rake task if anyone can point me in the right direction?


[Announce] Search & replace / batch process Discourse posts
(Kane York) #27

I think this would work:

root@container# discourse remap [soundcloudset] 'temp_soundcloudreplace '

or

discourse@container$ script/discourse remap ...

Using temp_soundcloudreplace so that you can turn it back if there’s a mistake.


(Axel Naumann) #28

Hi,

I ran into an issue with migratepassword and my phpBB3 import. I got the database fields I expected to see from the hash import into an ubuntu development installation - but it somehow didn’t work: many people couldn’t log in. This was the silver bullet:

$ cd /var/discourse
$ sudo ./launcher enter app
$ su - postgres
$ psql discourse
discourse=# update "user_custom_fields" SET "value" = REPLACE("user_custom_fields"."value", '$2y$', '$2a$') WHERE left("user_custom_fields"."value", 4) = '$2y$';

The docker’ed discourse was unable to validate against 2y hashes. With that transformation applied everyone was able to log in using the old phpBB3 passwords!

Cheers, Axel.


(Gerhard Schlager) #29

Thanks @AxelN for that information. I’ll look into it.


(Adrian D'atri Guiran) #30

If your forum was like mine, with 1000s of topics and you were using PHPBB SEO mod. and you want a simple solution to doing redirects from old post urls to new urls, take a look at this method:


(@SenpaiMass) #31

how do i import on a docker based setup? and where do place the SQL file which is 2 gb approximately.


(Jeff Wong) #32

These outline the steps I did for docker, though there isn’t a one-click easy method currently – it’s more of a workaround. The nice thing about the docker method is once you have everything installed, you can rebuild without all the import stuff on it, and you have a fresh version of discourse, with no mysql dependencies installed.

Most of the information was taken from the old importer link above:


#33

Hi! Very nice to see a phpBB3 migration topic here, sadly I have hit a wall trying to import my phpBB3 to Discourse.
So, my plan was to setup Ubuntu 15.10 development environment with mysql server, import the phpbb3 sql and then use the import script to get everything from phpbb3 to discourse, if everything looked well, I would backup discourse and restore it on my production server.
Everything has been going smooth until the import itself:

roman@discourse:~/discourse/script/import_scripts$ ruby phpbb3.rb phpbb3/settings.yml
loading existing groups...
loading existing users...
loading existing categories...
loading existing posts...
loading existing topics...
importing from phpBB 3.0.13-PL1
creating users
        4 / 26257 (  0.0%)  /home/roman/discourse/script/import_scripts/phpbb3/importers/avatar_importer.rb:27:in `import_avatar': undefined method `persisted?' for nil:NilClass (NoMethodError)
	from /home/roman/discourse/script/import_scripts/phpbb3/importers/user_importer.rb:40:in `block in map_user'
	from /home/roman/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.5/lib/active_support/core_ext/object/try.rb:77:in `call'
	from /home/roman/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.5/lib/active_support/core_ext/object/try.rb:77:in `public_send'
	from /home/roman/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.5/lib/active_support/core_ext/object/try.rb:77:in `try!'
	from /home/roman/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.5/lib/active_support/core_ext/object/try.rb:63:in `try'
	from /home/roman/discourse/script/import_scripts/base.rb:333:in `create_user'
	from /home/roman/discourse/script/import_scripts/base.rb:240:in `block in create_users'
	from /home/roman/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rack-mini-profiler-0.9.8/lib/patches/db/mysql2.rb:6:in `each'
	from /home/roman/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rack-mini-profiler-0.9.8/lib/patches/db/mysql2.rb:6:in `each'
	from /home/roman/discourse/script/import_scripts/base.rb:228:in `create_users'
	from /home/roman/discourse/script/import_scripts/phpbb3/importer.rb:61:in `block in import_users'
	from /home/roman/discourse/script/import_scripts/base.rb:702:in `block in batches'
	from /home/roman/discourse/script/import_scripts/base.rb:701:in `loop'
	from /home/roman/discourse/script/import_scripts/base.rb:701:in `batches'
	from /home/roman/discourse/script/import_scripts/phpbb3/importer.rb:157:in `batches'
	from /home/roman/discourse/script/import_scripts/phpbb3/importer.rb:55:in `import_users'
	from /home/roman/discourse/script/import_scripts/phpbb3/importer.rb:28:in `execute'
	from /home/roman/discourse/script/import_scripts/base.rb:45:in `perform'
	from /home/roman/discourse/script/import_scripts/phpbb3/importer.rb:20:in `perform'
	from phpbb3.rb:31:in `<module:PhpBB3>'
	from phpbb3.rb:12:in `<module:ImportScripts>'
	from phpbb3.rb:11:in `<main>'

Any help would be appreciated!


(Gerhard Schlager) #34

Make sure that you configured a valid phpbb_base_dir in settings.yml. The folder should contain the folders files (for attachments) and images (for smilies and avatars) from your phpBB3 installation.

Disable importing of avatars and/or attachments in settings.yml if you don’t have those files or you don’t want to import them.


#35

Thanks for the reply!
The base dir is correct, I now disabled importing avatars and it got much better, but crashed here:

29476 / 224041 ( 13.2%) [644316 items/min] Failed to upload /home/roman/xxx/htdocs/images/smilies/icon_e_geek.gif
29667 / 224041 ( 13.2%) [629240 items/min] /home/roman/discourse/script/import_scripts/phpbb3/importers/poll_importer.rb:90:in []=': no implicit conversion of String into Integer (TypeError) from /home/roman/discourse/script/import_scripts/phpbb3/importers/poll_importer.rb:90:inupdate_poll’
from /home/roman/discourse/script/import_scripts/phpbb3/importers/poll_importer.rb:25:in map_poll' from /home/roman/discourse/script/import_scripts/phpbb3/importers/post_importer.rb:75:inadd_poll’
from /home/roman/discourse/script/import_scripts/phpbb3/importers/post_importer.rb:55:in map_first_post' from /home/roman/discourse/script/import_scripts/phpbb3/importers/post_importer.rb:35:inmap_post’
from /home/roman/discourse/script/import_scripts/phpbb3/importer.rb:106:in block (2 levels) in import_posts' from /home/roman/discourse/script/import_scripts/base.rb:421:inblock in create_posts’
from /home/roman/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rack-mini-profiler-0.9.8/lib/patches/db/mysql2.rb:6:in each' from /home/roman/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rack-mini-profiler-0.9.8/lib/patches/db/mysql2.rb:6:ineach’
from /home/roman/discourse/script/import_scripts/base.rb:420:in create_posts' from /home/roman/discourse/script/import_scripts/phpbb3/importer.rb:105:inblock in import_posts’
from /home/roman/discourse/script/import_scripts/base.rb:702:in block in batches' from /home/roman/discourse/script/import_scripts/base.rb:701:inloop’
from /home/roman/discourse/script/import_scripts/base.rb:701:in batches' from /home/roman/discourse/script/import_scripts/phpbb3/importer.rb:157:inbatches’
from /home/roman/discourse/script/import_scripts/phpbb3/importer.rb:99:in import_posts' from /home/roman/discourse/script/import_scripts/phpbb3/importer.rb:31:inexecute’
from /home/roman/discourse/script/import_scripts/base.rb:45:in perform' from /home/roman/discourse/script/import_scripts/phpbb3/importer.rb:20:inperform’
from phpbb3.rb:31:in <module:PhpBB3>' from phpbb3.rb:12:inmodule:ImportScripts
from phpbb3.rb:11:in `'
roman@discourse:~/discourse/script/import_scripts$

Here is my settings.yml:

# This is an example settings file for the phpBB3 importer.
database:
  type: MySQL # currently only MySQL is supported - more to come soon
  host: localhost
  username: root
  password:
  schema: d30638sd43883
  table_prefix: phpbb # Usually all table names start with phpbb. Change this, if your forum is using a different prefix.
  batch_size: 1000 # Don't change this unless you know what you're doing. The default (1000) should work just fine.
import:
  # Enable this option if you want to have a better conversion of BBCodes to Markdown.
  # WARNING: This can slow down your import.
  use_bbcode_to_md: false
  # This is the path to the root directory of your current phpBB installation (or a copy of it).
  # The importer expects to find the /files and /images directories within the base directory.
  # This is only needed if you want to import avatars, attachments or custom smilies.
  phpbb_base_dir: /home/roman/xxx/htdocs/
  site_prefix:
    # this is needed for rewriting internal links in posts
    original: xxx.xx    # without http(s)://
    new: http://xxx.xxx.xx       # with http:// or https://
  avatars:
    uploaded: false  # import uploaded avatars
    gallery: false   # import the predefined avatars phpBB offers
    remote: false   # WARNING: This can considerably slow down your import. It will try to download remote avatars.
  # When true: Anonymous users are imported as suspended users. They can't login and have no email address.
  # When false: The system user will be used for all anonymous users.
  anonymous_users: true
  # Enable this, if you want import password hashes in order to use the "migratepassword" plugin.
  # This will allow users to login with their current password.
  # The plugin is available at: https://github.com/discoursehosting/discourse-migratepassword
  passwords: true
  # By default all the following things get imported. You can disable them by setting them to false.
  bookmarks: true
  attachments: true
  private_messages: true
  polls: true
  # This tries to fix Private Messages that were imported from phpBB2 to phpBB3.
  # You should enable this option if you see duplicate messages or lots of related
  # messages as topics with just one post (e.g. 'Importer', 'Re: Importer', 'Re: Importer'
  # should be one topic named 'Importer' and consist of 3 posts).
  fix_private_messages: false
  # When true: each imported user will have the original username from phpBB as its name
  # When false: the name of each user will be blank
  username_as_name: false
  # Map Emojis to smilies used in phpBB. Most of the default smilies already have a mapping, but you can override
  # the mappings here, if you don't like some of them.
  # The mapping syntax is: emoji_name: 'smiley_in_phpbb'
  # Or map multiple smilies to one Emoji: emoji_name: ['smiley1', 'smiley2']
  emojis:
    # here are two example mappings...
    smiley: [':D', ':-D', ':grin:']
    heart: ':love:'

Any help would be appreciated!


(Jeff Wong) #36

It seems to be choking taking a closer look is your poll importers.


(Gerhard Schlager) #37

The value of phpbb_base_dir seems to be wrong or doesn’t contain the files and images directories. That’s why you can’t import avatars, attachments and custom smilies.

That’s just a warning which can be ignored. The real error is this:

poll_importer.rb:90:in []=': no implicit conversion of String into Integer (TypeError)

Looks like Discourse fails to parse a poll in the post and subsequently the importer fails to update the vote count. I’ve contacted @Neso in order to find the post that’s causing this.


(Jeff Wong) #38

Ah, you’re right, sorry, didn’t finish my edit – the error that it’s choking on has to do with the importing of polls, for some reason.

The issue is throwing from this query:

Not sure what’s wrong with that query, to be honest. you can try to query with topic ID = 29667 and see if anything suspicious is in that specific poll. Without having access to the specific DB, it’s difficult to debug from this end, but that’s where I’d start.


(Gerhard Schlager) #39

I don’t believe that the query is causing the error. The problem is that it’s not possible to extract the default poll.


I guess I could add some error handling, but I’d really like to know what’s causing this…

Unfortunately that’s not always the topic ID. It’s just the number of posts that have been imported.

OK, I sent a PR to improve the error handling for importing files and polls:


#40

Somehow after yet another clean install of Ubuntu 15.10 I was able to get everything imported (had to turn off use_bbcode_to_md but everything else was set as default).
P.S: Is there a way to disable sending emails from the cli? I think I forgot to disable emails from the admin panel before the import and I don’t want to accidentally send out 27k emails :confounded: when I start discourse.


(Gerhard Schlager) #41

You don’t have to. Sending of emails is disabled during the import and Discourse shouldn’t start to send emails right away. If I recall correctly it takes a few days until it starts sending digest mails and those are the only ones that are sent without user interaction.


(@SenpaiMass) #42

Hope we get this done on a docker based setup to import from phpBB3, would be easy for the beginners. :slightly_smiling: