phpBB 3 Importer (old)

Feature
21

Thanks. I had to change it a little bit since the bundle install as user discourse resulted in a permission denied error. But the following works:

apt-get update && apt-get install libmysqlclient-dev -y
git clone https://github.com/nlalonde/ruby-bbcode-to-md.git /tmp/ruby-bbcode-to-md
cd /tmp/ruby-bbcode-to-md
gem build ruby-bbcode-to-md.gemspec
gem install ruby-bbcode-to-md-0.0.13.gem

su - discourse
echo "gem 'mysql2'" >> /var/www/discourse/Gemfile
echo "gem 'ruby-bbcode-to-md', path: '/tmp/ruby-bbcode-to-md'" >> /var/www/discourse/Gemfile
exit

cd /var/www/discourse
bundle install --no-deployment --path vendor/bundle

su - discourse
cd /var/www/discourse
RAILS_ENV=production bundle exec ruby script/import_scripts/phpbb3.rb bbcode-to-md
4 Likes
22

Thanks for all the excellent work here. I have just imported my phpBB forum to discourse.

I am seeing an issue with attachments.

When the jpg is added to the post as an attachment and inserted using the [attachment ] tag, the result is simply the tag:
[attachment=0]IMG_2911a.JPG[/attachment]

Has anyone encountered this?
Does the attachment get pulled over to discourse anywhere, so I could fix this manually?

thanks

23

The current version of the importer canā€™t import attachments. Iā€™m going to implement this in the next few weeks unless someone else does it earlierā€¦

2 Likes
24

Thanks for the info, Gerhard.

If you run the import a second time, what happens? I have users on the system already making new posts.

25

The importer detects the already imported topics/posts and does nothing. Iā€™m not sure yet, but I think the import function Iā€™m planning to implement wont be able to update already imported posts and add those attachmentsā€¦

26

Is there a way to clear the imported posts? And still keep the posts that were used to load images for logo etcā€¦

27

Not automatically. But you can do this in the Rails console, if the converter script has set some custom field on imported posts. This should be the case for most current importer scripts.

system_user = Discourse.system_user
PostCustomField.joins(:post).includes(:post).where(name: 'import_id').map {|pcf| pcf.post.topic_id }.sort.uniq.each do |topic_id|
  PostDestroyer.new(system_user, Topic.find(topic_id).ordered_posts.first).destroy
end
3 Likes
28

Does anyone know if the phpBB script sets such a custom field?

29

Yes, it does set the custom field. Actually all the official importers set that field.

1 Like
30

Excellent.

I think Iā€™ll give these scripts a good look over. Iā€™m new to ruby but it looks neat.

31

Looking at the code, trying to understand the rubyā€¦

I see that there is a method ā€œcreate_uploadā€ in base but that does not seem to be called anywhere in base or in phpBB script yet some posts have uploads that were pulled across. What am I missing?

Note, when the script rendered the links to these uploads, the IP address was used and not my host name. Is this a script issue or configuration on my part?

32

Have you edited the variables at the top of the script?

  ORIGINAL_SITE_PREFIX = "oldsite.example.com/forums" # without http(s)://
  NEW_SITE_PREFIX      = "http://discourse.example.com"  # with http:// or https://

Also, as discussed earlier, that script isnā€™t handling phpbb uploads at all. Iā€™m working on it now, but with a site that has already been imported. Iā€™ll try to backport my work to that script.

@gerhard said he was going to work on it ā€œin the next few weeksā€, but I canā€™t wait. Iā€™ll try to share what I come up with.

1 Like
33

Thanks @neil, I missed those variables.

That would be great. My site has no active users just an import from the phpBB site (with active users). I would be willing to test what you have.

34

When I try to reload, after running the destroy script (above), it will only import new posts. It also reports many posts as not having a parent post:

ā€œParent post pm:503 doesnā€™t exist. Skipping 506: ā€¦ā€

How does the imported keep track of where it left off? Do I need to reset something?

35

Ouch. Discourse doesnā€™t do recursive destroys. Which means that your database now contains a bunch of orphaned posts.

[TopicAllowedUser, TopicAllowedGroup, TopicUser, TopicCustomField].each do |model|
  model.includes(:topic).select {|p| p.topic.nil? }.each(&:destroy)
end
system_user = Discourse.system_user
Post.includes(:topic).select {|p| p.topic.nil? }.each do |post|
  PostDestroyer.new(system_user, post).destroy
end

That should at least catch the most obnoxious garbage, but the remaining users may have wrong topic counters. At least for a while, until sidekiq updates the stats.

36

No worries. Iā€™ll give that a try tonight. Thanks.

You should edit your post maybe to indicate the issue for others.

37

Good point, done. :thumbsup:

38

Tested this since @elberetā€™s last edit and it does seem to clean the imported posts. Thanks @elberet!

39

Still getting parent post does not exist. It looked like it cleaned out everything. Still not enough though.

40

Drat. If we assume that at least the topics table has been cleaned out, you could try to just flush the orphaned post custom fields, and live with a few orphan posts remaining in the database. Without those, the importer should not recognize anything as already imported.

PostCustomField.includes(:post => :topic).select {|f| f.post.topic.nil? }.each(&:destroy)