phpBB 3 Importer (old)

(Gerhard Schlager) #21

Thanks. I had to change it a little bit since the bundle install as user discourse resulted in a permission denied error. But the following works:

apt-get update && apt-get install libmysqlclient-dev -y
git clone https://github.com/nlalonde/ruby-bbcode-to-md.git /tmp/ruby-bbcode-to-md
cd /tmp/ruby-bbcode-to-md
gem build ruby-bbcode-to-md.gemspec
gem install ruby-bbcode-to-md-0.0.13.gem

su - discourse
echo "gem 'mysql2'" >> /var/www/discourse/Gemfile
echo "gem 'ruby-bbcode-to-md', path: '/tmp/ruby-bbcode-to-md'" >> /var/www/discourse/Gemfile
exit

cd /var/www/discourse
bundle install --no-deployment --path vendor/bundle

su - discourse
cd /var/www/discourse
RAILS_ENV=production bundle exec ruby script/import_scripts/phpbb3.rb bbcode-to-md
4 Likes
Importing from phpBB3
Importing from phpBB3
(Jeff Vienneau) #22

Thanks for all the excellent work here. I have just imported my phpBB forum to discourse.

I am seeing an issue with attachments.

When the jpg is added to the post as an attachment and inserted using the [attachment ] tag, the result is simply the tag:
[attachment=0]IMG_2911a.JPG[/attachment]

Has anyone encountered this?
Does the attachment get pulled over to discourse anywhere, so I could fix this manually?

thanks

(Gerhard Schlager) #23

The current version of the importer can’t import attachments. I’m going to implement this in the next few weeks unless someone else does it earlier…

2 Likes
(Jeff Vienneau) #24

Thanks for the info, Gerhard.

If you run the import a second time, what happens? I have users on the system already making new posts.

(Gerhard Schlager) #25

The importer detects the already imported topics/posts and does nothing. I’m not sure yet, but I think the import function I’m planning to implement wont be able to update already imported posts and add those attachments…

(Jeff Vienneau) #26

Is there a way to clear the imported posts? And still keep the posts that were used to load images for logo etc…

(Jens Maier) #27

Not automatically. But you can do this in the Rails console, if the converter script has set some custom field on imported posts. This should be the case for most current importer scripts.

system_user = Discourse.system_user
PostCustomField.joins(:post).includes(:post).where(name: 'import_id').map {|pcf| pcf.post.topic_id }.sort.uniq.each do |topic_id|
  PostDestroyer.new(system_user, Topic.find(topic_id).ordered_posts.first).destroy
end
3 Likes
(Jeff Vienneau) #28

Does anyone know if the phpBB script sets such a custom field?

(Gerhard Schlager) #29

Yes, it does set the custom field. Actually all the official importers set that field.

1 Like
(Jeff Vienneau) #30

Excellent.

I think I’ll give these scripts a good look over. I’m new to ruby but it looks neat.

(Jeff Vienneau) #31

Looking at the code, trying to understand the ruby…

I see that there is a method “create_upload” in base but that does not seem to be called anywhere in base or in phpBB script yet some posts have uploads that were pulled across. What am I missing?

Note, when the script rendered the links to these uploads, the IP address was used and not my host name. Is this a script issue or configuration on my part?

(Neil Lalonde) #32

Have you edited the variables at the top of the script?

  ORIGINAL_SITE_PREFIX = "oldsite.example.com/forums" # without http(s)://
  NEW_SITE_PREFIX      = "http://discourse.example.com"  # with http:// or https://

Also, as discussed earlier, that script isn’t handling phpbb uploads at all. I’m working on it now, but with a site that has already been imported. I’ll try to backport my work to that script.

@gerhard said he was going to work on it “in the next few weeks”, but I can’t wait. I’ll try to share what I come up with.

1 Like
(Jeff Vienneau) #33

Thanks @neil, I missed those variables.

That would be great. My site has no active users just an import from the phpBB site (with active users). I would be willing to test what you have.

(Jeff Vienneau) #34

When I try to reload, after running the destroy script (above), it will only import new posts. It also reports many posts as not having a parent post:

“Parent post pm:503 doesn’t exist. Skipping 506: …”

How does the imported keep track of where it left off? Do I need to reset something?

(Jens Maier) #35

Ouch. Discourse doesn’t do recursive destroys. Which means that your database now contains a bunch of orphaned posts.

[TopicAllowedUser, TopicAllowedGroup, TopicUser, TopicCustomField].each do |model|
  model.includes(:topic).select {|p| p.topic.nil? }.each(&:destroy)
end
system_user = Discourse.system_user
Post.includes(:topic).select {|p| p.topic.nil? }.each do |post|
  PostDestroyer.new(system_user, post).destroy
end

That should at least catch the most obnoxious garbage, but the remaining users may have wrong topic counters. At least for a while, until sidekiq updates the stats.

(Jeff Vienneau) #36

No worries. I’ll give that a try tonight. Thanks.

You should edit your post maybe to indicate the issue for others.

(Jens Maier) #37

Good point, done. :thumbsup:

(Jeff Vienneau) #38

Tested this since @elberet’s last edit and it does seem to clean the imported posts. Thanks @elberet!

(Jeff Vienneau) #39

Still getting parent post does not exist. It looked like it cleaned out everything. Still not enough though.

(Jens Maier) #40

Drat. If we assume that at least the topics table has been cleaned out, you could try to just flush the orphaned post custom fields, and live with a few orphan posts remaining in the database. Without those, the importer should not recognize anything as already imported.

PostCustomField.includes(:post => :topic).select {|f| f.post.topic.nil? }.each(&:destroy)