Yeah. If you’re dragging the database from across the internet, it’ll be slow going.
@pfaffman it doesn’t seem to matter how I do this it’s slow going Its run for 11 more hours now using the MariaDB included in the import container and has only imported an additional 11%. So its now been running a total of 41 hours and completed 56%. The CPU on the 2 GB, 2 core DO server is maxed out. Any other thoughts on why this would be so slow? I’ve rebooted the server several times through this process and it speeds up for a while then drops to a crawl again.
It’s really hard to say what’s causing it. From the top of my head:
You caught a really shitty droplet. That happens sometimes. Creating a new one might help. Also, using a CPU Optimized Droplet is highly recommended for imports.
Your posts contain lots of images and/or attachments. I think that might cause a slow down.
And I’m out of ideas for now…
I’d really recommend running the import on your local machine, assuming that your PC is reasonable fast. With docker it’s quite easy and the performance is likely to be a lot better than with cloud servers. If it isn’t than there’s definitely something weird going on.
I have a last minute question guys. My forum is about to be launched.
I reorganized my forum categories after an import and I merged a lot of them.
However I have changed my mind and I’d like to “de-merge” some of them.
I think I could do something like:
- Remove all the message from the category which a lot of messages from other categories were merged in
- Import again my phpbb dump, so as it supports incremential import, it should re-import only all the message that were deleted, each in their own category
Would that work? How do I delete my entire main category with all its topics and messages (5000+ topics)?
Deleting the topics doesn’t really delete them, so they won’t get re-imported.
I’m not quite clear what you’re trying to do, but I think your best bet is to just move those topics wherever you want them.
The incremental import doesn’t care what category the topics are in, so the new posts (has the forum remained active on the old site?) will be added to their respective topics.
Or maybe you want to just delete everything and run the whole import again.
Thank you for your quick reply.
On phpbb I had a category A and a category B.
After my import on Discourse, I had a category A, a category B, then I moved all topics from A and B to a new category C, and I deleted my categories A and B.
I want to re-create my categories A and B (which is done in seconds), and get back my C topics in their original A and B categories. I can’t do this manually by selecting each topic one by one on hundred topics.
I don’t want to do this on my whole forum, it’s “just” hundreds of topics on several thousand.
Uh… Even by some rails command?
Any risk removing posts from the database directly? The Discourse forum isn’t active yet, no user activity at all.
After having imported from phpbb, besides configuring stuff, no topics or replies were posted etc.
I also don’t want to do a full re-import because I spent so much time configuring stuff since my import.
@gerhard I cleaned up a lot of old irrelevant 2-years-ago posts in this topic. You may want to take a peek and see if anything was worth editing into the first post as additional info.
Here’s an idea. Leave the stuff in category C and set permissions so no one can see it, and/or delete them.
Then, in Rails, do
That will delete the custom fields that let the importer know that the posts have been imported and it’ll import them again.
And just what “stuff” is that? If it’s just site settings, you can save them with https://github.com/pfaffman/discourse-settings-uploader. If it’s theme stuff, then it’s not too hard to export it and reload it.
Woah, thank you so much, I’m going to try that!
I’d be able to like you more than once, I’ll do it.
Yes, it is site settings actually, plus some users permissions, changed usernames/emails, etc.
My import is fairly long, plus I had to do some custom commands in order to remove old bb tags (size and color), bulk move thousands of topics from categories to anothers and update the counts, etc…
Juste wondering about something. Does that mean that i’ll have about 5000 topics deleted but still in the database? What about the permalinks if I import again the topics with the same names?
Yes. They’ll be there. Yes, you’ll want to similarly delete the permalinks.
You could also delete the topics and posts from the rails console. If you want to delete all of them you could use a similar command. I’m not sure what other tables you might need to clean out.
I might give a try as it’s the simplest solution. Hoping that won’t break things…
Uh… I noticed something weird. My backups were about 300 MB before and I noticed that they are now more than 600 MB each.
I’m starting wondering if I didn’t remove some message through rails command a while ago and re-import them again or something like. I don’t remember doing such a thing, but I see no reason to have my backup size doubled… Is there any way to check the total number of messages (including deleted ones)? So I could compare with my phpbb forum number of messages.
edit: nevermind, I used data-explorer and I didn’t see any weird thing. Maybe it was because I had issues with imported images and I had to make a recovery from tombstone, which bringed back a lot of images (currently 300 MB of attached files).
Yeah I’ve created this topic to not be too off topic here.
Hi, I have another idea.
I guess that
import_id is equal to the phpbb post id?
Since I merged topics from category A and B in category C,
- Store all the phpbb topics id from topics that belong to my categories A and B in a json file, then
- Iterate the json, get the corresponding discourse topic on each iteration
- Update the discourse topic by changing their category from C to A or B (maybe via /topics/bulk) in each iteration
Would that do the trick on the paper?
edit : I post here since my message mentions a custom_field relative to the phpbb import, I hope I’m not too much off topic!
That part is definitely correct!
Something like your idea could definitely work if you have a mapping of the PHPbb post ID with the category you want.
Yeah, this is pretty far afield from anything to do with phpBB.
@codinghorror will probably want to split this off into its own topic.
Thank you, I’m going to work on this one.
And sorry about the off topic!
Edit : my script worked well, so here’s how I did, in case it would interest some people who’d like to “unmerge” a category after reorganizing their phpbb topics into Discourse:
- Installed Data Explorer Plugin
- Get the Discourse imported posts from the category of your choice with this query and download the result as a json:
SELECT topics.id as d_topic_id, slug, value as phpbb_topic_id FROM topics JOIN topic_custom_fields ON topic_id = topics.id AND name = 'import_topic_id' AND category_id = 5
3.Get the phpbb topic list from the category(ies) of your choice from phpmyadmin and export the result as a json:
SELECT topic_id FROM phpbb_topics JOIN phpbb_forums ON phpbb_topics.forum_id = phpbb_forums.forum_id AND phpbb_forums.forum_id = 20
Then you can loop iterate one of your json, and on each iteration if
phpbb_topic_id is also present in the second array, then you can move it to a new category of your choice.
To do so I used https://github.com/communiteq/discourse-api-php and added a method to update the category of a topic.
Any update on phpbb 3.2 support? There are thousands of phpbb forums out there and we’d love to move to Discourse if we can save the community. We have 2000 users and tired of looking like it is 2002. PHPBB is a huge pile of old outdated forum principles, extensions that are obsolete and not supported, sprinkle it with a lot of pain in the UI, etc. Everytime we go into the admin control panel, God kills a cat. And the only thing that is stopping us from moving over to something better is exporting all the existing content.
I’d love to work on it but I have no experience in Ruby and database.
If you can at least partially sponsor the work we can take it on.