Hello, me and my group of crazy idiots are really close to finally migrate our Vbulletin3 forum into Discourse after having wrote an ad-hoc script that finally manages to migrate all the 21 millions replies from the original database into discourse. Now, we have the problem of links to the topics/replies written in the replies themselves. In the migration we have written, we write a mapping of the “old” topic and post Ids and to what they map into in discourse. For example: id | topic_id | name | value | created_at | updated_at --------+----------+-----------+--------+----------------------------+---------------------------- 581727 | 581736 | import_id | 599137 | 2023-02-08 16:30:01.600759 | 2023-02-08 16:30:01.600759 What I was thinking now is a plugin that simply intercept links to the old forum format and transform them with reference to the new thread/reply. So for example, something like: https://oldforum.something.com/showthread.php?t=123456 Will trigger a query search using the topics_custom_field for the value 123456, find the discourse topic_id, then query the topic_links table with that id and find the url. Finally replace it in the post on the client side (assuming js to manipulate the content). Something similar for posts. However, I can’t find any good example of how to even start creating something like that for discourse. Can someone give me some hints, example or plugins that would do something similar (check replies for some substring and replace it, query the API? DB? for one value to retrieve another?). Thank you

This already exists in core, it is called Permalinks, and the existing VB4 importer has code for it script/import_scripts/vbulletin.rb main topics.each do |thread| topic_id = "thread-#{thread["threadid"]}" topic = topic_lookup_from_imported_post_id(topic_id) if topic.present? url_slug = "thread/#{thread["threadid"]}" if thread["title"].present? if url_slug.present? && topic[:topic_id].present? Permalink.create(url: url_slug, topic_id: topic[:topic_id].to_i) end end end end end def import_posts You should enter something like /showthread\.php\?(\d*)/thread/\1 into the permalink_normalizations setting.

Just to confirm, I should run this logic after the migration has completed right? So it goes through all the replies again and changes the permalinks

[image] Crius: changes How do you mean, changes? Do you already have permalinks?

When we migrate the content of a reply with, for example: https://oldforum.something.com/showthread.php?t=123456 doesn’t know what id that topic will have on discourse… no?

it will if you use the above code to create permalinks. showthread.php?t= refers to a topic/thread and not to a reply btw

[image] Richard - Communiteq: showthread.php?t= refers to a topic/thread and not to a reply btw I was just using that link as an example :slight_smile: Unfortunately we cannot use that code because the import takes ages to import 20 million posts and the bulk import simply doesn’t work. There are missing pieces. That’s why we had to write our own migration script. It does it all (pm, users, usergroups, categories, topic, replies) in about 6 hours with a 4 cores, 8gb ram but we noticed that we were missing the permalinks :slight_smile:

We discussed internally and will simply do a second pass when all replies have been migrated. Thanks for bouncing back ideas with me Richard :heart:

Did your script create import_ids? If so, even if your didn’t create the permalinks, you can fairly quickly process those to create them.

Hey, yes Jay, we do. [image] Crius: In the migration we have written, we write a mapping of the “old” topic and post Ids and to what they map into in discourse. For example: id | topic_id | name | value | created_at | updated_at --------+----------+-----------+--------+----------------------------+---------------------------- 581727 | 581736 | import_id | 599137 | 2023-02-08 16:30:01.600759 | 2023-02-08 16:30:01.600759 We were trying to avoid cycling the whole 20 million + replies again but realised that alternatives solutions (plugin, nginx redirect, etc) would be quite convoluted or rely on external factors that would make it a half-assed solution so, we will simply cycle the replies again and process the permalinks. It will add some time to the migration but hopefully not as much. Everything else is “cooked” on the fly as we know what “raw” need to be converted into html. For the permalinks we cannot do that as if a permalink is added with an edit, it could reference a topic that has not yet been processed (higher topic Id) and those not being found in the topics_custom_field table at the time it is being processed.

Plugin to help mapping pre-migration threads after migration

Development

RGJ (Richard - Communiteq) February 8, 2023, 8:27pm 8

Maybe you can consider the pre-permalinks nginx map solution? Redirect vBulletin URLs to Discourse URLs

Topic		Replies	Views
Migration from FluxBB while preserving incoming links Migration fluxbb	8	1607	November 14, 2022
Bulk importing Redirects to New Discourse Topics Migration	2	335	April 12, 2024
Remapping old imported forum permalinks to posts and topics? Development	7	1591	October 31, 2016
Problem with permalinks after migration Migration	8	807	January 17, 2021
Redirect old forum URLs to new Discourse URLs using permalinks Migrating to Discourse how-to	10	30302	September 13, 2024

Plugin to help mapping pre-migration threads after migration

Related topics