We have 26gb of data dump from discourse 1.3million users 3million topics 21million posts our problem is we are importing at 500k/min but after a few minutes it decreases up to 2k/min

You’ll need lots of ram. You might look at the bulk importers, but I don’t believe that there is one for vanilla.

hi Jay. We are using c5.4xlarge instance with AWS and at first it is at 500k/min and slows down after a few minutes.

The import script is restartable, but this is unfortunately just normal with the import scripts.

yup when i restart it just skips the data already imported but the same process decreases overtime :frowning:

thanks for confirming this. :frowning: total of 31million of data will take a month or so if it keeps on decreasing. Any suggestion for this to improve? or it is just the way it is?

You need a CPU with fast single core speed which is quite hard to find in the cloud. Or give the bulk import script a try. Importers for large forums There is one for Vanilla: discourse/script/bulk_import/vanilla.rb at main · discourse/discourse · GitHub

we use c5.4xlarge from AWS vCPU - 16 Memory (GiB) - 32 is this enough or should we upgrade? sure, will try that bulk import script. Thanks!

You will need a CPU from the top of PassMark CPU Benchmarks - Single Thread Performance if you want to run the regular import script as fast as possible. I have no idea what you get on AWS or any other cloud provider with vCPUs. :man_shrugging:

You want to use the bulk importer.

[Screen Shot 2020-11-06 at 2.42.12 AM] whenever i tried the bulk import it stops there. Since the traceback stops at category ids i tried changing the -1 to 0 @last_imported_category_id = imported_category_ids.max || -1 to @last_imported_category_id = imported_category_ids.max || 0 I even tri…

Vanilla에서 Discourse로 대규모 데이터 가져오기 (속도 저하)

제품 지원

pfaffman (Jay Pfaffman) 11월 5, 2020, 12:56오후 2

RAM이 많이 필요합니다. 벌크 임포터를 살펴보시는 것도 좋을 것 같지만, 바닐라용은 없는 것으로 알고 있습니다.

글		댓글	조회수
Migrating a large forum Product Support	10	1833	12월 7, 2018
Importers for large forums Announcements	49	10074	5월 5, 2018
Import from vbulletin to discourse forums Development	5	2344	8월 28, 2018
Migrate a Vanilla forum to Discourse Sysadmins how-to	44	16907	1월 30, 2023
Bypass UploadCreator for Import Product Support	8	622	1월 9, 2021

Vanilla에서 Discourse로 대규모 데이터 가져오기 (속도 저하)

Related topics