Members problem on importing Mybb to Discourse

Migration
1

Hi,

I’ve installed development enviroment on my Ubuntu local machine. I did it importing steps, it’s work but there are some problems.

My Mybb forum has 9k user and 300k post. Posts are seems successfully imported. But many users not imported (Specially who using space character in the username like ‘User Name’) When i go Users section on Discourse panel, i’ve seen only 300 - 400 users. (No one has space character)

1- On the Discourse admin panel seems 9.8k users but on the users tab when i search don’t seem users who has space character in the username. These members topics are seems but like created by my account.

2- Banned users seems like normal users.

So what is your suggestions about this problem? Should i re-install my development enviroment and repeat importing steps? Or should i some edit on mybb.rb?

@pfaffman :slight_smile:

2

I’m nearly clueless about the importer, but one thing you could do is perhaps replace the spaces with underscores with a query? Or perhaps, remove them entirely?

1 Like
3

It’s good idea. But i think edit mybb.rb file is better. I am not Ruby developer, so i need some advice about that. :frowning:

Missing thing users import code in the mybb.rb, i see:

Space character and special character cause a problem.
(The solution i think: When user name has a space character or special character user import code delete space or special character and import like that.) One problem is that idea username conflict maybe.

So we need who someone knows Ruby :slight_smile:

5

I don’t know Ruby, but I’m the next best thing, a programmer :stuck_out_tongue:

I can see a few problems with this script. It seems to check the group names to see if someone’s an admin or super mod, but admins can rename those groups or create custom admin / super mod groups.

Do you have any custom admin or super mod groups?
If I recall, there’s a cancp column on the groups table which designates someone as an admin (I’ve wrangled the MyBB code / database before), which should be used instead.

If I learn Ruby, I should open a pull request to fix that. Moving on.
I’m going to check the phpBB one to see if and how it handles bans.

2 Likes
6

https://github.com/discourse/discourse/blob/master/script/import_scripts/phpbb3/importers/user_importer.rb#L79

That’s how they migrate bans for phpBB, but I don’t think it’s implemented in the MyBB importer, it seems incomplete. It tries to convert the spaces to underscores, from the looks of it, but I’m not familiar enough with Ruby to tell you why that’s failing.

1 Like
7

As you say admin, moderators and banned groups cause problem with custom groups. It will be fixed easily change group names in the mybb.rb.

But still import script has a problem about space and special characters. If we edit mybb.rb for delete space and special characters than we will have trouble about username conflict. I’m looking for a way fix that problem on Mybb forum, maybe i can change usernames who has a space and special characters on SQL in a mass. Develop import script still better way for everyone i think.Thank you for all ideas. It was quite useful :slight_smile:

8

I can’t understand. Mybb team will convert my username’s to non-special character names for me?

9

https://github.com/mybb/mybb/blob/ebfa9bc02721a171d042732711fb52ec84989cf8/install/resources/mysql_db_tables.php#L955

To translate the columns on the usergroups table:

  • cancp = Is this an admin? 1 or 0.

  • issupermod = Can this user moderate everywhere? 1 or 0.

  • canmodcp = Does this user have access to the Moderator Control Panel? This is where the ban, announcements, logs, user edit tools (a restricted version of the one in the Administrator Control Panel), etc. are kept. 1 or 0.

  • isbannedgroup = Are the users in this group considered banned? 1 or 0.

These are the important ones. Discourse doesn’t have mod / admin perms, so the rest probably aren’t too useful.

2 Likes
10

He’s probably trolling you.

1 Like
11

That’s odd. It calls UserNameSuggester, which should suggest a valid username in almost every case. Did you see errors during the import?

It sounds like the users are being created (looks like the right number) but that there’s a problem matching the users to the new usernames.

I don’t recommend that.

Thanks! That’s very helpful. I’ve got a tweak to the permalink code that I recently submitted. Perhaps I’ll add this bit in when I get around to that.

12

It sounds like the users are being created (looks like the right number) but that there’s a problem matching the users to the new usernames.

Import process took 8 hours, so i can’t see errors. But if there is a log file somewhere i can check.

I don’t recommend that.

What is your suggestion? Develop import script?

13

Yes. Modifying the import script is probably easier than changing the raw data.

I can’t quite make sense of your description of what happened and a likely fix. If the users are getting imported but not recognized to be connected to their posts, then changing the data probably won’t help.

I don’t have an explanation for either why the users would not be imported nor any they’d not be matched to their posts.

14

To grab the log, use the good old tee command:

RAILS_ENV=production ruby mybb.rb | tee `date +%Y-%m-%d-%N`-import.log

This will create a log file in the current directory.

2 Likes
15

Is there any way to run import script for just import users again? Because running import script took 8 hours.

If it is possible maybe I can try grab log this way. But I know what is the problem anyway. Space and special characters in the username is the problem.

16

You can run the script multiple times and it won’t cause any trouble. Also, the script recognizes what has been imported already and skips it. The following runs will be significantly faster. But your case may be different. Try and see what happens.

2 Likes
17

You can run the script multiple times and it won’t cause any trouble. Also, the script recognizes what has been imported already and skips it. The following runs will be significantly faster. But your case may be different. Try and see what happens.

I run import scritpt again. Here is a log file:

There was no change.

18

Special character seems fixed. But banned users is not banned :slight_smile:

I changed mybb.rb banned users string according to my custom banned group name. But it not fixed.

19

the log says that some of your users have been imported already

creating users
Skipping 1000 already imported users
Skipping 1000 already imported users
Skipping 1000 already imported users

then it starts importing new ones:

     3001 / 9795 ( 30.6%)  
     3002 / 9795 ( 30.6%)  
     [...]

some are skipped, like

 3925 / 9795 ( 40.1%)  Failed to create user id: 4341, username: [HIDDEN] user errors: ["Primary email is invalid"]

until the end

     5000 / 9795 ( 51.0%)  Skipping 1000 already imported users
Skipping 1000 already imported users
Skipping 1000 already imported users
Skipping 1000 already imported users
Skipping 795 already imported users

then it goes to import topics and posts, skipping the ones that are already in discourse:

creating topics and posts
Skipping 1000 already imported posts

I see the failure to convert some of the mybb content to markdown. This is a bug I noticed in the current beta. Use the stable version (1.8.5 at the moment) instead:

 1472 / 329944 (  0.4%)  [99168 items/min]  Exception while creating post 1781. Skipping.
TypeError: Cannot set property 'start' of null
JavaScript at findInlineCloseTag (<anonymous>:194:28)
JavaScript at applyBBCode (<anonymous>:241:20)
JavaScript at Array.md.block.ruler.after.alt (<anonymous>:352:16)
JavaScript at ParserBlock.tokenize (<anonymous>:1249:20)
JavaScript at ParserBlock.parse (<anonymous>:1285:8)
JavaScript at Array.block (<anonymous>:4136:20)
JavaScript at Core.process (<anonymous>:1345:13)
JavaScript at MarkdownIt.parse (<anonymous>:1110:13)
JavaScript at MarkdownIt.render (<anonymous>:1130:36)
JavaScript at cook (<anonymous>:357:32)
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/mini_racer-0.1.11/lib/mini_racer.rb:176:in `eval_unsafe'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/mini_racer-0.1.11/lib/mini_racer.rb:176:in `block (2 levels) in eval'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/mini_racer-0.1.11/lib/mini_racer.rb:256:in `timeout'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/mini_racer-0.1.11/lib/mini_racer.rb:175:in `block in eval'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/mini_racer-0.1.11/lib/mini_racer.rb:68:in `block in with_lock'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/mini_racer-0.1.11/lib/mini_racer.rb:68:in `synchronize'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/mini_racer-0.1.11/lib/mini_racer.rb:68:in `with_lock'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/mini_racer-0.1.11/lib/mini_racer.rb:173:in `eval'
/home/bekircem/discourse/lib/pretty_text.rb:192:in `block in markdown'
/home/bekircem/discourse/lib/pretty_text.rb:390:in `block in protect'
/home/bekircem/discourse/lib/pretty_text.rb:389:in `synchronize'
/home/bekircem/discourse/lib/pretty_text.rb:389:in `protect'
/home/bekircem/discourse/lib/pretty_text.rb:137:in `markdown'
/home/bekircem/discourse/lib/pretty_text.rb:236:in `cook'
/home/bekircem/discourse/app/models/post_analyzer.rb:17:in `cook'
/home/bekircem/discourse/app/models/post.rb:235:in `cook'
/home/bekircem/discourse/lib/post_creator.rb:233:in `before_create_tasks'
/home/bekircem/discourse/app/models/post.rb:508:in `block in <class:Post>'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activesupport-4.2.9/lib/active_support/callbacks.rb:446:in `instance_exec'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activesupport-4.2.9/lib/active_support/callbacks.rb:446:in `block in make_lambda'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activesupport-4.2.9/lib/active_support/callbacks.rb:164:in `block in halting'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activesupport-4.2.9/lib/active_support/callbacks.rb:504:in `block in call'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activesupport-4.2.9/lib/active_support/callbacks.rb:504:in `each'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activesupport-4.2.9/lib/active_support/callbacks.rb:504:in `call'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activesupport-4.2.9/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activesupport-4.2.9/lib/active_support/callbacks.rb:778:in `_run_create_callbacks'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/callbacks.rb:306:in `_create_record'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/timestamp.rb:57:in `_create_record'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/persistence.rb:504:in `create_or_update'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/callbacks.rb:302:in `block in create_or_update'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activesupport-4.2.9/lib/active_support/callbacks.rb:117:in `call'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activesupport-4.2.9/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activesupport-4.2.9/lib/active_support/callbacks.rb:505:in `call'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activesupport-4.2.9/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activesupport-4.2.9/lib/active_support/callbacks.rb:778:in `_run_save_callbacks'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/callbacks.rb:302:in `create_or_update'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/persistence.rb:120:in `save'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/validations.rb:37:in `save'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/attribute_methods/dirty.rb:21:in `save'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/transactions.rb:286:in `block (2 levels) in save'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/connection_adapters/abstract/database_statements.rb:211:in `transaction'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/transactions.rb:220:in `transaction'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/transactions.rb:348:in `with_transaction_returning_status'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/transactions.rb:286:in `block in save'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/transactions.rb:301:in `rollback_active_record_state!'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/transactions.rb:285:in `save'
/home/bekircem/discourse/lib/post_creator.rb:442:in `save_post'
/home/bekircem/discourse/lib/post_creator.rb:149:in `block in create'
/home/bekircem/discourse/lib/distributed_mutex.rb:21:in `synchronize'
/home/bekircem/discourse/lib/distributed_mutex.rb:5:in `synchronize'
/home/bekircem/discourse/lib/post_creator.rb:303:in `block in transaction'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/activerecord-4.2.9/lib/active_record/transactions.rb:220:in `transaction'
/home/bekircem/discourse/lib/post_creator.rb:297:in `transaction'
/home/bekircem/discourse/lib/post_creator.rb:146:in `create'
/home/bekircem/discourse/script/import_scripts/base.rb:491:in `create_post'
/home/bekircem/discourse/script/import_scripts/base.rb:444:in `block in create_posts'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/rack-mini-profiler-0.10.5/lib/patches/db/mysql2.rb:6:in `each'
/home/bekircem/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/rack-mini-profiler-0.10.5/lib/patches/db/mysql2.rb:6:in `each'
/home/bekircem/discourse/script/import_scripts/base.rb:431:in `create_posts'
mybb.rb:125:in `block in import_posts'
/home/bekircem/discourse/script/import_scripts/base.rb:784:in `block in batches'
/home/bekircem/discourse/script/import_scripts/base.rb:783:in `loop'
/home/bekircem/discourse/script/import_scripts/base.rb:783:in `batches'
mybb.rb:103:in `import_posts'
mybb.rb:43:in `execute'
/home/bekircem/discourse/script/import_scripts/base.rb:45:in `perform'
mybb.rb:304:in `<main>'
1 Like
20

I am already use Mybb 1. 8.5 but markdown error doesn’t matter. Thank you for your explanations. Import users problem fixed substantially. I’ve problem just with banned users. We use custom user group for banned users. I am edited Mybb.rb doesn’t work anyway. Do you have any suggestion for that problem?

21

Write a function that queries those users from the database and then suspends them.