Ratelimit Exceeded when migrating threads and posts

Migration
1

Hi all,

I am currently writing something to migrate a Woltlab (Wbb) Forum to Discourse and I am willing to share it once it is somewhat done.

Right now I am more or less successfully migrating categories, forums, users and their groups without too many issues. However every now and then, during the runtime it gets worse, I get a “Ratelimit Exceeded” error, while processing the posts and threads.

My problem is I am still relatively new and I am mostly collecting bits and pieces from other migrators, so I am currently clueless where the message is coming from.

Is it coming from the API of Discourse? And if so can I somewhat disable ratelimiting during imports entirely?

def migrate_threads_and_posts
  puts "Migrating threads and posts..."
  @client.query("SELECT * FROM wbb1_thread ORDER BY time ASC").each do |thread_row|
    begin
      next if thread_row[:isDeleted] == 1 || thread_row[:movedThreadID]

      # Ensure lastPosterID maps to a valid user
      last_post_user_id = @user_map[thread_row[:lastPosterID]] || @user_map[thread_row[:userID]]

      # If lastPosterID and userID are null, create a placeholder user
      if last_post_user_id.nil? && thread_row[:lastPoster]
        last_post_user_id = find_or_create_placeholder_user(thread_row[:lastPoster])
      end

      # Ensure a valid user ID exists
      last_post_user_id ||= User.first.id

      # Get topic creator user ID or create placeholder
      creator_user_id = @user_map[thread_row[:userID]] || find_or_create_placeholder_user(thread_row[:username]) || User.first.id

      # Create the topic
      topic = Topic.new(
        title: thread_row[:topic],
        user_id: creator_user_id,
        category_id: @forum_map[thread_row[:boardID]],
        created_at: Time.at(thread_row[:time]),
        updated_at: Time.at(thread_row[:lastPostTime]),
        last_post_user_id: last_post_user_id
      )

      if topic.save(validate: false)
        puts "Imported thread: #{topic.title} (ID: #{topic.id})"

        # Migrate posts
        migrate_posts(thread_row[:threadID], topic.id)
      else
        puts "Error importing thread #{thread_row[:topic]}: #{topic.errors.full_messages.join(', ')}"
      end

      sleep(7) # Increased sleep to avoid rate limiting
    rescue => e
      puts "Exception importing thread #{thread_row[:topic]}: #{e.message}"
    end
  end
end

I tried to clean up the code and comment as much as possible. Hope that someone can help with the rate limiting.

I also tried to search through the forum here but I only saw many had similar issues but never really a solution. So if I missed something, sorry for the duplicate.

Thanks all.

1 Like
2

Okay. It is literally RateLimiter.disable

Nevermind then :slight_smile:

4 Likes
3

It’ll be much easier if you use one of the import scripts as a starting point.

It will solve a bunch of problems that you don’t yet know that you have.

2 Likes
4

Which problems could that be? I’m currently orienting myself on the phpbb and the xenforo import script.

1 Like
5

The phpbb3 script is well-written, but requires a deep understanding of Ruby. xenforo is ok, I think. You should find one that has the features you want (PMs? permalinks? groups? admins? suspended users?)

I can’t guess what you don’t know, but ways to connect old IDs to the new ones, ways to make sure that the same data doesn’t get imported twice are two that spring to mind.

2 Likes