Ratelimit Exceeded when migrating threads and posts

Migration
1

Hi all,

I am currently writing something to migrate a Woltlab (Wbb) Forum to Discourse and I am willing to share it once it is somewhat done.

Right now I am more or less successfully migrating categories, forums, users and their groups without too many issues. However every now and then, during the runtime it gets worse, I get a “Ratelimit Exceeded” error, while processing the posts and threads.

My problem is I am still relatively new and I am mostly collecting bits and pieces from other migrators, so I am currently clueless where the message is coming from.

Is it coming from the API of Discourse? And if so can I somewhat disable ratelimiting during imports entirely?

def migrate_threads_and_posts
  puts "Migrating threads and posts..."
  @client.query("SELECT * FROM wbb1_thread ORDER BY time ASC").each do |thread_row|
    begin
      next if thread_row[:isDeleted] == 1 || thread_row[:movedThreadID]

      # Ensure lastPosterID maps to a valid user
      last_post_user_id = @user_map[thread_row[:lastPosterID]] || @user_map[thread_row[:userID]]

      # If lastPosterID and userID are null, create a placeholder user
      if last_post_user_id.nil? && thread_row[:lastPoster]
        last_post_user_id = find_or_create_placeholder_user(thread_row[:lastPoster])
      end

      # Ensure a valid user ID exists
      last_post_user_id ||= User.first.id

      # Get topic creator user ID or create placeholder
      creator_user_id = @user_map[thread_row[:userID]] || find_or_create_placeholder_user(thread_row[:username]) || User.first.id

      # Create the topic
      topic = Topic.new(
        title: thread_row[:topic],
        user_id: creator_user_id,
        category_id: @forum_map[thread_row[:boardID]],
        created_at: Time.at(thread_row[:time]),
        updated_at: Time.at(thread_row[:lastPostTime]),
        last_post_user_id: last_post_user_id
      )

      if topic.save(validate: false)
        puts "Imported thread: #{topic.title} (ID: #{topic.id})"

        # Migrate posts
        migrate_posts(thread_row[:threadID], topic.id)
      else
        puts "Error importing thread #{thread_row[:topic]}: #{topic.errors.full_messages.join(', ')}"
      end

      sleep(7) # Increased sleep to avoid rate limiting
    rescue => e
      puts "Exception importing thread #{thread_row[:topic]}: #{e.message}"
    end
  end
end

I tried to clean up the code and comment as much as possible. Hope that someone can help with the rate limiting.

I also tried to search through the forum here but I only saw many had similar issues but never really a solution. So if I missed something, sorry for the duplicate.

Thanks all.

2

Okay. It is literally RateLimiter.disable

Nevermind then :slight_smile:

1 Like