How do I avoid rate limiting when importing a bunch of data?

Dev
,
1

I am trying to create a bunch of topics + posts automatically. However, I keep getting hit by rate limits.

I first tried this using the API. My script quickly hit 429 errors. I tried adjusting the rate limits as the admin (see screenshot), but I am unclear what the correct approach is here, or if this is a fool’s errand.

Then, I tried to use a ruby script with rails r <script> from inside the container. However, here again, I am seeing An error occurred RateLimiter::LimitExceeded. I was surprised at this, why is the ruby method subject to rate limits?

At any rate, is there a way to avoid this? What is the right way to import data in bulk? I thought about using SQL directly but would prefer a more high level API so I don’t have to worry about missing fields in the database when I create an item.

Thanks in advance.

rate-limits
rate-limits994×469 44.9 KB

2

What does it look like?

You should use rails to import the data, not the API.

You could look at the import scripts, but you might need something simpler.

3 
def add_all()
  limit = 2000
  translations = Dir["/shared/translations/vagrant-data/translations/*.txt"]
  ps = Dir["/shared/translations/vagrant-data/paragraphs/*.txt"]

  # RateLimiter.new.disable_rate_limits!

  translations.each_with_index do |t,i|
    p = ps[i]

    tr_text = File.read(t)
    p_text = File.read(p)
   

    description = "#{p_text}\n\nTranslated as:\n\n#{tr_text}"
    title = " the booki (paragraph ##{i+1})"
    if (i < limit)
      begin
        puts "Adding topic #{tr_text}"
        topic = Topic.create!( title: title, category_id: 5, user_id: 3 )
        Post.create!( topic_id: topic.id, raw: description, user_id: 3, skip_validation: true )
      rescue Exception => e
        puts "An error occurred #{e}"
      end
    end
  end
end

add_all()
4

And that generates a rate limit error? I would think that would work just fine.

5

Yes, see my prior post. It gets into the rescue block with that error.

1 Like
6

I wonder if my server is in a weird state. I had messed around with the rate limits in the screenshot, setting them to zero and maybe -1. Now I’m getting these rate limit errors everywhere. It’s strange though, some of the requests make it through so out of 1500 records I want to create, three or four make it into the database. Maybe I’ll blow away the database and refresh and see if that fixes it. Not sure how to check the state of the database and capture the bug if there is one.

1 Like
7

How was this instance built? Which version is it running?

1 Like
8 
#### Installed

### 3.1.0.beta5

( [ 5584fb1e3b ](https://github.com/discourse/discourse/commits/5584fb1e3b7a29d7ee5d7e43520191081dd10a16) )

I just did the regular docker installation. Nothing special.

9

Is it possible to temporarily disable rate limits while I’m importing and then restore later? Right not I’m the only one who will ever use the API (I don’t anticipate letting my users use the API). So, I could even permanently disable the rate limits if that’s possible.

10

I’ve never seen rate limits running in Rails. I don’t have any clues.

1 Like
11

@pfaffman I’m betting I screwed something up by changing rate limits and now the system is in a weird state and these errors are not truly the issue, something else is. I’m going to create the server anew and see if my script works this time and then report back.

2 Likes
12

Sadly, recreating the server did not fix things. :frowning:

I deleted the postgres_data directory inside shared, and then ran ./launcher rebuild app and got a new instance of the app.

My script now creates the user and the category owned by the user, and then attempt to create topics inside that category. But, it immediately hits the rate limit error: An error occurred RateLimiter::LimitExceeded

I’m really stumped as it sounds like the rails r scripts should not be subject to the rate limit issues, right? I do see code in the models that indicate that rate limiting code it executed, but I don’t know how to disable it nor how to verify that this should only happen when a request come in via the API.

Any other suggestions? I think going directly in via psql is now my best bet, which is disappointing because the rails r script is so much cleaner and easier to use.

13

I fixed it!

The power of tests are proved out right here. I went to this file:

That showed this method RateLimiter.enable. I thought, why not try RateLimiter.disable.

No more rate limit errors!

3 Likes
14

Nice work! I’ve never seen this before.

15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.