# How do I avoid rate limiting when importing a bunch of data?

**URL:** https://meta.discourse.org/t/how-do-i-avoid-rate-limiting-when-importing-a-bunch-of-data/270449
**Category:** Development
**Tags:** rest-api, rails-console
**Created:** [July 3, 2023, 11:17pm UTC](https://meta.discourse.org/t/how-do-i-avoid-rate-limiting-when-importing-a-bunch-of-data/270449 "2023-07-03T23:17:09Z")
**Posts on this page:** 15
**Page:** 1

<div class="post-metadata">

### Author: ![Chris\_Dawson](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/chris_dawson/32/184486_2.png) [@Chris\_Dawson](https://meta.discourse.org/u/Chris_Dawson)
#### Post date: [July 3, 2023, 11:17pm UTC](https://meta.discourse.org/t/how-do-i-avoid-rate-limiting-when-importing-a-bunch-of-data/270449/1 "2023-07-03T23:17:09Z")

</div>

I am trying to create a bunch of topics + posts automatically. However, I keep getting hit by rate limits.

I first tried this using the API. My script quickly hit 429 errors. I tried adjusting the rate limits as the admin (see screenshot), but I am unclear what the correct approach is here, or if this is a fool’s errand.

Then, I tried to use a ruby script with `rails r <script>` from inside the container. However, here again, I am seeing `An error occurred RateLimiter::LimitExceeded`. I was surprised at this, why is the ruby method subject to rate limits?

At any rate, is there a way to avoid this? What is the right way to import data in bulk? I thought about using SQL directly but would prefer a more high level API so I don’t have to worry about missing fields in the database when I create an item.

Thanks in advance.

 ![rate-limits](https://global.discourse-cdn.com/meta/original/4X/2/1/f/21f9daf11fa0815be329ea635389e029462be8ae.png)

---

<div class="post-metadata">

### Author: ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [July 4, 2023, 12:16am UTC](https://meta.discourse.org/t/how-do-i-avoid-rate-limiting-when-importing-a-bunch-of-data/270449/2 "2023-07-04T00:16:25Z")

</div>

> [@Chris\_Dawson](#):
>
> , I tried to use a ruby script with `rails r <script>` from inside the container

What does it look like?

You should use rails to import the data, not the API.

You could look at the import scripts, but you might need something simpler.

---

<div class="post-metadata">

### Author: ![Chris\_Dawson](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/chris_dawson/32/184486_2.png) [@Chris\_Dawson](https://meta.discourse.org/u/Chris_Dawson)
#### Post date: [July 4, 2023, 1:18am UTC](https://meta.discourse.org/t/how-do-i-avoid-rate-limiting-when-importing-a-bunch-of-data/270449/3 "2023-07-04T01:18:08Z")

</div>

```plaintext
def add_all()
  limit = 2000
  translations = Dir["/shared/translations/vagrant-data/translations/*.txt"]
  ps = Dir["/shared/translations/vagrant-data/paragraphs/*.txt"]

  # RateLimiter.new.disable_rate_limits!

  translations.each_with_index do |t,i|
    p = ps[i]

    tr_text = File.read(t)
    p_text = File.read(p)
   

    description = "#{p_text}\n\nTranslated as:\n\n#{tr_text}"
    title = " the booki (paragraph ##{i+1})"
    if (i < limit)
      begin
        puts "Adding topic #{tr_text}"
        topic = Topic.create!( title: title, category_id: 5, user_id: 3 )
        Post.create!( topic_id: topic.id, raw: description, user_id: 3, skip_validation: true )
      rescue Exception => e
        puts "An error occurred #{e}"
      end
    end
  end
end

add_all()

```

---

<div class="post-metadata">

### Author: ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [July 4, 2023, 1:28am UTC](https://meta.discourse.org/t/how-do-i-avoid-rate-limiting-when-importing-a-bunch-of-data/270449/4 "2023-07-04T01:28:20Z")

</div>

And that generates a rate limit error? I would think that would work just fine.

---

<div class="post-metadata">

### Author: ![Chris\_Dawson](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/chris_dawson/32/184486_2.png) [@Chris\_Dawson](https://meta.discourse.org/u/Chris_Dawson)
#### Post date: [July 4, 2023, 1:43am UTC](https://meta.discourse.org/t/how-do-i-avoid-rate-limiting-when-importing-a-bunch-of-data/270449/5 "2023-07-04T01:43:49Z")

</div>

Yes, see my prior post. It gets into the rescue block with that error.

---

<div class="post-metadata">

### Author: ![Chris\_Dawson](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/chris_dawson/32/184486_2.png) [@Chris\_Dawson](https://meta.discourse.org/u/Chris_Dawson)
#### Post date: [July 4, 2023, 2:23am UTC](https://meta.discourse.org/t/how-do-i-avoid-rate-limiting-when-importing-a-bunch-of-data/270449/6 "2023-07-04T02:23:21Z")

</div>

I wonder if my server is in a weird state. I had messed around with the rate limits in the screenshot, setting them to zero and maybe -1. Now I’m getting these rate limit errors everywhere. It’s strange though, some of the requests make it through so out of 1500 records I want to create, three or four make it into the database. Maybe I’ll blow away the database and refresh and see if that fixes it. Not sure how to check the state of the database and capture the bug if there is one.

---

<div class="post-metadata">

### Author: ![Stephen](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/stephen/32/95011_2.png) [@Stephen](https://meta.discourse.org/u/Stephen)
#### Post date: [July 4, 2023, 3:06am UTC](https://meta.discourse.org/t/how-do-i-avoid-rate-limiting-when-importing-a-bunch-of-data/270449/7 "2023-07-04T03:06:50Z")

</div>

How was this instance built? Which version is it running?

---

<div class="post-metadata">

### Author: ![Chris\_Dawson](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/chris_dawson/32/184486_2.png) [@Chris\_Dawson](https://meta.discourse.org/u/Chris_Dawson)
#### Post date: [July 4, 2023, 4:30pm UTC](https://meta.discourse.org/t/how-do-i-avoid-rate-limiting-when-importing-a-bunch-of-data/270449/8 "2023-07-04T16:30:42Z")

</div>

```plaintext
#### Installed

### 3.1.0.beta5

( [5584fb1e3b](https://github.com/discourse/discourse/commits/5584fb1e3b7a29d7ee5d7e43520191081dd10a16) )

```

I just did the regular docker installation. Nothing special.

---

<div class="post-metadata">

### Author: ![Chris\_Dawson](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/chris_dawson/32/184486_2.png) [@Chris\_Dawson](https://meta.discourse.org/u/Chris_Dawson)
#### Post date: [July 4, 2023, 4:33pm UTC](https://meta.discourse.org/t/how-do-i-avoid-rate-limiting-when-importing-a-bunch-of-data/270449/9 "2023-07-04T16:33:17Z")

</div>

Is it possible to temporarily disable rate limits while I’m importing and then restore later? Right not I’m the only one who will ever use the API (I don’t anticipate letting my users use the API). So, I could even permanently disable the rate limits if that’s possible.

---

<div class="post-metadata">

### Author: ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [July 4, 2023, 8:15pm UTC](https://meta.discourse.org/t/how-do-i-avoid-rate-limiting-when-importing-a-bunch-of-data/270449/10 "2023-07-04T20:15:41Z")

</div>

I’ve never seen rate limits running in Rails. I don’t have any clues.

---

<div class="post-metadata">

### Author: ![Chris\_Dawson](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/chris_dawson/32/184486_2.png) [@Chris\_Dawson](https://meta.discourse.org/u/Chris_Dawson)
#### Post date: [July 4, 2023, 8:57pm UTC](https://meta.discourse.org/t/how-do-i-avoid-rate-limiting-when-importing-a-bunch-of-data/270449/11 "2023-07-04T20:57:11Z")

</div>

@pfaffman I’m betting I screwed something up by changing rate limits and now the system is in a weird state and these errors are not truly the issue, something else is. I’m going to create the server anew and see if my script works this time and then report back.

---

<div class="post-metadata">

### Author: ![Chris\_Dawson](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/chris_dawson/32/184486_2.png) [@Chris\_Dawson](https://meta.discourse.org/u/Chris_Dawson)
#### Post date: [July 5, 2023, 12:17pm UTC](https://meta.discourse.org/t/how-do-i-avoid-rate-limiting-when-importing-a-bunch-of-data/270449/12 "2023-07-05T12:17:53Z")

</div>

Sadly, recreating the server did not fix things. ☹

I deleted the postgres\_data directory inside shared, and then ran `./launcher rebuild app` and got a new instance of the app.

My script now creates the user and the category owned by the user, and then attempt to create topics inside that category. But, it immediately hits the rate limit error: `An error occurred RateLimiter::LimitExceeded`

I’m really stumped as it sounds like the `rails r` scripts should not be subject to the rate limit issues, right? I do see code in the models that indicate that rate limiting code it executed, but I don’t know how to disable it nor how to verify that this should only happen when a request come in via the API.

Any other suggestions? I think going directly in via psql is now my best bet, which is disappointing because the `rails r` script is so much cleaner and easier to use.

---

<div class="post-metadata">

### Author: ![Chris\_Dawson](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/chris_dawson/32/184486_2.png) [@Chris\_Dawson](https://meta.discourse.org/u/Chris_Dawson)
#### Post date: [July 5, 2023, 1:29pm UTC](https://meta.discourse.org/t/how-do-i-avoid-rate-limiting-when-importing-a-bunch-of-data/270449/13 "2023-07-05T13:29:14Z")

</div>

I fixed it!

The power of tests are proved out right here. I went to this file:

> <https://github.com/discourse/discourse/blob/main/spec/integration/rate_limiting_spec.rb>

That showed this method `RateLimiter.enable`. I thought, why not try `RateLimiter.disable`.

No more rate limit errors!

---

<div class="post-metadata">

### Author: ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [July 6, 2023, 12:07am UTC](https://meta.discourse.org/t/how-do-i-avoid-rate-limiting-when-importing-a-bunch-of-data/270449/14 "2023-07-06T00:07:12Z")

</div>

Nice work! I’ve never seen this before.

---

<div class="post-metadata">

### Author: ![system](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/system/32/443519_2.png) [@system](https://meta.discourse.org/u/system)
#### Post date: [August 5, 2023, 12:07am UTC](https://meta.discourse.org/t/how-do-i-avoid-rate-limiting-when-importing-a-bunch-of-data/270449/15 "2023-08-05T00:07:23Z")

</div>

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.