I am trying to create a bunch of topics + posts automatically. However, I keep getting hit by rate limits.
I first tried this using the API. My script quickly hit 429 errors. I tried adjusting the rate limits as the admin (see screenshot), but I am unclear what the correct approach is here, or if this is a fool’s errand.
Then, I tried to use a ruby script with rails r <script> from inside the container. However, here again, I am seeing An error occurred RateLimiter::LimitExceeded. I was surprised at this, why is the ruby method subject to rate limits?
At any rate, is there a way to avoid this? What is the right way to import data in bulk? I thought about using SQL directly but would prefer a more high level API so I don’t have to worry about missing fields in the database when I create an item.
I wonder if my server is in a weird state. I had messed around with the rate limits in the screenshot, setting them to zero and maybe -1. Now I’m getting these rate limit errors everywhere. It’s strange though, some of the requests make it through so out of 1500 records I want to create, three or four make it into the database. Maybe I’ll blow away the database and refresh and see if that fixes it. Not sure how to check the state of the database and capture the bug if there is one.
Is it possible to temporarily disable rate limits while I’m importing and then restore later? Right not I’m the only one who will ever use the API (I don’t anticipate letting my users use the API). So, I could even permanently disable the rate limits if that’s possible.
@pfaffman I’m betting I screwed something up by changing rate limits and now the system is in a weird state and these errors are not truly the issue, something else is. I’m going to create the server anew and see if my script works this time and then report back.
I deleted the postgres_data directory inside shared, and then ran ./launcher rebuild app and got a new instance of the app.
My script now creates the user and the category owned by the user, and then attempt to create topics inside that category. But, it immediately hits the rate limit error: An error occurred RateLimiter::LimitExceeded
I’m really stumped as it sounds like the rails r scripts should not be subject to the rate limit issues, right? I do see code in the models that indicate that rate limiting code it executed, but I don’t know how to disable it nor how to verify that this should only happen when a request come in via the API.
Any other suggestions? I think going directly in via psql is now my best bet, which is disappointing because the rails r script is so much cleaner and easier to use.