Any way to turn off RateLimiter temporarily for bulk creation by admin?

Dev
1

So here is my issue. I have a requirement to essentially seed a Discourse production instance with around 200 posts at a time from a plugin admin action. These posts will be ‘created by’ 1 of 10 different regular users. The reason it’s a plugin action is because the users of this particular instance have a team of moderators they want to train up, and wanted some test posts to train them on, and the ability to seed more when they need them.

I got this working fine by passing skip_validations: true to PostCreator.new, however now there’s a requirement that some of the created posts are also flagged.

I’m using PostActionCreator.create to flag some of these posts, but now I’m getting hit by the rate limiter here: https://github.com/discourse/discourse/blob/ad7a13921f2af8c792530c84386b64911c8e7ea2/lib/post_action_creator.rb#L69

I first attempted to disable the RateLimiter but that was causing my action to crash the server process eventually, possibly when I was trying to turn it back on, and then I realised it probably wasn’t a good idea anyway.

So my question is, is there a better way to bypass the rate limiter when running some arbitrary code i.e. something like:

RateLimiter.bypass do
# run some code not affected by the rate limiter
end

Or do I need to basically just need to copy most of what the PostActionCreator is doing but leave out the troublesome line?

Any help would be greatly appreciated! I’m still absorbing a lot of the Discourse code so I’m aware I’ve probably missed something that makes this a lot easier!

1 Like
2

What I’d likely do is run a script in the Rails console. Have you checked out: Global rate limits and throttling in Discourse? It looks like you can change those rate limits.

2 Likes
3

This isn’t really a path I want to go down as I want it to initiated by users rather than them relying on me to run a script in the console.

3 Likes
4

Did you look at the link about changing the rate limits?

5

Did you figure this out? Modfying the yaml file is not ideal as it requires a rebuild. I’m not familiar enough with rails/discourse to figure out how to disable it temporarily in the console.

6

what is your use case? What problem are you trying to solve that rate limits are standing in the way of?

7

Trying to import 60k users via API as fast as possible without doing a rebuild. I also tried on a test installation to raise the ADMIN API limit via YAML, but it didn’t seem to work but maybe I did something wrong. (I’m running imports on the container so the http throttling shouldn’t apply).

8

This isn’t very helpful, but what I’d do is create the users in rails rather than via the API. discourse/script/import_scripts/csv_importer.rb at main · discourse/discourse · GitHub could be a start.

But you can go into the container and edit /var/www/discourse/config/discourse.conf and set those variables there and then do an sv restart unicorn (or maybe sv reload unicorn). You’ll probably want to do something like apt-get update; apt-get install -y vim to have an editor.

2 Likes
9

Strange. When I view the conf file the new limit is already set. So the YAML update worked:

max_admin_api_reqs_per_key_per_minute = '6000'

But it doesn’t seem to work. There’s still a 60 per minute limitation. When I throttle requests to 60 per minute, most go through but due to jitter a few still trigger the rate limiter:

{'success': True, 'active': True, 'message': 'Your account is activated and ready to use.', 'user_id': 3596}
{'success': False, 'message': ' New registrations are not allowed from your IP address (maximum limit reached). Contact a staff member.', 'errors': {'ip_address': ['New registrations are not allowed from your IP address (maximum limit reached). Contact a staff member.']}, 'values': {'name': None, 'username': 'asdfd', 'email': 'user@domain.com'}, 'is_developer': False}
{'success': True, 'active': True, 'message': 'Your account is activated and ready to use.', 'user_id': 3597}
10

I wonder if it’s an issue with the quotes?

You might check SiteSettings.max_admin_api_reqs_per_key_per_minute and see if it’s an integer

1 Like
11

in default_current_user_provider.rb (the only file I found that seemed to reference this value, it has:

limit = GlobalSetting.max_admin_api_reqs_per_minute.to_i

so it does seem to convert it. I even tried replacing with a number in the code.

1 Like
12

Darn. I was afraid of that.

1 Like
13

I think it is related to another limit: I think it is the maximum registrations from an IP limit that it is hitting but I don’t understand why after it hits this it doesn’t permanently block. Possibly a bug in this spam limit?

14

If those users all have the same IP, then I bet you’re right. I think the solution is to change that setting or give people bogus IP addresses like 127.0.0.x (or maybe use IPv6 to make it easier?)

1 Like
15

I think it is detecting where the request is made from (host computer) but I can work around it by adding a TL2 user on the same IP address (or tweak the IP limit.

16

user add API is very slow, though.

17

That’s why I recommend to do it in rails. It’ll likely do about 500/minute.

1 Like
18

Yes, I will try that. But I’m not clear how I’m supposed to run the script, where do I copy it to and how do I execute it (once I put the CSV in the right places)?

19

You can look at some of the other how to topics for the others. They mostly all work the same.

1 Like
20

Just as a follow-up given the effort to understand the scripts and ruby etc. and the likely performance, I went another route and bypassed all limits by writing directly to the database using SQL. This way I was able to get several thousand insertions per second.

There was a topic asking which tables were required, it is closed now so I reply here that I changed the following tables to insert the users:

user_avatars
user_emails
user_profiles
user_search_data
user_stats
users
1 Like