The discussion revolves around bypassing rate limits in Discourse, specifically for bulk creation of users and posts. jamesmarkcook initially raised the issue of needing to seed a production instance with around 200 posts at a time, but hit a rate limiter when trying to flag some posts. pfaffman suggested running a script in the Rails console or changing the rate limits, but jamesmarkcook preferred a user-initiated solution.
Later, Isambard joined the discussion, trying to import 60,000 users via API without hitting rate limits. pfaffman suggested creating users in Rails instead of via the API, and provided a script as a starting point. However, Isambard encountered issues with the rate limiter despite updating the discourse.conf file.
The conversation continued, with pfaffman suggesting that the issue might be related to the maximum registrations from an IP limit. Isambard eventually worked around the issue by adding a TL2 user on the same IP address or tweaking the IP limit.
Solution
Isambard ultimately bypassed all limits by writing directly to the database using SQL, achieving several thousand insertions per second. They shared their approach, including the tables that needed to be updated to insert users:
user_avatars
user_emails
user_profiles
user_search_data
user_stats
users
This solution allowed Isambard to successfully import a large number of users without hitting rate limits.
So here is my issue. I have a requirement to essentially seed a Discourse production instance with around 200 posts at a time from a plugin admin action. These posts will be ‘created by’ 1 of 10 different regular users. The reason it’s a plugin action is because the users of this particular instance have a team of moderators they want to train up, and wanted some test posts to train them on, and the ability to seed more when they need them.
I got this working fine by passing skip_validations: true to PostCreator.new, however now there’s a requirement that some of the created posts are also flagged.
I first attempted to disable the RateLimiter but that was causing my action to crash the server process eventually, possibly when I was trying to turn it back on, and then I realised it probably wasn’t a good idea anyway.
So my question is, is there a better way to bypass the rate limiter when running some arbitrary code i.e. something like:
RateLimiter.bypass do
# run some code not affected by the rate limiter
end
Or do I need to basically just need to copy most of what the PostActionCreator is doing but leave out the troublesome line?
Any help would be greatly appreciated! I’m still absorbing a lot of the Discourse code so I’m aware I’ve probably missed something that makes this a lot easier!
Did you figure this out? Modfying the yaml file is not ideal as it requires a rebuild. I’m not familiar enough with rails/discourse to figure out how to disable it temporarily in the console.
Trying to import 60k users via API as fast as possible without doing a rebuild. I also tried on a test installation to raise the ADMIN API limit via YAML, but it didn’t seem to work but maybe I did something wrong. (I’m running imports on the container so the http throttling shouldn’t apply).
But you can go into the container and edit /var/www/discourse/config/discourse.conf and set those variables there and then do an sv restart unicorn (or maybe sv reload unicorn). You’ll probably want to do something like apt-get update; apt-get install -y vim to have an editor.
Strange. When I view the conf file the new limit is already set. So the YAML update worked:
max_admin_api_reqs_per_key_per_minute = '6000'
But it doesn’t seem to work. There’s still a 60 per minute limitation. When I throttle requests to 60 per minute, most go through but due to jitter a few still trigger the rate limiter:
{'success': True, 'active': True, 'message': 'Your account is activated and ready to use.', 'user_id': 3596}
{'success': False, 'message': ' New registrations are not allowed from your IP address (maximum limit reached). Contact a staff member.', 'errors': {'ip_address': ['New registrations are not allowed from your IP address (maximum limit reached). Contact a staff member.']}, 'values': {'name': None, 'username': 'asdfd', 'email': 'user@domain.com'}, 'is_developer': False}
{'success': True, 'active': True, 'message': 'Your account is activated and ready to use.', 'user_id': 3597}
I think it is related to another limit: I think it is the maximum registrations from an IP limit that it is hitting but I don’t understand why after it hits this it doesn’t permanently block. Possibly a bug in this spam limit?
If those users all have the same IP, then I bet you’re right. I think the solution is to change that setting or give people bogus IP addresses like 127.0.0.x (or maybe use IPv6 to make it easier?)
I think it is detecting where the request is made from (host computer) but I can work around it by adding a TL2 user on the same IP address (or tweak the IP limit.
Yes, I will try that. But I’m not clear how I’m supposed to run the script, where do I copy it to and how do I execute it (once I put the CSV in the right places)?
Just as a follow-up given the effort to understand the scripts and ruby etc. and the likely performance, I went another route and bypassed all limits by writing directly to the database using SQL. This way I was able to get several thousand insertions per second.
There was a topic asking which tables were required, it is closed now so I reply here that I changed the following tables to insert the users: