What would you like done?
We have a custom theme component (the “Bio Book”) that recently started returning 429 errors. This started on November 26th. Before that it was working.
The Bio Book hacks the /u route to display user info in a more beautiful way. To do that it was also requesting additional user info from each user’s .json file. The Bio Book was lazy loading to avoid 429 errors, but that stopped working four days ago—I’m not sure what happened four days ago, but the Discourse team has confirmed that there weren’t any changes to the rate limit.
The developer who originally created the component for us is unavailable to take on this fix, so we need someone who can step into the existing code and resolve the error(s) so that the custom theme component is again working.
We don’t want any new features, we don’t need to change any existing features, and we don’t want to refactor anything. We just want to resolve the 429 errors and return the Bio Book to it’s previous functionality.
When do you need it done?
We’d like it completed by Monday, January 6th, 2020.
What is your budget, in $ USD that you can offer for this task?
Our budget is $250 for resolving the 429 errors.
Our understanding is that resolving the 429 errors will fix all the underlying issues. If that’s not the case please let us know and we’ll adjust our budget accordingly.
So it’s lazy loading in groups of 50 users, and the default value for max_reqs_per_ip_per_10_seconds is set to 50 as well. Including the main page this code will make at least 51 requests (in less than 10 seconds), so it’s exceeding the rate limit.
In other words: I don’t see how this could have worked before…?
We just flicked on the rate limit globally For all discourse instances everywhere last week, we can investigate the plugin next week and see if there is a reasonable batch api it can use instead
Wow, I recently ran into this same issue for a very similar reason (pulling user info to display cards!), implemented a small hack to fix it for the moment, and came to the forums literally to find out if there was a batch API I didn’t know about… or to learn how to implement such a route.
I just had a look at the theme component. It does indeed introduce an enormous amount of work and a huge number of reqs. The rate limiting we did is totally correct here as an immune system.
We are going to sort this out, but it may take a week or two to get this all done.
It seems to me the huge piece here is that you need directory to start returning all the info user cards have. The tricky thing is that theme components do not have the ability to amend the server, so we are going to need to make some core API change that allows us to ask for user cards for multiple people in one go.
We will update you as we make progress. Quick one re budget, we will do this one for free, but I estimate about 2 days of work here in repairing this which is way over what the initial budget here was. It seems simple on the surface but requires very complex internal changes.
Thank you @sam and team, this is tremendously helpful. The core API changes will definitely serve us and I imagine other Discourse users. I’m glad we’ll be able to use the Bio Book and to make it work more sensibly. A week or two works for us.
I hear you about the initial budget and the scope of the fix. It’s tremendously generous of you to do this one for free. Thank you.
