Github issues migrator rate limited by github


(Matt Zumwalt) #1

I’m trying to migrate all of the issues from Issues · ipfs/faq · GitHub to https://discuss.ipfs.io using the issues-to-discourse tool. There are a total of about 233 issues. After migrating about 30% of them the remaining issues failed to migrate with this message:

{"message":"You have triggered an abuse detection mechanism and have been temporarily blocked from content creation. Please retry your request again later.","documentation_url":"https://developer.github.com/v3#abuse-rate-limits"}

@erlend_sh do you have any suggestions how to address this?


(Jeff Atwood) #2

Hmm any ideas @cpradio? Can we throttle in the code?


(cpradio) #3

Can’t think of a way off the top of my head (at least not something simple to implement). I’d also need to better understand what sort of limitations we’re looking at from GitHub, as it could theorectically really slow down the entire process. Especially if you also have to worry about the comments being part of what is throttled too (which would be very complicated to code – doing that part was a real P.I.T.A. the first round, not something I’d enjoy reworking).


(Matt Zumwalt) #4

This made a pretty gross mess.

  • Most of the issues got migrated but about half of the gh issues didn’t get updated with links to discourse and didn’t get closed
  • Some of the issues only got partially migrated. Worse, the order of dropped comments is not predictable. Example: When Is it possible to map the blocks to existing files? · Issue #253 · ipfs/faq · GitHub was migrated it missed lidel’s comment, which happens to be the one with the answer to githubber314159’s question.
  • can’t roll back because many of the issues were migrated

(Jeff Atwood) #5

Is this based on size? How many issues are we talking about? Maybe we never did this on a “big” set of GH issues?

We could at least add warnings based on size and rate limiting.


(Matt Zumwalt) #6

I was attempting to migrate 233 issues. Most of them had 1-3 comments. Some had about 8 comments. Github’s daily rate limit is 5,000 requests.