Fail Fast and Fail Often: Handling API Errors at Scale

At Monolist, we’re building the software engineer’s ideal inbox. Our users depend on us to surface all relevant and actionable tasks and context across all the tools and services they use. For a typical engineer, this includes emails, outstanding Slack messages, Github pull requests, and Jira issues or Asana tasks.To do this, we make millions of API requests each day, spread over a dozen different services. These requests are made via ruby code running on multiple Sidekiq job processing containers across many hosts. (The following post uses Ruby and Sidekiq as examples, but the learnings are relevant to any language and job processing framework.)As these requests succeed, much more processing has to occur to decide which data is relevant to the user, and how to present it to them. In the process of building Monolist, we’ve learned a lot about how to do this scalably, which is probably a good topic for another blog post. In this post, however, I want to share our learnings about what to do with the 100,000 requests that fail each day.