Debugging Sidekiq Poison Pills

Like many web applications out there, Gusto uses asynchronous jobs to process various sets of information. Sidekiq is one such background processing library for Ruby, which we use at Gusto. Other such libraries include Resque, Beanstalkd, and delayed_jobs.Recently, a series of design choices made over time led to the complete halt of all asynchronous jobs in one of our main applications. This was particularly disastrous because it happened overnight while running a lot of routine maintenence code. Our maintenance code handles some really important work: it ensures that people are paid accurately and receive the appropriate health insurance. When almost all of it was prevented from running, we had an incident on our hands. Here's how we mitigated the incident that night — and prevented it from ever happening again.