Your chat bot needs a better rate limit strategy

Don’t let one ambitious user trigger a denial of service.I’m on a project where we’re connecting to OpenAI to build a chat bot. Because OpenAI’s rate limits are more complicated than other APIs, we made sure to proactively avoid rate limits by keeping track of how many tokens we’re using. However, this approach is incomplete in that it does nothing to limit an individual user’s usage. A chat bot feature is unique in that it can quickly exhaust your organization’s rate limits across multiple dimensions: requests per minute (RPM) and tokens per minute (TPM). This is because….