Rate Limit Architecture Working Group

Attributes

Summary

The Rate Limit Architecture WG was active from April - December 2022. In FY23-Q1 we regularly experienced site incidents related to a lack of rate limits on various features. We initiated an Engineering Allocation for Rate Limiting Across DevOps Platform which performed an Rate Limiting audit for each DevOps Stage, and updated development docs and guidelines to provide clearer rate limit related guidance, and closed 70+ rate limiting issues. This resulted in increased reliability and a major reduction in the number of rate limit related site incidents.

Following the Engineering Allocation we initiated this Working Group to establish a forward looking vision for Rate Limiting. The group developed a Rate Limiting Architecture Blueprint which captures this vision. The group also collaborated to identify which GitLab stage group would own this functionality long term. This was documented by adding the Rate Limiting Framework to the Application Performance group. The blueprint will continue to be reviewed regularly as part of the Architecture Evolution Workflow process.

As the Working Group has accomplished it’s exit criteria we made the decision to disband the group.

Context

During FY23-Q1 we prioritized an engineering allocation for Rate Limiting Across DevOps Platform focused on identifying and addressing denial of service (DoS) vectors via rate and application limits. So far that effort has been successful in addressing 70+ of DoS related issues.

In addition to addressing DoS vectors to ensure high availability we have goals around:

• consistent monitoring and alerting on limits
• consistent, comprehensive documentation of limits
• synchronization of GitLab.com limit settings with GitLab default limits

Currently there are multiple approaches to rate limiting documented in GitLab application limits development documentation.

Exit Criteria

This Working Group has the following goals:

1. Develop a recommendation for the preferred way to implement application rate limits within GitLab
2. Document this recommendation for GitLab developers in an Architecture Evolution Blueprint.
3. Document iterative project plan for short term, medium term, and long term rate limiting improvements
4. Determine organizational approach to funding effort on this work (e.g. new team, borrow, SIG, fan out, etc.)

Roles and Responsibilities

View page source - Edit this page - please contribute.