Rate Limiting
Overview
Rate limiting is a critical feature in GitLab that enhances security and performance by restricting the number of requests users or IP addresses can make within a set timeframe. GitLab’s approach to rate limiting helps prevent abuse, ensures fair resource allocation, and maintains system stability, effectively safeguarding against potential attacks and ensuring a reliable user experience.
Rate limited requests will return a 429 - Too Many Requests
response.
Processes
- Changes to rate limits require a Change Request.
- Request assistance for a user’s rate limiting settings with this issue template.
- For internal teams seeking a bypass, please refer to the Rate Limit Bypass Policy.
GitLab Rate Limit Architecture
GitLab.com
Cloud Connector
Rate limits exist in multiple layers for GitLab.com, each boxed area above represents a place where rate limits are implemented.
Limits
To minimise duplication, please see the following sources to determine the current active rate limits:
Cloudflare |
|
---|---|
Application |
|
Bypasses
Published rate limits apply to all customers and users with no exceptions.
Customers or internal teams seeking a bypass should refer to the Rate Limit Bypass Policy.
Traffic management Rate Limits
Cloudflare
Cloudflare serves as our “outer-most” layer of protection, sitting at the network edge on inbound traffic. We use Cloudflare’s standard DDoS (Distributed Denial of Service) protection plus Spectrum to protect git over ssh.
How rate limits are applied in Cloudflare:
Gitlab.com Page Rules |
|
---|---|
Gitlab.com Rate Limiting |
|
Cloud Connector Rate Limiting |
|
Note: Cloudflare is not application aware and does not know how to map to our users and groups.
Cloudflare is also responsible for applying the X-GitLab-Rate-Limit-Bypass
header to a subset of requests from:
- Legacy customer IP rate limit bypasses
- Third party vendors with whom we have integrations
- Internal infrastructure
For a full list of conditions where the header will be applied, see this configuration.
Our Cloudflare runbook contains more detail on configuring this layer of our infrastructure.
Changes to Cloudflare rate limits require a Change Request, and should be discussed with the Production Engineering::Foundations SRE team before implementing.
HAProxy
The majority of GitLab’s traffic management rate limits have been moved out of HAProxy and into Cloudflare (see this confidential issue for more details).
However, HAProxy is still responsible for Registry rate limits, as that component is not fronted by Cloudflare. See Registry for more details.
Currently HAProxy also handles applying the X-GitLab-Rate-Limit-Bypass
for a limited number of special paths. The list can be found here, but there is work underway to move this logic out into Cloudflare as well.
Application Rate Limits
There are multiple application rate limiting mechanisms in place that can often be referred to interchangeably. These allow for more informed traffic management compared to that implemented by Cloudflare as the application has access to user information:
Please see Application Limits Development to contribute application limits to GitLab.
Overview
The rate limit period is 1 minute (60 seconds).
Category |
Identifier |
---|---|
Unauthenticated |
IP address |
Authenticated |
User information (user, project, etc) |
Protected Paths |
Configurable list of paths e.g. |
RackAttack
GitLab utilises RackAttack as middleware to throttle Rack requests. These can be configured by extending Gitlab::RackAttack
and Gitlab::RackAttack::Request
.
For more information about configuring rate limits for a GitLab instance, see the User and IP rate limits doc.
You can read more information about rate limits specific to GitLab.com, alongside RackAttack configuration documentation in runbooks.
ApplicationRateLimiter
The GitLab application has simple rate limit logic that can be used to throttle certain actions which is used when we need more flexibility than what Rack Attack can provide, since it can throttle at the controller or API level. These rate limits are configured in application_rate_limiter.rb. The scope is up to the individual limit implementation and can be any ActiveRecord object or combination of multiple. It is commonly per-user or per-project (or both), but it can be anything, e.g. the RawController limits by project and path.
There is no way to bypass these rate limits (e.g. for select users/groups/projects); when the rate limit is reached a plain response with a 429 status code is issued without rate limiting headers.
Instructions for configuring these rate limits can be found in the GitLab docs.
For more information about introducing new Rate Limits for GitLab, see the Product Processes Handbook page.
Other Rate Limits
GitLab Pages
Because GitLab Pages is not behind CloudFlare and doesn’t have CDN support, rate limits are set in several places:
- Within the Kubernetes settings for GitLab.com.
- In the Go ratelimiter module for Pages.
For more information, see the Pages Rate Limit documentation.
Registry
Registry is not fronted by Cloudflare (see this confidential issue for full details around why). There is no application-level setting for it either (though there is an open feature request to add that), so the only place where there is rate limiting in place for Registry is in HAProxy.
There are no rate limit exceptions available for Registry.
Headers
The list of semi-standard rate limiting response headers can be found here.
Cloudflare
does not return rate limit response headers on any request.RackAttack
returns rate limit response headers on throttled requests only.ApplicationRateLimiter
does not return rate limit response headers.GraphQL
endpoints currently do not return rate limit response headers.
See this issue for improvements to returning rate limiting response headers.
Avoiding Rate Limits
To minimize the risk of hitting rate limits, you can try the following:
- Stagger the execution of your automated pipelines.
- Configure exponential back off and retry for failed attempts.
Troubleshooting
Please see Rate Limiting Troubleshooting.
Important Links
- docs: GitLab.com
- docs: Self Managed (and Dedicated)
- runbook: GitLab.com rate limiting
- handbook: Identifying the cause of IP Blocks on GitLab.com
Rate Limiting Troubleshooting
043436a8
)