Rate limiting at GitLab

Overview

Rate limiting is a critical feature in GitLab that enhances security and performance by restricting the number of requests users or IP addresses can make within a set timeframe. GitLab’s approach to rate limiting helps prevent abuse, ensures fair resource allocation, and maintains system stability, effectively safeguarding against potential attacks and ensuring a reliable user experience.

Rate limited requests will return a 429 - Too Many Requests response.

Processes

• Changes to rate limits require a Change Request.
• Request assistance for a user’s rate limiting settings with this issue template.
• For internal teams seeking a bypass, please refer to the Rate Limit Bypass Policy.

GitLab.com Rate Limit Architecture

flowchart TD
    A[Internet]
    A -->|gitlab.com| D(Cloudflare WAF)
    A -->|cloud.gitlab.com| E
    A -->|customers.gitlab.com| F
    subgraph ide3 [Cloudflare WAF]
    D[gitlab.com zone]
    E[cloud.gitlab.com zone]
    F[customers.gitlab.com zone]
    end
    A --> C
    D --> G
    E --> G
    F --> H
    subgraph ide2 [GitLab]
    subgraph ide1 [HAProxy]
    C[pages_http]
    G[http + ssh]
    end
    C --> K
    H[nginx]
    K[pages kubernetes limits]
    K-->M[pages application limits]
    I[Rack::Attack]
    J[Gitlab::ApplicationRateLimiter]
    end
    G --> I
    G --> J

Rate limits exist in multiple layers for GitLab.com, each boxed area above represents a place where rate limits are implemented.

Limits

To minimise duplication, please see the following sources to determine the current active rate limits:

Bypasses

Published rate limits apply to all customers and users with no exceptions.

Please see the Rate Limit Bypass Policy for more detailed information.

Traffic management Rate Limits

Cloudflare

Cloudflare serves as our “outer-most” layer of protection, sitting at the network edge on inbound traffic. We use Cloudflare’s standard DDoS (Distributed Denial of Service) protection plus Spectrum to protect git over ssh.

How rate limits are applied in Cloudflare:

Note: Cloudflare is not application aware and does not know how to map to our users and groups.

Our Cloudflare runbook contains more detail on configuring this layer of our infrastructure.

Changes to Cloudflare rate limits require a Change Request, and should be discussed with the Production Engineering::Foundations SRE team before implementing.

HAProxy

The Production Engineering::Foundations team has been moving the majority of GitLab’s traffic management rate limits out of HAProxy and into Cloudflare. As such, it is only configured to rate limit GitLab Pages as it does not have CDN support.

Additionally, HAProxy is responsible for handling the bypass header, which allows for a configured list of IP addresses to bypass rate limits in HAProxy or Rack Attack. There are two types:

1. Internal: Full bypass, including some other protections. Only CI runner managers (or similar) should ever be added to the internal list. Managed in Chef.
2. External IPs: An allowlist of IP addresses. Managed in Chef. Requests from these IP addresses are still subject to additional checks, before bypassing the rest of the rate-limiting.

A full list of cases where the X-GitLab-Rate-Limit-Bypass header is used can be found in the HAProxy cookbook.

Changes to HAProxy rate limits require a Change Request, and any customers or internal teams seeking a bypass should refer to the Rate Limit bypass policy.

Application Rate Limits

There are multiple application rate limiting mechanisms in place that can often be referred to interchangeably. These allow for more informed traffic management compared to that implemented by Cloudflare as the application has access to user information:

• Rack Attack
• Application Rate Limiter
• Plan Limits

Please see Application Limits Development to contribute application limits to GitLab.

Overview

The rate limit period is 1 minute (60 seconds).

RackAttack

GitLab utilises RackAttack as middleware to throttle Rack requests. These can be configured by extending Gitlab::RackAttack and Gitlab::RackAttack::Request.

For more information about configuring rate limits for a GitLab instance, see the User and IP rate limits doc.

You can read more information about rate limits specific to GitLab.com, alongside RackAttack configuration documentation in runbooks.

ApplicationRateLimiter

The GitLab application has simple rate limit logic that can be used to throttle certain actions which is used when we need more flexibility than what Rack Attack can provide, since it can throttle at the controller or API level. These rate limits are set configured in application_rate_limiter.rb. The scope is up to the individual limit implementation and can be any ActiveRecord object or combination of multiple. It is commonly per-user or per-project (or both), but it can be anything, e.g. the RawController limits by project and path.

There is no way to bypass these rate limits (e.g. for select users/groups/projects); when the rate limit is reached a plain response with a 429 status code is issued without rate limiting headers.

Instructions for configuring these rate limits can be found in the GitLab docs.

For more information about introducing new Rate Limits for GitLab, see the Product Processes Handbook page.

Other Rate Limits

GitLab Pages

Because GitLab Pages is not behind CloudFlare and doesn’t have CDN support, rate limits are set in several places:

1. There exist some limits in HAProxy.
2. Within the kubernetes settings for GitLab.com.
3. In the go ratelimiter module for Pages.

For more information, see the Pages Rate Limit documentation.

Headers

Below are the list of semi-standard rate limiting response headers that can be returned on rate limited requests.

Important Information

• Cloudflare does not return rate limit response headers on any request.
• RackAttack returns rate limit response headers on throttled requests only.
• ApplicationRateLimiter will return rate limit response headers once this issue is implemented.
• GraphQL endpoints currently do not return rate limit response headers.

Troubleshooting

Observability

Our support team also have a Kibana dashboard showing metrics around rate limited responses. Cloudflare rate limits responses can be seen in the Cloudflare UI (Gitlab Cloudflare account access required).

Investigating RackAttack logs

• json.meta.user field is set if a request is authenticated, and missing if it was anonymous.
• json.env will either be set to throttle or blocklist, the latter which comes from failed authentication bans.

Important Links

• GitLab.com Rate limits overview
• docs: GitLab.com
• docs: Self Managed (and Dedicated)

View page source - Edit this page - please contribute.