Rate Limiting

Overview

Rate limiting is a critical feature in GitLab that enhances security and performance by restricting the number of requests users or IP addresses can make within a set timeframe. GitLab’s approach to rate limiting helps prevent abuse, ensures fair resource allocation, and maintains system stability, effectively safeguarding against potential attacks and ensuring a reliable user experience.

Rate limited requests will return a 429 - Too Many Requests response.

Processes

• Changes to rate limits require a Change Request.
• Request assistance for a user’s rate limiting settings with this issue template.
• For internal teams seeking a bypass, please refer to the Rate Limit Bypass Policy.

GitLab Rate Limit Architecture

GitLab.com

flowchart TD
    A[Internet]
    A -->|gitlab.com| D(Cloudflare WAF)
    A -->|customers.gitlab.com| F
    subgraph ide3 [Cloudflare WAF]
        D[gitlab.com zone]
        F[customers.gitlab.com zone]
    end
    A -->|*.gitlab.io| C
    A -->|registry.gitlab.com| N
    D --> G
    F --> H
    subgraph ide2 [GitLab]
        subgraph ide1 [HAProxy]
            C[pages_http]
            N[registry_https]
            G[https + ssh]
        end
        C --> K
        H[nginx]
        K[pages kubernetes limits]
        K-->M[pages application limits]
        I[Rack::Attack]
        J[Gitlab::ApplicationRateLimiter]
    end
    G --> I
    G --> J

Cloud Connector

flowchart LR
  A[Internet]
  subgraph cf [Cloudflare WAF]
      E[cloud.gitlab.com zone]
  end
  subgraph G [GitLab]
    subgraph cc_be [CloudRun / GKE]
      F[Cloud Connector backends]
    end

    subgraph GKE
      C[gitlab.com]
    end

    subgraph AWS
      D[GitLab Dedicated]
    end
  end

  E --> F
  A -->|cloud.gitlab.com| E
  C -->|cloud.gitlab.com| E
  D -->|cloud.gitlab.com| E

Rate limits exist in multiple layers for GitLab.com, each boxed area above represents a place where rate limits are implemented.

Limits

To minimise duplication, please see the following sources to determine the current active rate limits:

Bypasses

Published rate limits apply to all customers and users with no exceptions.

Please see the Rate Limit Bypass Policy for more detailed information.

Traffic management Rate Limits

Cloudflare

Cloudflare serves as our “outer-most” layer of protection, sitting at the network edge on inbound traffic. We use Cloudflare’s standard DDoS (Distributed Denial of Service) protection plus Spectrum to protect git over ssh.

How rate limits are applied in Cloudflare:

Note: Cloudflare is not application aware and does not know how to map to our users and groups.

Our Cloudflare runbook contains more detail on configuring this layer of our infrastructure.

Changes to Cloudflare rate limits require a Change Request, and should be discussed with the Production Engineering::Foundations SRE team before implementing.

HAProxy

The majority of GitLab’s traffic management rate limits have been moved out of HAProxy and into Cloudflare (see this confidential issue for more details).

However, HAProxy is responsible for GitLab Pages and Registry rate limits, as those components are not fronted by Cloudflare.

HAProxy also handles the bypass header, which allows for a configured list of IP addresses to bypass rate limits in HAProxy or Rack Attack. There are two types:

1. Internal: Full bypass, including some other protections. Only CI runner managers (or similar) should ever be added to the internal list. Managed in Chef.
2. External IPs: An allowlist of IP addresses. Managed in Chef. Requests from these IP addresses are still subject to additional checks, before bypassing the rest of the rate-limiting.

A full list of cases where the X-GitLab-Rate-Limit-Bypass header is used can be found in the HAProxy cookbook.

Customers or internal teams seeking a bypass should refer to the Rate Limit Bypass Policy.

Application Rate Limits

There are multiple application rate limiting mechanisms in place that can often be referred to interchangeably. These allow for more informed traffic management compared to that implemented by Cloudflare as the application has access to user information:

• Rack Attack
• Application Rate Limiter
• Plan Limits

Please see Application Limits Development to contribute application limits to GitLab.

Overview

The rate limit period is 1 minute (60 seconds).

RackAttack

GitLab utilises RackAttack as middleware to throttle Rack requests. These can be configured by extending Gitlab::RackAttack and Gitlab::RackAttack::Request.

For more information about configuring rate limits for a GitLab instance, see the User and IP rate limits doc.

You can read more information about rate limits specific to GitLab.com, alongside RackAttack configuration documentation in runbooks.

ApplicationRateLimiter

The GitLab application has simple rate limit logic that can be used to throttle certain actions which is used when we need more flexibility than what Rack Attack can provide, since it can throttle at the controller or API level. These rate limits are configured in application_rate_limiter.rb. The scope is up to the individual limit implementation and can be any ActiveRecord object or combination of multiple. It is commonly per-user or per-project (or both), but it can be anything, e.g. the RawController limits by project and path.

There is no way to bypass these rate limits (e.g. for select users/groups/projects); when the rate limit is reached a plain response with a 429 status code is issued without rate limiting headers.

Instructions for configuring these rate limits can be found in the GitLab docs.

For more information about introducing new Rate Limits for GitLab, see the Product Processes Handbook page.

Other Rate Limits

GitLab Pages

Because GitLab Pages is not behind CloudFlare and doesn’t have CDN support, rate limits are set in several places:

1. There exist some limits in HAProxy.
2. Within the kubernetes settings for GitLab.com.
3. In the go ratelimiter module for Pages.

For more information, see the Pages Rate Limit documentation.

Registry

Registry is not fronted by Cloudflare (see this confidential issue for full details around why). There is no application-level setting for it either (though there is an open feature request to add that), so the only place where there is rate limiting in place for Registry is in HAProxy.

Headers

The list of semi-standard rate limiting response headers can be found here.

• Cloudflare does not return rate limit response headers on any request.
• RackAttack returns rate limit response headers on throttled requests only.
• ApplicationRateLimiter will return rate limit response headers once this issue is implemented.
• GraphQL endpoints currently do not return rate limit response headers.

Troubleshooting

Observability

The below are internal links to support troubleshooting rate limiting related issues:

• Grafana: Rate Limiting Overview dashboard
• Kibana: Support - Rate limit dashboard
• Cloudflare: Security Analytics dashboard (GitLab Cloudflare account access required)

Investigating RackAttack logs

• json.meta.user field is set if a request is authenticated, and missing if it was anonymous.
• json.env will either be set to throttle or blocklist, the latter which comes from failed authentication bans.

Important Links

• docs: GitLab.com
• docs: Self Managed (and Dedicated)
• runbook: GitLab.com rate limiting
• handbook: Identifying the cause of IP Blocks on GitLab.com

View page source - Edit this page - please contribute.