Managing Limits

Overview

GitLab environments require a multi-layered approach to rate limiting to effectively protect services and resources. Each layer provides distinct advantages and serves as a complementary control in a comprehensive defense in depth strategy to protect our platform.

Follow this guide when introducing or managing limits.

Where to configure the limit

Use the following diagram to determine where the limit should be configured, then follow the corresponding guidance for the appropriate configuration method.

Considerations

Rate limits should be enabled by default. If this is not the case, then this process should be followed for the following cases:

• Introducing new rate limits
• Lowering existing rate limits
• Re-enabling disabled rate limits
• Increasing rate limits

Process

1. Determine if a rate limit already exists
   • Is there an Application limit for this already? What about Cloudflare?
   • Is it possible an existing limit could be adjusted (higher or lower)?
   • See Rate Limiting: Limits for where these limits are configured.
2. Compare proposed limit with existing limits
   • Will customers be negatively impacted by introducing this limit?
   • Are there risks to our platform if we don’t introduce these limits?
3. Where possible, enable in log or track mode first
   • It should be left in this mode for at least one week to understand weekly traffic patterns.
   • This will allow you to gauge potential impact.
   • Use observability tooling (Cloudflare dashboard, logs, metrics) to analyze impact.
4. Produce evidence for the proposed limit
   • Why have you selected the value you have?
   • Using the findings from the previous step to support your proposal.
   • Are there any knock on effects to customers or backend systems we need to consider?
5. Determine a rollout plan
   • Will you use brownouts (temporarily introduce for short periods of time)?
   • If an Application Limit: will you use feature flags?
6. Inform relevant stakeholders of the change
   • Consider informing #customer_success, #support_gitlab-com, and #security.
   • Make sure they have access to documentation, and relevant information to help customers who contact them regarding the limit.
7. Communicate with customers
   • Announce the rate limits on the GitLab blog, see example for Projects, Groups, and Users APIs.
   • If an Application Limit: Document this change in the next release post.
   • Raise a contact request by following the Support: Contacting Customers workflow.
     • Notify of the change, and time frames where possible.
     • Provide guidance on any available remedies, workarounds, or best practices to help mitigate the impact.
8. Document the limits on docs.gitlab.com
   • Make sure the limit is documented, and if it’s configurable, what the default is.
   • Note: This may not always be possible for Cloudflare limits.
9. Follow the Change Management process
   • Any change to rate limits is considered a Criticality 2 change, as they have the potential to disrupt traffic flow.
   • This requires approval from @gitlab-org/saas-platforms/inframanagers

Cloudflare

Enforcing limits at the edge network before traffic reaches the underlying GitLab infrastructure enables us to block malicious traffic before it consumes backend resources, protecting us against large-scale volumetric attacks. This is however limited in the configuration options we can use to limit on - primarily IP address, though there are a few other options.

1. Cloudflare rate limits are managed with Terraform, for GitLab.com through the cloudflare-waf-rules module. Rules added to this module will affect all other services using this module (currently GitLab Dedicated).
2. When creating a new rule, it is advised to instantiate it with action = "log" to analyze impact.
3. After ensuring the rate limit performs as expected, the rule can be set to action = "block" in Terraform.

GitLab Dedicated

Rate limits needed for only GitLab Dedicated Tenants (not GitLab.com) will need to be added to Terraform in Instrumentor. For further assistance please open an issue in the Dedicated Tracker or #g_dedicated_team on Slack (internal link).

Application

Enforcing limits in the application level within GitLab itself enable us to be more opinionated, as they are more context aware (understanding GitLab-specific resources) that provide us more granular control over specific features, and supports the ability to apply business logic and user/project-based dimensions to limiting decisions. New Application Rate Limits are configured in the GitLab application code, and should be introduced following the guide in Product Processes.

For information about rate limits that are currently configured within a GitLab instance, see the Rate Limits docs. For information about rate limits specifically configured to the GitLab.com instance, see Rate Limits on GitLab.com.

Rate limits in RackAttack

GitLab utilises RackAttack as middleware to throttle Rack requests. Most application-level rate limiting is managed with RackAttack. For instructions for configuring already existing rate limits, see the User and IP Rate Limits docs.

New rate limits can be configured by extending Gitlab::RackAttack and Gitlab::RackAttack::Request. Instructions for this can be found in the GitLab Development Docs.

You can read more information about rate limits specific to GitLab.com, alongside RackAttack configuration documentation in runbooks.

Rate limits in ApplicationRateLimiter

The GitLab application has simple rate limit logic that can be used to throttle certain actions which is used when we need more flexibility than what Rack Attack can provide, since it can throttle at the controller or API level. These rate limits are configured in application_rate_limiter.rb. The scope is up to the individual limit implementation and can be any ActiveRecord object or combination of multiple. It is commonly per-user or per-project (or both), but it can be anything, for example the RawController limits by project and path. Currently there is no way to bypass limits created in the ApplicationRateLimiter.

New rate limits may be created in the ApplicationRateLimiter by following the guide in GitLab docs.

Identifying Potentially Impacted Customers

There are several dimensions you can use when identifying impacted customers.

Monitoring and metrics around rate limits on GitLab.com are most easily accessed on the Rate Limiting Dashboard (internal link).

Using the Project ID to identify a customer namespace

If you have access to the project ID for requests identified as potentially hitting rate limits, there are two methods to attribute these to a namespace:

1. Using the API with an admin token

   curl gitlab.com/api/v4/projects/:id
   

2. Using a production Rails console

   [ gprd ] production> p = Project.find(PROJECT_ID)
   => #<Project id:REDACTED redacted/redacted>>
   [ gprd ] production> p.full_path
   => "redacted/redacted"
   

Requesting Further Assistance

If you require further assistance, please open a Production Engineering::Foundations request issue.

View page source - Edit this page - please contribute.