Identifying the Cause of IP Blocks on GitLab.com

At times, users of GitLab.com can find that their IP address has been blocked due to rate limiting. Currently, rate limit parameters on GitLab.com are best described on the GitLab.com settings docs page. When this happens we may be able to determine what caused a block and relay that information back to the user.

See also opstrace#1949 for possible future improvements.

Responding

A standard response is available as a macro: Support::SaaS::Temp IP Ban)

Please also see the log requests workflow for what information we can provide when responding.

Searching

Search Condition

Start by adding a positive filter on json.remote_ip for the IP address provided by the user:

You can then drill down from there with positive and negative filters on fields to get the best results.

Checking for Rack Attack Blocks

It can sometimes be unclear if a user has actually been blocked by our end or not. If they’ve been blocked by Rack Attack, we should be able to locate requests in Kibana that were blocked because of it.

To do so, enter the IP address into the main search field and set a positive filter on json.message for Rack_Attack.

You should see results similar to the following:

The existence of these results tells us that this user was blocked by Rack Attack and we can add the json.fullpath field to see which exact path on GitLab.com each request tried to access.

It’s common to see multiple failed authentication requests (401) trigger a Rack Attack block which causes a 403 Forbidden message. We block IP addresses if we receive 30 failed requests from a single IP in a three minute period. It’s worth noting that by default, Git operations are first tried unauthenticated so it’s expected to see two 401 responses for every Git operation.

Rack Attack can also throttle traffic. If that is the case, this is recognizable by the HTTP 429 response code. The preferred solution to this is to have the user make fewer requests. If that is not possible you can create an infrastructure issue with this template.

Fields

Primary

The following fields are the best to add to your search query in order to get the most important details on multiple requests at a glance.

• json.status - Outputs the HTTP status code that was returned for the request. We’re usually looking for 401 (Unauthorized) and/or 403 (Forbidden).
• json.path - The path on GitLab.com that was accessed by the request or the API endpoint that was hit.
• json.method - Can be either GET, POST, PUT, PATCH, or DELETE. The first three are the most common.
• json.env - Can be blocklist, throttle or track. track is used for diagnostics when changing rate limiting settings and does not affect users: from a support perspective, ignore track. blocked happens in reaction to too many failed authentication attempts, for example with automated Git HTTP traffic. throttle means a user or IP is making too many requests per minute.

Secondary

These fields can be helpful but aren’t essential.

• json.controller - Gives you a clue as to what part of GitLab.com was being accessed by a particular request.
• json.params - Shows what user made the request, what action was taken, and on what resource it was taken on. This field shows what repository was targeted for requests to the container registry.
• json.matched - This is the name of the Rack Attack rule used to limit this request (when json.env is throttle). This can help you find out which of the current limits the request surpassed.

Common Causes

Container Registry

Numerous failed pushes or pulls to registry.gitlab.com can result in an IP block, from the docs:

GitLab.com responds with HTTP status code 403 for 1 hour, if 30 failed authentication requests were received in a 3-minute period from a single IP address.

You can list all log results for hits on the container registry by searching for the provided IP address and setting a positive filter on json.path for /jwt/auth.

Useful Fields

• json.status
• json.path
• json.params
• json.controller

Failed Push and Pull Examples

Push

Failed pushes to the registry will always have JwtController for the json.controller field and /jwt/auth for the json.path field. Watch for :push,pull in the json.params field, indicating that the request is for a push.

A failed push will look like the following in Kibana.

Pull

Similar to a push, failed pulls from the registry will always have JwtController for the json.controller field and /jwt/auth for the json.path field. However, in json.params only :pull will be present.

A failed pull will look like the following in Kibana.

gitlab-ci-token Pulls

The gitlab-ci-token user is exempted from rate-limitting.

LFS

Pushes or pulls to repositories containing LFS objects can result in an IP block if the user is unauthorized.

Useful Fields

• json.action
• json.controller
• json.method
• json.status

Examples

Push

Failed LFS pushes will always have upload_authorize in the json.action field, Projects::LfsStorageController for the json.controller field, and PUT for json.method.

A failed LFS push will look like the following in Kibana.

Cloning Fails

An IP can become blocked if a repository is cloned via HTTPS without authentication enough times.

The request will look like this:

Useful Fields

• json.status: 401

• json.controller: Projects::GitHttpController

• json.action: info_refs

• json.path: /namespace/project.git/info/refs

• json.ua: If the request did not come from a browser this will reveal the operating system of the machine that attempted the clone along with the version of Git installed on it. Examples: git/2.11.1.windows.1, git/2.17.2 (Apple Git-113), git/2.17.1 (Ubuntu 18.04 LTS), etc.

You should also see git-upload-pack in the json.params field.

Project Export Rate-Limiting

An IP can become rate-limited if a customer attempts to export or download project exports too rapidly. See Project Import/Export

Useful Fields

• json.method: Application_Rate_Limiter_Request

• json.controller: ProjectsController

• json.action: download_export

• json.path: /namespace/project/download_export

Handling Gitlab.com “Access Denied” errors (CloudFlare Block)

There may be cases where a user is being blocked by CloudFlare and they are not being blocked due to rate limiting. You can typically request a screenshot of the CloudFlare “Access Denied” page or have the customer perform a curl with the -i flag to retrieve the relevant headers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

curl -i --header "PRIVATE-TOKEN: *****" https://gitlab.com

HTTP/2 403
date: Thu, 18 Nov 2021 13:48:09 GMT
content-type: text/plain; charset=UTF-8
content-length: 16
x-frame-options: SAMEORIGIN
referrer-policy: same-origin
cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
expires: Thu, 01 Jan 1970 00:00:01 GMT
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=AbCdeFgXXXXX"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0.01,"report_to":"cf-nel","max_age":604800}
strict-transport-security: max-age=31536000
x-content-type-options: nosniff
server: cloudflare
cf-ray: 6b01a12345abcde-KBP
error code: 1020

Note the HTTP 403 response and error code 1020.

Once you obtain this information you should open an issue in our Reliability tracker providing the cf-ray ID and the timestamp (date) to request that the IP address block be removed. You can also consult the #infrastructure-lounge Slack channel with the open issue for further assistance. Some blocks may happen as a result of a mitigation effort, so you may want to verify that a contact request is not open on the internal board.

Note that IP addresses may be blocked if they are identified as being from a current US embargoed country as per our Terms of Use. Blocks are done automatically through CloudFlare’s GeoLocation block methods and cannot be changed. You can enter an IP address to determine how it is classified and verify against the list of countries. A user can consider requesting a data correction of their IP address but it is not guaranteed and GitLab has no control over this process.

Applying for an exception

If a customer has concerns about being rate limited, work with them as much as possible to lower their traffic from a single IP address.

In rare cases, all traffic must be routed through a single IP address. If this is a concern for the customer, please work with a support manager to consider opening an issue in the reliability tracker to ask for an exception.