Investigate Blocked Pipeline
GitLab.com uses an external pipeline validation service to verify pipelines as legitimate activity. This includes pipelines run on specific runners. If a customer is reporting that their pipelines are not running we can review the logs on our end to see what activity was logged in the validation service.
Because of the sensitive nature of the rules we use to detect abuse, in most cases we can provide only general feedback to the customer as to what
.gitlab-ci.yml is causing their pipelines to be blocked. When in doubt, request
@trust-and-safety to review.
What does it look like for a user?
A pipeline may have been blocked by the pipeline validation service if an expected pipeline is not created. A pipeline might be generated via:
- a new push
- a manual action
- a scheduled pipeline
- the API
There may be a different underlying cause though, so it’s important to validate by searching the logs.
In some cases a user may see an error:
Pipeline cannot be run. External validation failed.
Identifying a block
- Identify the affected
- Search Kibana in the
pubsub-pvs-inf-gprd*index in the appropriate time frame for:
- Observe the
rejection_hintfields to understand why the pipeline was blocked.
For more, see the pvs-runbook
The rules are documented in the Trust and Safety - Repository Validation Service repository(GitLab internal only)
Trends / High Priority cases
When rules change or are updated there’s the potential for a batch of false positives. If there are many reported cases in a short period, report an incident.
It will be helpful to:
- have a list of tickets reporting blocked pipelines
- verify that they were blocked by the pipeline validation service
- identify any commonalities
Responding to Customers
We have the
General::Blocked Pipeline macro which will ask users for their
.gitlab-ci.yml (or permission to access it) and explains that their pipeline may have been blocked, in addition to tagging the ticket. Make sure to remove any items the customer has already provided in their initial response.
Once you’ve verified the block you can respond with postive affirmation that the pipeline was blocked as a result of detected abuse.
In responding avoid mentioning any of the specific rules. For example, say we had a rule that filtered out imaginary crypto mining script
use-gitlab-to-mine-crypto.sh. You might respond:
It looks like an item in the
scriptsection of your
.gitlab-ci.ymltripped our anti-mining measures. In general, you’ll want to remove any references to mining tools.
I see you’ve got an item on line 22 that calls a script with a name similar to scripts that have been flagged as abuse in the past. If you remove that line, I suspect your pipeline will pass validation.
You would not want to say:
We scan all pipelines for the string
.gitlab-ci.ymlcontains this string, so please rename that script to avoid being flagged.
Reporting a false positive
Create an issue with Trust and Safety. Include the username of the affected user, a copy of the JSON output from Kibana that shows the block happening, and a link to the Zendesk ticket reporting the request.