Vulnerability Management

Vulnerability Management is the continual process of identifying, prioritizing, mitigating and remediating vulnerabilities. At GitLab we identify vulnerabilities in a number of different ways depending on the component being analyzed. This process and assosciated tooling is owned by the Vulnerability Management team.

This page primarily outlines our vulnerability management standards and processes at GitLab. The GitLab Vulnerability Management Standard defined on this page is a consistent process to identify, document, categorize, manage, and remediate all vulnerabilities that impact in-scope GitLab-managed systems and software projects. The goal is to reduce the risk relating to security vulnerabilities which could impact the achievement of GitLab’s goals. For details on vulnerability management procedures, refer to the references section at the bottom of this page.

Scope

This standard applies to all systems which store, process or transmit GitLab production and/or GitLab customer data, as well as GitLab-managed software and software dependencies, based on the GitLab critical system tiering methodology. Specifically:

Roles & Responsibilities

Additional Notes for Vulnerability Triaging

1. The VulnMapper creates vulnerability issues if not available already, adds appropriate severity, priority and other necessary labels, and reports Deviation Request issues of vendor dependencies automatically.
2. For the vulnerabilities that VulnMapper cannot process automatically, the Vulnerability Management Team logs vulnerability issues with necessary information such as CVSS score, severity and priority, package name and version etc., and reports Deviation Request issues with risk adjustment evaluation when applicable. The AppSec team will collaborate and provide support to the Vulnerability Management team as needed.
3. The Vulnerability Management Team assigns vulnerability issues to a development group based on the best guess.
   1. Note that group::not_owned was deprecated. There isn’t a default development group to assign vulnerability issues to and here’s how development groups handle vulnerabilities. Make the best effort guess and avoid assigning the majority of vulnerabilities to a single development group.
   2. Vulnerability triage is a shared responsibility although the Vulnerability Management Team is the DRI. AppSec, InfraSec and respective feature domain development groups are responsible for providing evidence and/or analysis of GitLab-specific impact during vulnerability triage upon the Vulnerability Management Team’s request. Vulnerability tracking issues should be assigned to the appropriate AppSec stable counterpart or development group for further analysis.
4. A typical workflow of triaging vulnerabilities will look like -
   1. Identify those issues where we know 100% sure that it goes to a specific group and have the VulnMapper Bot assign the issue to that development group.
   2. For all issues the VulnMapper Bot cannot assign and the Vulnerability Management Team is uncertain, assign those to the AppSec team.
   3. The Appsec counterpart could then assign the issue to the appropriate group after their triage.

Standard

• Where possible, GitLab should be used as the source of truth for vulnerability detection and management. GitLab issues are currently used to track vulnerability management activities.
• Application (WebApp and container) vulnerability scanning must occur on a minimum of a monthly basis, and use scanners and scanner configuration tuned for the code being scanned. This includes enabling appropriate dependency scanning and SAST templates.
• Container images shipped by GitLab should be scanned at a minimum with GitLab container scanning before being shipped.
• Infrastructure (operating systems and databases) vulnerability scanning must occur on a minimum of a monthly basis.
• Authenticated/credentialed scans from a privileged network domain, in addition to external scanning of infrastructure, should be performed wherever possible.
• Vulnerability scanners must be up to date (the latest operable version), and hardened to resist unauthorized use or modification.
• Third party penetration testing must be performed annually.
• Vulnerability remediation SLA timeframes begin as soon as a vulnerability is detected by a scanner, or reported by a 3rd party or team member.

Tracking Issue Labels

GitLab is used to track vulnerabilities as both vulnerabilities and issues, across GitLab’s infrastructure and projects. As part of this process, various standard labels are applied to track the status of managing vulnerabilities through to mitigation and remediation.

Please see the main entry detailing the standard Vulnerability Management Labels for information on how these labels are used by groups, security and engineering tooling, and the vulnerability management process.

Remediation SLAs

The following timelines or service level agreements (SLAs) are based on many factors - such as regulatory compliance, customer SLOs & SLAs, vulnerability impact, scope, prevalence in GitLab environments, impact if exploited, and defining reasonable turn-around times for mitigation and remediation to protect GitLab and our customers. All of these factors will be considered when mapping the priority to GitLab’s priority labels. All components in scope of Vulnerability Management are subject to the same SLAs. The SLAs are as follows:

* A vulnerability’s severity is assigned using the first available value from this list:

1. CVSSv3 base score.
2. CVSSv2 base score.
3. Base risk score reported by a vulnerability scanner.

Tracking Issue Lifecycle

The typical lifecycle of a vulnerability detection at GitLab, at a high level, follows the following steps:

1. Vulnerability detected by scanner or otherwise reported
2. A GitLab issue is created to track the vulnerability
3. The vulnerability is remediated within SLA
4. The issue is closed

Depending on the environment and what is being scanned, the tracking issue will be handled by different GitLab team members. Issues may also be enriched by different automated tools, such as (VulnMapper, which assist with streamlining the detection and remediation process. When an issue cannot be remediated within SLA, additional procedures exist for adjusting or exempting vulnerabilities from SLA. See the section in this page on Risk Acceptance & SLA Exception for detail.

Automation

Vulnerability management maintains automation tooling which intends to automate as much of this standard and the related procedures as possible. We’re always iterating on ways we can save time for development groups whilst keeping GitLab and our users safe.

The key features of our main automation VulnMapper are:

• Correlation of Container Scanning findings with assosciated vulnerability advisory data
• Automated labelling of container scan findings for FedRAMP (Red Hat) image findings with fix availability information
• Automation of Operational Requirement and Risk Adjustmenet Deviation Requests for FedRAMP findings

The key features we are developing currently are

• The automated creation & triage of vulnerability tracking issues from scan findings
• Non-FedRAMP finding support (Debian, Alpine based container scan findings, for example)
• Automation of non-FedRAMP SLA exceptions for commonly approved scenarios (vendor package availability, vendor risk adjustment)

Current issues & limitations are tracked on the private VulnMapper issue tracker. Some of the key limitations currently are:

• Base Container fix availability data and labels are not as consistently applied as Package fix availability data, which is very reliable
• Fix availability information requires the OS package information to be determined, for correlation with advisory data

For any feedback or ideas for improvement of the vulnerability automation used at GitLab, team members can file an issue.

Closing tracking issues

To reduce the number of non-actionable issues being tracked by development groups, we’ve looked at the various situations where issues can be safely closed, without compromising our ability to react to changes in fix availability or mitigating factors.

Vulnerability tracking issues can be closed when:

• A vulnerabile dependency or package has been updated, and an updated software artifact (container image, package) has been shipped
• A vendor dependency in non-FedRAMP container images will not be fixed by the software vendor (i.e. Debian) as it is not impactful
• A vendor dependency in a FedRAMP container image will not be fixed by the software vendor (i.e. Red Hat) as it is not impactful, and a DR has been opened and linked to the tracking issue
• There is a false positive detection in a non-FedRAMP container image or project and an exception issue has been created or updated and linked to the tracking issue
• There is a false positive detection in a FedRAMP container image and a DR has been opened or updated and linked to the tracking issue

Example vulnerability lifecycles

An impactful vulnerable package is detected by Tenable.io on a GitLab-managed infrastructure system

1. An issue is created by tenable_gitlab_sync
2. The issue is remediated as part of GitLab’s production patching cycle
3. The issue is closed

An impactful vulnerable package is detected in a non-FedRAMP GitLab container image

1. A container scan finding is created in GitLab
2. An issue is created to track remediation of the finding
3. The vulnerable package is updated
4. An updated image is shipped
5. The issue is closed

An vulnerable package which will not be fixed by the vendor is detected in a non-FedRAMP GitLab container image

1. A container scan finding is created in GitLab
2. An issue is created to track remediation of the finding
3. No fix will be made available by the vendor, as indicated by the automated issue labels or manual research
4. The linked vulnerability finding from the container scan report is dismissed
5. The issue is closed

An impactful but unfixed vulnerable package is detected in a non-FedRAMP GitLab container image

1. A container scan finding is created in GitLab
2. An issue is created to track remediation of the finding
3. No fix will be made available by the vendor, as indicated by the automated issue labels or manual research
4. The linked vulnerability finding from the container scan report is dismissed
5. The issue is closed

An impactful vulnerability in a language library used as a dependency in a GitLab software project

1. A dependency scan finding is created in GitLab
2. An issue is created to track remediation of the finding
3. The dependency is updated
4. An updated version of the software is shipped
5. The issue is closed

An impactful but unfixed/unfixable vulnerability in a language library used as a dependency in a GitLab software project

1. A dependency scan finding is created in GitLab
2. An issue is created to track remediation of the finding
3. The responsible development group works with the library maintainer or GitLab owner of the dependency to understand fix availability
4. Either there is no response or a response indicates a fix will not be available with SLA
5. Mitigating controls are explored to either validate or mitigate the impact of the vulnerability
6. Mitigating controls or switching to an alternate library can not be implemented within SLA
7. An exception issue is opened detailing the options explored to extend the SLA as needed
8. An updated version of the software is shipped with either a fix or mitigation in place
9. The issue is closed

An impactful vulnerable package is detected in a GitLab FedRAMP container image

1. A container scan finding is created in GitLab
2. An issue is created to track remediation of the finding
3. The issue is automatically labelled by VulnMapper with the appropriate Vulnerability Management Labels based on fix availability data published by Red Hat
4. A fix is available as indicated by the labels, and the responsible development group updates the image with the fixed package
5. The issue is closed once the updated image is shipped to FedRAMP environments

A vulnerable package which the vendor has decided will not be fixed is detected in a GitLab FedRAMP container image

1. A container scan finding is created in GitLab
2. An issue is created to track remediation of the finding
3. The issue is automatically labelled by VulnMapper with the appropriate Vulnerability Management Labels based on fix availability data published by Red Hat
4. If a fix will not be made available by the vendor as indicated by labels or manual research, a Deviation Request needs to be created
5. A DR is created, either via automation in VulnMapper, or by the responsible development group
6. Once the issue is linked to the DR issue, the vulnerability tracking issue may be closed

A vulnerable package which has not been fixed yet by the vendor is detected in a GitLab FedRAMP container image

1. A container scan finding is created in GitLab
2. An issue is created to track remediation of the finding
3. The issue is automatically labelled by VulnMapper with the appropriate Vulnerability Management Labels based on fix availability data published by Red Hat
4. If the fix will likely not be made available by the vendor before SLA is due (labels indicate it is unfixed), a Deviation Request needs to be created
5. A DR is created, either via automation in VulnMapper, or by the responsible development grou
6. The issue remains open, and once a fix is available, labels are updated by VulnMapper or a team member
7. The vulnerable package is updated to the fixed version and a new image is shipped
8. When the issue is remediated & deployed, the tracking issue can be closed

Risk Acceptance & SLA Exceptions

We understand that it is not always technologically feasible to use the latest dependency, package or container base image due the need to ensure the stability and performance of GitLab as a product and a service. Business decisions may be made to not remediate a vulnerability, or delay remediation, because remediation would impact performance or reliability too greatly, or introduce significant risk of doing so. Low risk vulnerabilities that may not be prioritized within the remediation SLAs should have an exception created and approved - documenting the low likelihood of exploitation due to layered security, other compensating controls, complexity of exploitation, etc. With this in mind we have a vulnerability exception process.

There are currently two processes, depending on whether or not a finding is for a project or image intended for use in FedRAMP environments or not. In both cases, the following examples are some of the situations in which it may be appropriate to open a Deviation Request (FedRAMP) or Exception (non-FedRAMP) to adjust the SLA for a detected vulnerability.

Example scenarios for risk acceptance & SLA exception

Mitigating circumstances or controls

Sometimes vulnerabilities will be detected on an infrastructure system, software project or container image for a component which is used in such a way that the vulnerability is either not present or not exploitable. In these cases, the stated impact of the vulnerability does not match the actual impact based on how the component is used. An example: A vulnerability in the XML-import functionality of a software library used as a dependency has an XXE vulnerability, however the project disables or does not use the XML import functionality of the library.

In non-FedRAMP environments & software artifacts, if updating the component can be safely delayed to make way for higher priority engineering work, then an exemption issue is appropriate. Analysis explaining the mitigating circumstances or controls will be required and reviewed to make certain remediation can be safely delayed.

In non-FedRAMP environments, this is not a scenario which is appropriate for exemption, as it will be rejected by the reviewing auditors we report vulnerability detections to.

False positive detections

Sometimes vulnerabilities will be found in software by vulnerability scanners which are not actually present. This can be based on innacurate detection methods, or limited context around the detection due to limited scanner functionality. In these cases, there is no impact and the vulnerability can be treated as a false-positive detection.

In non-FedRAMP environments, an exception can be created to temporarily extend the SLA until the false-positive detection can be addressed in the scan configuration or scanner itself. If this is not feasible, then a permanent exception may also be appropriate.

In FedRAMP environments, a False Positive Deviation Request can be opened to address not meeting SLA for this finding.

Risk & severity adjustments

Software which has been repackaged by a vendor (such as operating system packages) often have a different severity rating for vulnerabilities in a package based on how the software has been packaged and the default configuration. Mitigating controls and circumstances can also exist which impact the actual severity of a vulnerability.

In these cases, an exception to typical SLAs can be appropriate.

In non-FedRAMP environments, an exception can be created to extend the SLA and adjust the severity of the vulnerability.

In FedRAMP environments, there are much more specific rules for adjusting the risk of a finding, which are detailed on the Risk Adjustment section of the Deviation Request Procedure.

Vendor dependencies & fix availability

Vendor dependencies, depending on the software vendor, have their own lifecycle of reporting, analysis, and scoring which is represented in vendor advisories. For example, Red Hat releases RHSA (Red Hat Security Advisories) detailing the outcomes and fixed versions of software packaged and distributed by Red Hat. The outcome of this process can often mean that a fix will not be made available, as the vendor has found their packaged version of the software to not carry the vulnerable code, not be exploitable under default configuration, or that exploitation requires features to be enabled which are disabled in their packages.

In the situation where releasing a fix is delayed, labels or manual research of advisory information will indicate that a fix is not yet available or “unavailable”. An exception to extend the SLA is appropriate in this case.

In the situation where a fix will not be released at all, as indicated by manual research or advisory information or automatically applied “will not be fixed” labels, an exception to be exempt from SLA is appropriate in this case.

In non-FedRAMP environments, an exception issue should be created for either a temporary or permanent SLA exemption, based on findings being either “fix unavailable”, or “will not be fixed” respectively.

In FedRAMP environments, a Deviation Request should be opened. There are specific rules around how this should be handled, detailed in the Operational Requirement DR Procedure.

Operational and technical requirements

When operational or technical requirements mean that resolving a vulnerability within SLA is either not technically feasible, or represents risk to the stability or availability of GitLab software or systems, an exception SLA may be appropriate if other mitigations can be put in place.

In non-FedRAMP environments, an exception issue should be created so Security and the responsible development group can collaborate on potential mitigations and if appropriate extend the SLA for remediating the finding.

In FedRAMP environments, this is not considered a valid reason to extend or exempt the SLA of a finding. Security should work with development groups in this case to find alternative approaches which meet SLA.

SLA Exception Procedures

FedRAMP Deviation Request Procedure

For FedRAMP related vulnerabilities, all exception requests must follow the FedRAMP Vulnerability Deviation Request Procedure. For vulnerabilities in 3rd party (non-GitLab) dependencies, the SLA remediation timeframe re-starts once a patch/fix is released/published.

Non-FedRAMP Risk Acceptance & SLA Exception Procedure

If you’ve identified a vulnerability that is a candidate for an exception, in a non-FedRAMP image, project, or infrastructure system, please open a vulnerability exception issue using the exception template.

Please fill all out the pertinent information requested in the template. For reference, the information required is as follows:

• Vulnerability description, including CVE or other identifiers
• Priority/Severity of Vulnerability
• Original remediation due date per above SLAs
• Length of requested exception
• List of applicable assets or projects (hosts, container images, GitLab repositories, etc)

You will need to provide a high level justification of the exception and why it is appropriate. It is also helpful (and welcome!) if you can provide any technical detail you think helps explain any mitigations, compensating controls, or any other analysis which adds context around lessened impact of the vulnerability.

All of this helps us make quick decisions around how appropriate an exception is, and if there is any other way we can help to mitigate the vulnerability within SLA.

Vulnerability exceptions can be grouped into the following categories:

The listed labels are useful for reporting on our SLA compliance and the assosciated risks to GitLab, and if you are not sure which one should be applied, please ask when opening the exception request and the Vulnerability Management team will apply the correct one.

Exception length restrictions

We currently allow exception lengths based on priority/severity as follows:

For exceptions which are not time based (i.e. for false positive detections) we have permanent exceptions. If you think a permanent exception might be appropriate, let us know when opening the issue and we’ll review to see if a permanent exception might be appropriate. Generally speaking, time based exceptions are preferred as they give us an opportunity to explore options with you to address a finding, even if it is a false positive.

Exception approvers

After the issue is open, the requestor should assign the due date to match that of the associated remediation issue and assign to the proper approver. The request will be reviewed by a memeber of the Vulnerability Management team, and approved if appropriate. Please feel free to reach out in the #g_security_vulnmgmt or #security channels on Slack if you have any questions or need to bring any more immediate concerns to our attention.

Contact

If you have any questions or concerns related to vulnerability management please contact the Vulnerability Management Team in the #g_security_vulnmgmt or the #security channels on Slack, or you can open an issue in the Vulnerability Management issue tracker. All work being done to improve this process is also tracked in the issue tracker, and we’d love your feedback.

Any questions regarding ownership around vulnerability management can be answered in GitLab’s tech stack documentation.

Exceptions

Exceptions to this standard will be tracked as per the Information Security Policy Exception Management Process.

References

• Parent Policy: Information Security Policy
• Application Vulnerability Management Procedure
• Infrastructure Vulnerability Management Procedure
• Bug Bounties