Application Vulnerability Management Procedure

Purpose

This procedure applies to vulnerabilities identified in GitLab the product or its dependency projects and ensures implementation of the Vulnerability Management Standard. This procedure is designed to provide insight into our environments, promote healthy patch management among other preventative best-practices, and remediate risk; all with the end goal to better secure our environments and our product.

Scope

The procedure laid out here applies to vulnerabilities identified in GitLab the product or its dependency projects.

Issues in Scope

During the time period defined for the backlog review issues labeled with bug::vulnerability will be reviewed.

Below is a non-exhaustive list of what may be in scope:

• GitLab (gitlab-org/gitlab)
  • HackerOne
  • bug::vulnerability
• Gitaly
• Pages
• Runner
  • HackerOne
  • bug
• Shell
• Customers
• GitLab Dedicated

Roles & Responsibilities

Role	Responsibility
Appsec Engineers	The stable counterpart for each development team will be responsible for ensuring the backlog is accurate and assisting the development team review and triage the backlog for their team. For teams with larger backlogs, help will be requested from other engineers.
Development	Responsible for reviewing the backlog, triaging, and remediating vulnerabilities according to procedure.
Appsec Manager(s)	Responsible for providing support and help make determinations for findings that need additional input.
Security Leadership	Responsible for making final determination on risk acceptance (exceptions).

Procedure

Vulnerability Reports

Vulnerability reports have many sources but always end up as an issue in GitLab.

Some sources include:

• HackerOne reports, which are imported as GitLab issues in the relevant project (see HackerOne Process)
• Users or customers, often submitted via security@gitlab.com or ZenDesk (which are redirected to HackerOne or the GitLab issue tracker)
• Valid scanner findings, which are imported directly from the vulnerability reports page (see this documentation)
• Confidential issues created in GitLab repositories, typically by users, customers, or GitLab team members

Engaging Product Teams

Once the issue is created, a security engineer will set the correct ~group:: and ~devops:: labels and @-mention the engineering and product managers for the team that owns the affected feature.

The security engineer can try to find a group based on the last people who modified the code by using the git blame feature or by asking in the #development Slack channel.

Occasionally the receiving group won’t be the correct one, but the security engineer and the group can work together to find the appropriate group to reassign to. A ~devops:: label is good enough if a ~group:: cannot be determined. In this case, the security engineer should assign the issue to the stage’s director/Sr. EM.

If an issue is judged to not be a vulnerability but rather a security enhancement, it will be made public, labeled ~"type::feature" and the prioritization will be left to the product team as with any other regular issue. (See: Vulnerability vs. Feature vs. Bug?). For vulnerabilities, the security engineer will set the ~severity:: and ~priority:: labels and those will determine the due date and SLO for the issue as described in the main Product Security page of the handbook.

If the report is a ~severity::1 report, there is a special procedure to follow to engage SIRT and leadership.

Fixing The Vulnerability

Vulnerabilities in the GitLab web application and all other products bundled with our Omnibus packages are fixed following the patch release process. GitLab has two types of security releases: a regular monthly release, and a critical security release whenever a ~severity::1 issue is reported. See the Security Releases section of the main Product Security handbook page for more information.

If you’re working on a vulnerability that you feel could be treated as a security enhancement and skip the security release process, feel free to ping @gitlab-com/gl-security/product-security/appsec.

GitLab products that follow an independent release cycle and are not aligned with the monthly GitLab security release are also required to have a process for developing security fixes and disclosing them to the customers. Either the GitLab patch release process or a recommended bare minimum process as below can be followed:

• Do not fix the security issue in public so that the vulnerability and the fix are not disclosed before it is ready to be publicly disclosed. Use a security mirror project under the private Security group for working on the security issue. In case the product is maintained in a private project then the fixes can be done directly in that project.
• Get the security fix reviewed by Appsec team.
• Request the Appsec team create a CVE request for the fixed vulnerability if the product is released to customers for their own installation.
• Notify customers about security fixes.
• Disclose the security issue following the disclosure policy.

Fixing In Public

The Application Security Team may give approval for a security issue to be addressed in public, either wholly or in part. Public issues and MRs must never include information that should remain confidential. Also, backports are not required for security issues fixed in public. Application Security engineers should:

• First, evaluate whether the entire issue can be reclassified and made public.
• Second, consider creating a public linked issue for specific work items.
• Finally, if the issue must remain confidential and creating additional issues is not desired then update the issue with a clear implementation proposal on what can be done in public and what must remain confidential, and add the ~"security-fix-in-public" label.

The security team monitors for unintended information disclosure via public MRs referencing confidential issues and will delete public branches and MRs that do not follow this process.

Notifying GitLab Users

Fixed vulnerabilities are mentioned in the security release blog posts and it’s possible to receive notifications either through email or RSS.

Issues that were handled outside of the security release process can be mentioned in the regular release posts if the product manager deems it appropriate.

Security issues fixed in GitLab products may be mentioned in the security release blog post only if they are released along with the monthly GitLab security release. All security issues that need to be mentioned in the security release blog posts are expected to be linked to the ~"upcoming security release" issue. If linking is not possible, please contact the Security release managers to include security issue in the security release blog post.

GitLab products that follow an independent release cycle and are not aligned with the monthly GitLab security release could make use of CHANGELOG.md file or a separate security advisory file in the project repository to publish details about security fixes in their new releases.

Backlog Review

The application security team will do reviews of the vulnerability backlog to ensure that it accurately reflects the vulnerabilities present in the product. The frequency of these reviews will be determined based on compliance requirements or as determined by the team. The process is intended to determine if open issue are still valid, can be closed, or are eligible for risk acceptance.

Labels

Scoped labels will be used to categorize each issue. The labels are as follows:

• ~security-backlog::valid
• ~security-backlog::closed
• ~security-backlog::reclassified
• ~security-backlog::risk-acceptance
• ~security-backlog::needs-input
• ~security-backlog::review-complete

Issue Review Process

For each issue, it’s categorization will need to be determined. The following are guidelines the appsec engineer should follow to determine the appropriate category, and the steps that should be taken.

Review Complete

The ~security-backlog::review-complete label should be applied to issues that have been reviewed to help us filter and track.

Valid

An issue is valid if it is a vulnerability that continues to be present in the product.

• Apply the ~security-backlog::valid label

Closed

An issue is closed if it is determined that the issue has previously been fixed, is a known duplicate of another issue, or is not longer relevant due to evolution of the product.

• Apply the ~security-backlog::closed label
• Leave a comment stating the determined reason and close the issue.
  • If duplicate: Use the /duplicate <#issue> slash command to close issue

Reclassified

An issue is reclassified if it can be considered a ~type::feature, ~type::maintenance, or a non-security ~type::bug, based on the evolution of the product, our threat model, or our understanding of the issue. Use the Vulnerability vs. Feature vs. Bug questions below to guide you.

• Apply the ~security-backlog::reclassified label
• Remove severity and priority if assigned by a Security Engineer
• Either:
  • Remove the ~type::bug label and apply the appropriate label.
  • Remove the ~security label.
• Leave a comment stating the determined reason for the reclassification.
• Review the issue for any confidential information and make it public if possible.

Needs Risk Acceptance

The general guidelines for issues eligible for risk acceptance are laid out below. There is no perfect formula, so the judgement of the engineer will be relied upon.

• Long-lived, with little evidence of an upcoming fix
• Could it be made public with no impact?
  • Could it be labeled ~“Seeking community contributions”?
• Could it be ~security::reclassified as ~"type::maintenance" by the product managers based on a use case? (e.g. Guest user can see a count of X but don’t have access to view individual X)
• Is it S3/P3 or lower?

If an issue is eligible for risk acceptance:

• Add comment stating why the issue should be considered for risk acceptance
• Apply the ~security-backlog::risk-acceptance label
• Security will follow Vulnerability Management Standard Risk Acceptance (Exceptions)

Needs Input

An issue needs input if the appsec engineer cannot determine the appropriate category.

• Apply the ~security-backlog::needs-input label

Vulnerability vs. Feature vs. Bug?

When receiving or reclassifying issues, especially S4/P4s, it can be difficult to determine whether it’s a vulnerability, an enhancement, or a bug. Here are some questions which might help:

• Is it exploitable and can it affect the confidentiality, integrity and/or availability of GitLab’s systems and data? Can you create a CVSS score for it? Vulnerability
• Have similar issues been treated as vulnerabilities, and does that still make sense? Vulnerability
• Is the issue introducing something new which improves security or defense in depth? Enhancement (probably ~type::feature)
• Is the functionality in question acting as designed, but we’re refining that to improve security or defense-in-depth? Enhancement (probably ~type::maintenance)
• Is the functionality not acting as designed, but there’s no security impact and you can’t create a CVSS for it? Non-security Bug
• Are we removing a vulnerable dependency, or piece of code, which GitLab is/was not using in an exploitable way in any of the currently supported versions? (A “theoretical” / unexploitable vulnerability). Enhancement (probably ~type::maintenance)

If it’s unclear, err on the side of caution, and treat the issue as a vulnerability. The Engineering Work Type Classification page may also serve as a guide.

When does a security enhancement become escalated to a vulnerability?

Ensuring that appropriate measures are taken to mitigate the associated risks effectively, a security enhancement must be escalated to a vulnerability when any of the following conditions are met:

• When there’s evidence demonstrating that it can be combined with another vulnerability, leading to a Critical or High severity security issue or incident
• When there’s evidence that the issue is actively exploited in the wild

Updating CVSS after patch

Occassionally after a release has occurred, a discussion in the AppSec team may lead to a better understanding of a bug and its CVSS. Follow the S1/P1 process if an issue is increased in severity to Critical. Otherwise:

• Update the report in HackerOne, if appropriate, and award any additional amount
• Make an MR in the cves-private project to update the CVE record, and mention @gitlab-org/vulnerability-research
• Make an MR to update the security release blog post
• Identify and open an MR for improvements to the triage or remediation processes, if appropriate

Exceptions

Refer to Vulnerability Management Standard Risk Acceptance (Exceptions)

References

• Vulnerability Management Standard
• Infrastructure Vulnerability Management Procedure
• Bug Bounties