Verifying Security Fixes

The review of a fix by an application security engineer is triggered by the engineer implementing the fix. As the engineer engaging in the verification, these are the steps that should be followed:

  1. Doing a code review from the security perspective.

  2. Validating the security fix using the Docker image generated by package-and-qa, see the section below for more details.

  3. Once the validation is done, update the security issue with a link to the CVE.

    • The link goes in the Summary > Links table of the security implementation issue, next to the cell labeled CVE ID request
    • If the fix does not require a CVE, post in the security implementation issue that no CVE will be requested with the reason why.
    • If a CVE was created on import from HackerOne, double check the submission contains up-to-date details

Note: Approval is only required for the merge request targeting master, it’s not required for the merge requests targeting a versioned stable branch (X-Y-stable-ee) . Note: The process described above is particular to validate GitLab fixes. The process to verify fixes for other subcomponents (Omnibus GitLab, Gitaly, and GitLab Pages) might be different.

Validating security fixes using Gitpod

  1. Start a new Gitpod instance https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/gitpod.md
  2. When your Gitpod instance has started, download the patch from the security MR by clicking on the Code option on the top right of the MR.
  3. In your Gitpod instance on the terminal: cat > mr.patch <hit return> <paste contents> <hit control+d>. Then apply the patch: git apply mr.patch
  4. Restart GDK on Gitpod: gdk restart
  5. Make the GitPod webserver reachable: In the Ports pane of Gitpod, find the row with port 3000 and click the lock icon to make the port public. In the same Ports pane, and the same port 3000 row, click the URL to visit the GitLab instance.

Validating security fixes using package-and-qa

The relevant application security stable counterpart is responsible for validating the security fixes using the package-and-qa process described below. The security engineer that triaged the vulnerability report can be involved as necessary or be considered a backup in the event that the stable counterpart is unavailable.

Note: The Delivery team recorded a video about summarizing this process, it’s highly encouraged for AppSec team members to see it.

The package-and-qa build runs end-to-end tests against an Omnibus package that is generated from the merge request changes. In order to validate that the security fix introduced on the merge request remediates the vulnerability, Security engineers will use this package to spin up a docker image in their local environment.

For that, Security Engineers need to follow these steps:

  1. On the security merge request, click on the qa stage and then trigger the package-and-qa build.
  2. Internally the package-and-qa build will trigger a pipeline on the [Omnibus GitLab Mirror] project.
  3. On the Omnibus GitLab Mirror pipeline, wait for the Trigger:gitlab-docker build to finish.
  4. In the meantime,
    • Ensure you’re logged in to registry.gitlab.com. You can login with your pre-configured Docker credentials, or with a Personal Access Token or a Deploy Token using the command docker login registry.gitlab.com (or nerdctl login registry.gitlab.com -u <username> depending on what you’re using).
    • Complete the Set up volumes location on the Omnibus
  5. Once Trigger:gitlab-docker has been completed, scroll down to the end of the log and find the docker image that was pushed to registry.gitlab.com.
  6. To start the docker image on your local environment, follow the documentation and replace the gitlab/gitlab-ee:latest image with the one from the previous step.
  7. Wait for the installation to be completed, and after that, you’ll be able to access the local instance by going to 0.0.0.0:80 in your browser.

Requesting CVEs

  1. Visit the gitlab-org/cves repository issue tracker. You will be requesting CVEs by making issues using the templates in this repository.

  2. For each vulnerability (CVSSv3 score > 0) affecting the self-managed version of GitLab, request a CVE identifier by creating a new issue and selecting the Internal GitLab Submission template. Please note that some merge requests included in the security release do not require a CVE identifier (e.g., only third-party dependency upgrades, gitlab.com-only issues, etc.).

    • The CVSSv3 score should have been approved by at least two team members in the Bug Bounty Council issue. For reports that predate this process, do not hesitate to add a note to the current Council issue only to discuss the CVSSv3 score with other team members.

  3. If a CVE identifier is almost certainly going to be needed, please check the ‘Automatically request a CVE ID’ box. This will make sure that a CVE identifier is requested at the beginning of the process.

  4. For the Title of each issue, please use the respective vulnerability title that is going to be included in the blog post entry.

  5. The reporter section refers to us as requesters of the CVE and not the vulnerability reporter, the default values for those fields should be used.

  6. Fill in the remaining fields with the relevant information using the template or other previous submissions as examples. Please note that you will likely need to look up the appropriate cwe value (see CWE) and also use the CVSSv3 calculator to determine the impact.

  7. Note: some information, such as fixed_versions and solution, may not be readily available when the CVE request issue is created. Be sure to update the CVE issue once you have that information.

  8. Submit the issue. It will be reviewed by the vulnerability research team. They will follow up with any questions and then submit it to be assigned a CVE identifier.

  9. For each CVE request submission, link the relevant security issue to it. Once a CVE identifier is assigned, please also comment with that on the relevant issue and ensure that it gets included in the blog post.

CVE credit

Most vulnerabilities are credited either to a HackerOne researcher or a team member. These reports are always credited using the existing CVE template.

When a vulnerability is reported directly as a GitLab issue or by a customer via ZenDesk, provide an opportunity to be publicly credited:

Hello `[contact]`,

We are preparing to patch the issue you reported. Would you like to be publicly credited with discovering the issue, and if so, how?

Some suggestions are `@social_media_handle`, `FirstName LastName`, `FirstName LastName from CustomerName`, or `the team at CustomerName`. We can also format the credit with a link.

Our default credit will be `This vulnerability was reported by a customer` for customers and `This vulnerability was disclosed using our [Coordinated Disclosure Process](https://about.gitlab.com/security/disclosure/).` for other members of the wider GitLab community.

Updating CVEs

To update a CVE after it has been published, open a merge request in https://gitlab.com/gitlab-org/cves which modifies published/CVE-YYYY-IDIDID.json. Ping gitlab-org/secure/vulnerability-research to review and merge. The Vulnerability Research team will then handle updating the CVE with MITRE and/or NVD.

Last modified July 9, 2024: Fix links and spelling (e30f31b6)
View page source - Edit this page - please contribute. Creative Commons License