General process for the application security team in patch releases

• release-management

GitLab Security Patch Release Process

This document outlines the process and responsibilities for managing security patch releases at GitLab for an Application Security Engineer.

You can find the current month’s release managers (from PSIRT) by utilizing our dynamically updated Release-Manager-Scheduling wiki.

Introduction and Process Overview

The security patch release process ensures that security vulnerabilities in GitLab products are properly addressed, documented, and communicated to users based on vulnerability SLA’s. This systematic approach helps protect GitLab users by quickly resolving security issues while maintaining appropriate transparency about the fixes.

For information about which versions receive security fixes, see the GitLab Maintenance Policy.

Security patch releases come in two main types:

• Planned patch releases: Scheduled twice a month on Wednesdays of the weeks before and after the monthly release.
• Unplanned patch releases: Immediate patches required outside the planned cadence to mitigate high-severity (critical) vulnerabilities. These ad-hoc patches typically result from security incidents and may only include the fix for the critical vulnerability.

This handbook page describes the responsibilities and processes for Application Security engineers involved in these releases.

Expected Outputs

1. Patched GitLab Release with all security fixes included
2. Published Blog Post detailing vulnerabilities and fixes
3. Communication email sent to customers with blog post link
4. Published CVEs with accurate version and description information
5. HackerOne reports marked as resolved and rewarded

High-Level Overview

Thid diagram shows a high level overview of the process for an Application Security engineer working on a security release:

flowchart TD
    A[PSIRT task issue created by release tools bot] --> B[Engineer ensures all CVE are formatted correctly]
    B -.->|Delivery team: Generates draft blog post| C[Engineer reviews blog post]
    C -.->|Delivery team: Publishes blog post| D[Blog post is published]
    D --> E[Engineer closes all security issues,\nnotifies team members on associated HackerOne reports and publishes CVE's]
    D --> F[Engineer works with communications team to validate\nand approve the content of the email sent to customers]

PSIRT Release Management Procedures

Planned Patch Releases Process

The release process spans several days. The task are captured withing the PSIRT release checklist, which is automatically created by the delivery automation. The template for it serves as the single source of truth for the patch release.

Upcoming Patch Releases can be found here.

Pre-Release (starts 2 days before release):

• CVE verification and updates:

  The engineer on rotation is responsible of ensuring that the information in all CVEs associated with a release have the information properly filled in each of them. The engineer that imported a HackerOne report has the CVE issue assigned to them automatically and can help if more information is required. The engineer on security rotation is responsible for ensuring the required information is present in the CVEs for the given release. Note that to reduce the burden on the engineer on rotation, the the security release tools takes care of updating much of the information on CVE’s as soon as they are linked to the upcoming release.

  Given the following example CVE YAML (template source):

  vulnerability:
  # If the affected product is GitLab, please mention if it's GitLab CE/EE or GitLab EE
  description: TODO   # "[VULNTYPE] in [COMPONENT] in [VENDOR][PRODUCT] affecting all versions from 1X.X prior to 16.X.X, 16.Y prior to 16.Y.Y, and 16.Z prior to 16.Z.Z allows [ATTACKER] to [IMPACT] via [VECTOR]"
  cwe: CWE-400 # TODO, for example "CWE-22" # Path Traversal
  product:
    gitlab_path: gitlab-org/gitlab   # "namespace/project" # the path of the project within gitlab
    vendor: GitLab   # "iTerm2"
    name: GitLab   # "iTerm2"
    # GitLab advisories MUST reference at least one CPE. Uncomment the best fitting CPE below and feel free to modify
    # the wildcard values to be more specific if applicable.
    # See https://en.wikipedia.org/wiki/Common_Platform_Enumeration#Scheme_format for more information on the format.
    cpes:
      - cpe:2.3:a:gitlab:gitlab:*:*:*:*:*:*:*:*
      # - "cpe:2.3:a:gitlab:gitaly:*:*:*:*:*:*:*:*"
      # - "cpe:2.3:a:gitlab:runner:*:*:*:*:*:*:*:*"
      # - "cpe:2.3:a:gitlab:gitlab-shell:*:*:*:*:*:*:*:*"
      # - "cpe:2.3:a:gitlab:dast_api_scanner:*:*:*:*:*:*:*:*"
      # - "cpe:2.3:a:gitlab:dynamic_application_security_testing_analyzer:*:*:*:*:*:*:*:*"
      # - "cpe:2.3:a:gitlab:gitlab-vscode-extension:*:*:*:*:*:*:*:*"
    affected_versions:
      - '>=1X.X, <16.X.X' # "1.2.3"
      - '>=16.Y, <16.Y.Y' # "1.2.3"
      - '>=16.Z, <16.Z.Z' # "1.2.3"
    fixed_versions: [] # If empty, fixed versions will be automatically filled based on affected version constraints
    # fixed_versions:
    #  - "16.X.X"
    #  - "16.Y.Y"
    #  - "16.Z.Z"
  impact: AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:H   # CVSS:3.1 string, without the prefix
  solution: '' # If empty, solution will be automatically filled based on affected version constraints
  # solution: "Upgrade to version 16.X.X, 16.Y.Y or 16.Z.Z"
  credit: Thanks [example](https://hackerone.com/) for reporting this vulnerability
    through our HackerOne bug bounty program
  # credit: "This vulnerability has been discovered internally by GitLab team member <TEAM MEMBER NAME>"
  references:
     - https://gitlab.com/
  

  The engineer should ensure that the following fields are properly filled out:

  • description: The description should reflect the vulnerability associated with it, and have the proper versions in it. It should be vague, the details of the vulnerabilities are released only after a period of time, following our process for disclosing security issues.
  • cwe:: The cwe field should contain a value that fits the vulnerability. It needs to be a non-discouraged one by MITRE. Duo is typically pretty helpful at suggesting good values for this.
  • cpes:: The cpe field should properly reflect the affected product
  • affected_versions: The first line should contain the first affected version. This is determined by engineering and put in the Details table of a security release as Version affected (example)
  • fixed_versions:: This should match release version numbers
  • impact:: This needs to be the CVSS determined in the associated bug bounty council discussion
  • credit:: It should contain the person who found the CVE, typically a HackerOne reporter or a GitLab team member
  • references:: It should have the canonical issue as a reference, as well as the HackerOne report in such cases

  Also, the title of the issue needs to be updated to the following format: “${CWE_TITLE} issue in ${COMPONENT} impacts GitLab ${GITLAB_EDITION}”. This title will be used by the automation that generates the blog post.

• Draft blog post review:

  The engineer on rotation is responsible for the content of the blog post. While much of it is generated from the CVE contents when all of the automation works, its important to double verify that all the information within it is accurate.

  The Security blog post is automatically generated one day before the patch release due date.

  Things to look for when reviewing the blog post:

  • It contains all of the issues resolved in a release
  • It does not include any issues that are not in the release. This can happen when some MR’s are reverted last minute.
  • The CVE titles and descriptions do not contain more information than necessary. They should be vague, the details of the vulnerabilities are released after a period of time following our process for disclosing security issues.
  • All versions mentioned in the blog post match the release version
  • No ‘TODO’ is left behind in the blog post.
  • The ‘Table of security fixes’ and the CVE’s are properly ordered by severity
  • Ensure that we follow the Documentation Style Guide as much as possible. For example, if an acronym is used, we spell it out on first use on a page.
  • The author: and author_gitlab: should be set to the engineer on rotation

Release Day:

• Final verification of included fixes:

  Ensure that nothing has been removed last minute. If that were the case, the blog post should not contain any entry removed last minute.

• Blog post finalization:

  Final review of the blog post before it is being sent out. Ensure you have had at least one other team member review the blog post. The delivery release manager will ask whether PSIRT is ready on release day before releasing the blog post.

• Communication to stakeholders:

  Let the Marketing team know that the blog post is about to be released by using the linked security communication issue (a link should be available in the task list.). In case of critical security patches, ensure that communications is aware of it by posting a message in #comms-team-security and has given the proper greenlight on the various communications.

Post-Release (After publication):

This part is automated through the close_release.rb script:

• Issue closure:

  Close all issues on the canonical repository, with a reminder to pay a bounty in cases where it fits.

• CVE publishing:

  • Review each CVE to ensure the advisory::invalid label is absent first. If it is present, look for comments from the cnaworkflow bot that describes why the submission is considered invalid.
  • Set all CVE’s to Publish Immediately by ticking the appropriate box in the issue description for the relevant ones.

• HackerOne notifications:

  Notify the engineer associated with the HackerOne reports associated with the release that the fix is out, and on which version.

• Public acknowledgments(optional):

  Use the make_thankyou.rb script to create a thank you message and post it in the #thanks slack channel in appreciation of the efforts team members put in creating the release.

Summary

For each security patch release:

1. Verify CVE information - Ensure all descriptions, CWE values, affected/fixed versions, and CVSS scores are accurate
2. Review draft blog post - Confirm all issues are properly represented with appropriate version information
3. Verify included fixes - Ensure nothing was removed last-minute before release
4. Finalize blog post - Complete final review with at least one team member
5. Communicate with stakeholders - Notify Marketing team and coordinate with Communications for critical patches
6. Close associated issues - Close all issues in canonical repository with bounty reminders where applicable
7. Publish CVEs - Review for validity and set to “Publish Immediately”
8. Send HackerOne notifications - Inform assigned engineers about fix availability and version information
9. Acknowledge contributors - Create thank you messages using the make_thankyou.rb script (optional)

Security Fix Development and Verification

Security fixes follow a thorough development and verification process that starts well before the release date. This process can be found here: https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/engineer.md

PSIRT Team Organization and Responsibilities

Cross-regional PSIRT Release Managers

To ensure continuous coverage since there is no guaranteed time at which each steps of the release can happen, the PSIRT team assigns two members to each release:

Primary PSIRT Release Manager:

• Main DRI for the patch release
• Responsible for executing the core release steps in the checklist

Secondary PSIRT Release Manager (in a different region):

• Provides regional coverage when the Primary is offline
• Answers any questions from delivery that arises in their timezone
• Assists with the completion of release tasks
• Reviews blog post in the security project

The Primary and Secondary should post handover messages in the #sec-psirt-team Slack channel at the end of their workday to maintain continuous coverage.

The following template is an example for a handover message that can be used:

Patch release handover to @username
Status: (brief status statement)
Next: (brief description of what should happen next)
Links
- Patch release issue:
- PSIRT patch release checklist:
- Release draft blog post MR:
- Release blog post MR:
- Release communications issue:

Additional Stakeholders

Instructions to contact these groups are in the PSIRT task issue under “Collaborative Tasks”.

• Release Managers: Initiates planned patch releases
• Legal Team: Reviews security disclosures
• Corporate Communications: Handles external communication (@gitlab-com/marketing/corporate_marketing/security-communications)
• Field Security Team: Notified for Critical/High severity issues (@gitlab-com/gl-security/security-assurance/governance-and-field-security/field-security-team/field-security-field-ciso-collaboration)
• GitLab Dedicated Team: Notified for Critical severity issues
• Support Team: Coordinated for Critical/High severity issues

Overall Patch Release Process

For a detailed diagram and explanation of the overall patch release process, see the Patch Releases documentation.

Special Considerations for Unplanned Critical Releases

For a detailed process about remediating GitLab environments in case of a critical vulnerability, see the GitLab Critical Vulnerability Remediation Runbook.

As an PSIRT engineer handling a critical (S1) vulnerability, your key responsibilities include:

Initial Assessment and Coordination

• Work with SIRT, Release Managers, GitLab Dedicated engineers, and the Stage team to assess impact
• Request CVE immediately upon vulnerability verification
• Create security tracking issue with ~security and ~“upcoming security release” labels

Vulnerability Analysis

• Help evaluate impact across GitLab environments (multi-tenant SaaS, single-tenant SaaS, self-managed)
• Contribute to developing a comprehensive remediation plan

Security Fix Validation

• Review, validate, and approve the security merge request that remediates the vulnerability
• Ensure the security fix adequately addresses the vulnerability
  • This is not a pro-forma verification. PSIRT must thoroughly validate that the vulnerability is properly addressed
  • For complex vulnerabilities with multiple dependencies or attack vectors, take sufficient time for a comprehensive review
  • While critical patches are time-sensitive, a proper and robust fix takes priority over speed. It’s acceptable to delay a fix to ensure complete verification
  • For extensive reviews, consider dividing the work among multiple team members

Post-Deployment Validation

• After production deployment, perform security validation on the production environment
• Confirm that GitLab.com has been remediated (EOC assistance might be required)

Communication

• Coordinate with release managers and SIRT on timing and disclosure
• Help prepare appropriate communications for different stakeholder groups

Definitions

• CVE: Common Vulnerabilities and Exposures - standardized identifiers for security vulnerabilities
• Security Repository: Private repository where security issues are initially tracked (e.g., gitlab-org/security/gitlab)
• Canonical Repository: Public repository where issues are moved after disclosure (e.g., gitlab-org/gitlab)
• Master MR: The merge request containing the security fix targeting the default branch (master or main)
• Backport MR: A merge request that applies a security fix to a previous stable version
• S1: Security severity level 1 (Critical) - highest severity level requiring immediate attention
• DRI: Directly Responsible Individual - the person accountable for the task
• Security Communications Issue: Tracking issue for release communications
• Security Implementation Issue: Issue in the security repository that tracks the implementation of a fix
• Security Tracking Issue: Issue that tracks all fixes included in a specific security release

Frequently Asked Questions

What if I realize there is a mistake in the blog post after its been published?

You can create an MR to modify its content by going to https://gitlab.com/gitlab-com/www-gitlab-com/-/tree/master/sites/uncategorized/source/releases/posts?ref_type=heads and finding the file with the blog post to adjust.

What if one of the scripts throws an error?

Complete the steps manually by referring to the present guide and open an issue in the release tools tracker with the error to help identify the issue.

Resources

• General release docs: https://gitlab.com/gitlab-org/release/docs
• Preparing security fixes for a patch release: https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/engineer.md
• AppSec task list: https://gitlab.com/gitlab-org/release-tools/-/blob/master/templates/patch_release/app_sec_issue.md.erb
• List of currently opened CVE’s waiting publication: https://gitlab.com/gitlab-org/cves/-/issues?sort=created_date&state=opened
• Repository of Security Release Tools: https://gitlab.com/gitlab-com/gl-security/product-security/appsec/tooling/security-release-tools

View page source - Edit this page - please contribute.