General process for the application security team in patch releases

• release-management

GitLab Security Patch Release Process

This document outlines the process and responsibilities for managing security patch releases at GitLab for an Application Security Engineer.

You can find the current month’s release managers (from AppSec) by utilizing our dynamically updated whos-on wiki.

Introduction and Process Overview

The security patch release process ensures that security vulnerabilities in GitLab products are properly addressed, documented, and communicated to users based on vulnerability SLA’s. This systematic approach helps protect GitLab users by quickly resolving security issues while maintaining appropriate transparency about the fixes.

For information about which versions receive security fixes, see the GitLab Maintenance Policy.

Security patch releases come in two main types:

• Planned patch releases: Scheduled twice a month on Wednesdays of the weeks before and after the monthly release.
• Unplanned patch releases: Immediate patches required outside the planned cadence to mitigate high-severity (critical) vulnerabilities. These ad-hoc patches typically result from security incidents and may only include the fix for the critical vulnerability.

This handbook page describes the responsibilities and processes for Application Security engineers involved in these releases.

Expected Outputs

1. Patched GitLab Release with all security fixes included
2. Published Blog Post detailing vulnerabilities and fixes
3. Communication email sent to customers with blog post link
4. Published CVEs with accurate version and description information
5. HackerOne reports marked as resolved and rewarded

High-Level Overview

Thid diagram shows a high level overview of the process for an Application Security engineer working on a security release:

flowchart TD
    A[AppSec task issue created by release tools bot] --> B[Engineer ensures all CVE are formatted correctly]
    B -.->|Delivery team: Generates draft blog post| C[Engineer reviews blog post]
    C -.->|Delivery team: Publishes blog post| D[Blog post is published]
    D --> E[Engineer closes all security issues,\nnotifies team members on associated HackerOne reports and publishes CVE's]
    D --> F[Engineer works with communications team to validate\nand approve the content of the email sent to customers]

AppSec Release Management Procedures

Planned Patch Releases Process

The release process spans several days. The task are captured withing the AppSec release checklist, which is automatically created by the delivery automation. The template for it serves as the single source of truth for the patch release.

Pre-Release (starts 2 days before release):

• CVE verification and updates:

  The engineer on rotation is responsible of ensuring that the information in all CVEs associated with a release have the information properly filled in each of them. Each engineer that opened a CVE issue has it assigned to them and are responsible for providing the required information. Note that to reduce the burden on the engineer on rotation, the cve_description_update.rb script takes care of updating much of the information on CVE’s. Moreover, some information is also automatically filled out when a CVE is created through importing a report from HackerOne.

  Given the following example CVE YAML (template source):

  vulnerability:
  # If the affected product is GitLab, please mention if it's GitLab CE/EE or GitLab EE
  description: TODO   # "[VULNTYPE] in [COMPONENT] in [VENDOR][PRODUCT] affecting all versions from 1X.X prior to 16.X.X, 16.Y prior to 16.Y.Y, and 16.Z prior to 16.Z.Z allows [ATTACKER] to [IMPACT] via [VECTOR]"
  cwe: CWE-400 # TODO, for example "CWE-22" # Path Traversal
  product:
    gitlab_path: gitlab-org/gitlab   # "namespace/project" # the path of the project within gitlab
    vendor: GitLab   # "iTerm2"
    name: GitLab   # "iTerm2"
    # GitLab advisories MUST reference at least one CPE. Uncomment the best fitting CPE below and feel free to modify
    # the wildcard values to be more specific if applicable.
    # See https://en.wikipedia.org/wiki/Common_Platform_Enumeration#Scheme_format for more information on the format.
    cpes:
      - cpe:2.3:a:gitlab:gitlab:*:*:*:*:*:*:*:*
      # - "cpe:2.3:a:gitlab:gitaly:*:*:*:*:*:*:*:*"
      # - "cpe:2.3:a:gitlab:runner:*:*:*:*:*:*:*:*"
      # - "cpe:2.3:a:gitlab:gitlab-shell:*:*:*:*:*:*:*:*"
      # - "cpe:2.3:a:gitlab:dast_api_scanner:*:*:*:*:*:*:*:*"
      # - "cpe:2.3:a:gitlab:dynamic_application_security_testing_analyzer:*:*:*:*:*:*:*:*"
      # - "cpe:2.3:a:gitlab:gitlab-vscode-extension:*:*:*:*:*:*:*:*"
    affected_versions:
      - '>=1X.X, <16.X.X' # "1.2.3"
      - '>=16.Y, <16.Y.Y' # "1.2.3"
      - '>=16.Z, <16.Z.Z' # "1.2.3"
    fixed_versions: [] # If empty, fixed versions will be automatically filled based on affected version constraints
    # fixed_versions:
    #  - "16.X.X"
    #  - "16.Y.Y"
    #  - "16.Z.Z"
  impact: AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:H   # CVSS:3.1 string, without the prefix
  solution: '' # If empty, solution will be automatically filled based on affected version constraints
  # solution: "Upgrade to version 16.X.X, 16.Y.Y or 16.Z.Z"
  credit: Thanks [example](https://hackerone.com/) for reporting this vulnerability
    through our HackerOne bug bounty program
  # credit: "This vulnerability has been discovered internally by GitLab team member <TEAM MEMBER NAME>"
  references:
     - https://gitlab.com/
  

  The engineer should ensure that the following fields are properly filled out:

  • description: The description should reflect the vulnerability associated with it, and have the proper versions in it.
  • cwe:: The cwe field should contain a value that fits the vulnerability. It needs to be a non-discouraged one by MITRE. Duo is typically pretty helpful at suggesting good values for this.
  • cpes:: The cpe field should properly reflect the affected product
  • affected_versions: The first line should contain the first affected version. This is determined by engineering and put in the Details table of a security release as Version affected (example)
  • fixed_versions:: This should match release version numbers
  • impact:: This needs to be the CVSS determined in the associated bug bounty council discussion
  • credit:: It should contain the person who found the CVE, typically a HackerOne reporter or a GitLab team member
  • references:: It should have the canonical issue as a reference, as well as the HackerOne report in such cases

• Draft blog post review:

  The engineer on rotation is responsible for the content of the blog post. While much of it is generated from the CVE contents when all of the automation works, its important to double verify that all the information within it is accurate.

  The Security blog post is automatically generated one day before the patch release due date.

  Things to look for when reviewing the blog post:

  • It contains all of the issues resolved in a release
  • It does not include any issues that are not in the release. This can happen when some MR’s are reverted last minute.
  • The CVE title and description do not contain more information than necessary. They should be vague, the details of the vulnerabilities are release at least 30 days after the blog post.
  • All versions mentioned in the blog post match the release version
  • No ‘TODO’ is left behind in the blog post.
  • The ‘Table of security fixes’ and the CVE’s are properly ordered by severity
  • Ensure that we follow the Documentation Style Guide as much as possible. For example, if an acronym is used, we spell it out on first use on a page.
  • The author: and author_gitlab: should be set to the engineer on rotation

Release Day:

• Final verification of included fixes:

  Ensure that nothing has been removed last minute. If that were the case, the blog post should not contain any entry removed last minute.

• Blog post finalization:

  Final review of the blog post before it is being sent out. Ensure you have had at least one other team member review the blog post. The delivery release manager will ask whether AppSec is ready on release day before releasing the blog post.

• Communication to stakeholders:

  Let the Marketing team know that the blog post is about to be released by using the linked security communication issue (a link should be available in the task list.). In case of critical security patches, ensure that communications is aware of it by posting a message in #comms-team-security and has given the proper greenlight on the various communications.

Post-Release (After publication):

This part is automated through the close_release.rb script:

Usage: close_release [options]
    -i, --release-issue IID          IID of the security release tracking issue. Usually not needed, but sometimes there are two opened at once and the correct one cannot be auto-detected
        --do-all-closing-tasks       Completes all tasks, equivalent to using all the individual switches
        --close-security-issues      Closes GitLab issues that were fixed in the security release
        --publish-cves               Set the CVE requests to be published
        --notify-hackerone           Post in related HackerOne reports that the issue has been resolved
        --live                       [OPTIONAL] Pass this option to actually close issues, comment on H1, publish CVEs, etx. (Dry run is the default).

• Issue closure:

  Close all issues on the canonical repository, with a reminder to pay a bounty in cases where it fits.

• CVE publishing:

  • Review each CVE to ensure the advisory::invalid label is absent first. If it is present, look for comments from the cnaworkflow bot that describes why the submission is considered invalid.
  • Set all CVE’s to Publish Immediately by ticking the appropriate box in the issue description for the relevant ones.

• HackerOne notifications:

  Notify the engineer associated with the HackerOne reports associated with the release that the fix is out, and on which version.

• Public acknowledgments(optional):

  Use the make_thankyou.rb script to create a thank you message and post it in the #thanks slack channel in appreciation of the efforts team members put in creating the release.

Summary

For each security patch release:

1. Verify CVE information - Ensure all descriptions, CWE values, affected/fixed versions, and CVSS scores are accurate
2. Review draft blog post - Confirm all issues are properly represented with appropriate version information
3. Verify included fixes - Ensure nothing was removed last-minute before release
4. Finalize blog post - Complete final review with at least one team member
5. Communicate with stakeholders - Notify Marketing team and coordinate with Communications for critical patches
6. Close associated issues - Close all issues in canonical repository with bounty reminders where applicable
7. Publish CVEs - Review for validity and set to “Publish Immediately”
8. Send HackerOne notifications - Inform assigned engineers about fix availability and version information
9. Acknowledge contributors - Create thank you messages using the make_thankyou.rb script (optional)

Security Fix Development and Verification

Security fixes follow a thorough development and verification process that starts well before the release date. This process can be found here: https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/engineer.md

AppSec Team Organization and Responsibilities

Cross-regional AppSec Release Managers

To ensure continuous coverage since there is no guaranteed time at which each steps of the release can happen, the AppSec team assigns two members to each release:

Primary AppSec Release Manager:

• Main DRI for the patch release
• Responsible for executing the core release steps in the checklist

Secondary AppSec Release Manager (in a different region):

• Provides regional coverage when the Primary is offline
• Answers any questions from delivery that arises in their timezone
• Assists with the completion of release tasks
• Reviews blog post in the security project

The Primary and Secondary should post handover messages in the #sec-appsec Slack channel at the end of their workday to maintain continuous coverage.

The following template is an example for a handover message that can be used:

Patch release appsec handover to @username
Status: (brief status statement)
Next: (brief description of what should happen next)
Links
- Patch release issue:
- Appsec patch release checklist:
- Release draft blog post MR:
- Release blog post MR:
- Release communications issue:

Additional Stakeholders

Instructions to contact these groups are in the AppSec task issue under “Collaborative Tasks”.

• Release Managers: Initiates planned patch releases
• Legal Team: Reviews security disclosures
• Corporate Communications: Handles external communication (@gitlab-com/marketing/corporate_marketing/security-communications)
• Field Security Team: Notified for Critical/High severity issues (@gitlab-com/gl-security/security-assurance/governance-and-field-security/field-security-team/field-security-field-ciso-collaboration)
• GitLab Dedicated Team: Notified for Critical severity issues
• Support Team: Coordinated for Critical/High severity issues

Overall Patch Release Process

For a detailed diagram and explanation of the overall patch release process, see the Patch Releases documentation.

Special Considerations for Unplanned Critical Releases

For a detailed process about remediating GitLab environments in case of a critical vulnerability, see the GitLab Critical Vulnerability Remediation Runbook.

As an AppSec engineer handling a critical (S1) vulnerability, your key responsibilities include:

Initial Assessment and Coordination

• Work with SIRT, Release Managers, GitLab Dedicated engineers, and the Stage team to assess impact
• Request CVE immediately upon vulnerability verification
• Create security tracking issue with ~security and ~“upcoming security release” labels

Vulnerability Analysis

• Help evaluate impact across GitLab environments (multi-tenant SaaS, single-tenant SaaS, self-managed)
• Contribute to developing a comprehensive remediation plan

Security Fix Validation

• Review, validate, and approve the security merge request that remediates the vulnerability
• Ensure the security fix adequately addresses the vulnerability
  • This is not a pro-forma verification. AppSec must thoroughly validate that the vulnerability is properly addressed
  • For complex vulnerabilities with multiple dependencies or attack vectors, take sufficient time for a comprehensive review
  • While critical patches are time-sensitive, a proper and robust fix takes priority over speed. It’s acceptable to delay a fix to ensure complete verification
  • For extensive reviews, consider dividing the work among multiple team members

Post-Deployment Validation

• After production deployment, perform security validation on the production environment
• Confirm that GitLab.com has been remediated (EOC assistance might be required)

Communication

• Coordinate with release managers and SIRT on timing and disclosure
• Help prepare appropriate communications for different stakeholder groups

Definitions

• CVE: Common Vulnerabilities and Exposures - standardized identifiers for security vulnerabilities
• Security Repository: Private repository where security issues are initially tracked (e.g., gitlab-org/security/gitlab)
• Canonical Repository: Public repository where issues are moved after disclosure (e.g., gitlab-org/gitlab)
• Master MR: The merge request containing the security fix targeting the default branch (master or main)
• Backport MR: A merge request that applies a security fix to a previous stable version
• S1: Security severity level 1 (Critical) - highest severity level requiring immediate attention
• DRI: Directly Responsible Individual - the person accountable for the task
• Security Communications Issue: Tracking issue for release communications
• Security Implementation Issue: Issue in the security repository that tracks the implementation of a fix
• Security Tracking Issue: Issue that tracks all fixes included in a specific security release

Frequently Asked Questions

What if I realize there is a mistake in the blog post after its been published?

You can create an MR to modify its content by going to https://gitlab.com/gitlab-com/www-gitlab-com/-/tree/master/sites/uncategorized/source/releases/posts?ref_type=heads and finding the file with the blog post to adjust.

What if one of the scripts throws an error?

Complete the steps manually by referring to the present guide and open an issue in the release tools tracker with the error to help identify the issue.

Resources

• General release docs: https://gitlab.com/gitlab-org/release/docs
• Preparing security fixes for a patch release: https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/engineer.md
• AppSec task list: https://gitlab.com/gitlab-org/release-tools/-/blob/master/templates/patch_release/app_sec_issue.md.erb
• List of currently opened CVE’s waiting publication: https://gitlab.com/gitlab-org/cves/-/issues?sort=created_date&state=opened
• Repository of Security Release Tools: https://gitlab.com/gitlab-com/gl-security/product-security/appsec/tooling/security-release-tools

View page source - Edit this page - please contribute.