General process for the application security team in patch releases

The main difference is in who initiates the release process.

• Planned patch releases are scheduled on the Wednesdays of the weeks before and after the monthly release and are initiated by the release management team.
• Critical patch releases are initiated on-demand by the security team to resolve the most severe vulnerabilities that are labelled as S1. They are highly urgent and need to be released as soon as the patch is ready.

Cross-regional AppSec Release Managers

In order to make sure there is always someone available and ready to help get the release finished, the AppSec team assigns two team members to each release.

The ‘Primary’ team member is responsible for the steps described below in the overall process section. They are the main DRI for the patch release and the core responsibilities associated with it.

A ‘Secondary’ team member located in a different region is also assigned to each release. As with other rotations, the Secondary should find someone to switch with if they are going to be out of office or otherwise unavailable during their assigned rotation.

The Secondary is expected to help with any tasks needed to get the release finalized such as:

• Reviewing blog post in the security project.
• Working with the Communications team to get the e-mail sent out to customers
• Assisting the Delivery team as needed in order to get the release finalized

The Primary and Secondary should post handover messages in the #sec-appsec Slack channel at the end of their workday when appropriate leading up to release, and daily during release week. These messages should provide information such as:

• The current status of the release
• Links to any relevant Slack threads or merge requests
• What still needs to be done
• Any other information that might be helpful to the other team member or the rest of the team

Here is an example template:

Patch release appsec handover to <username>
Status: (brief status statement)
Next: (brief description of what should happen next)
Links
- [Patch release issue](https://gitlab.com/gitlab-org/gitlab/-/issues?sort=created_date&state=opened&label_name[]=upcoming+security+release) 
- Appsec patch release checklist:
- Release draft blog post MR:
- Release blog post MR: 
- Release communications issue:

Overall process

The following guidance can be used while also refering to the documentation included in the security release tools. Use the tooling to create the checklist task issue, which is the single source of truth for patch releases.

You can refer to the following mermaid diagram in accordance with the steps attached below:

flowchart TD
    A[MR for vulnerability fix on security mirror]
    B["Review AppSec Task Issue"]

    subgraph stage1[" "]
    CVE["<b>CVE TASKS</b> \n &bull; Versions affected has been updated in the CVE issue and also the appsec checklist issue \n &bull; CVE description is updated \n &bull; CVSS and fixed version numbers are filled out"] -.-> PERFIX
    PERFIX["<b>PER FIX TASKS</b> \n &bull; MR is (still) included in the Release Task issue \n &bull; MR has been approved / merged \n &bull; CVE issue is complete \n"]
    end

    A -->|Vulnerability added to Release Task issue| stage1
    stage1 --> B

    C[Review Blog Post on security mirror]
    subgraph stage2[" "]
    VERIFY["<b>VERIFY</b> \n &bull; Verify that MRs in Release Task issue match MRs included in Blog Post \n &bull; Verify all the fixes are complete and merged to the security mirror \n &bull; Verify that canonical URI for Blog Post is correct \n &bull; Verify wording of each fix description"] 
    VERIFY -.-> GENERAL["<b>GENERAL TASKS</b> \n &bull; Date in filename is correct \n &bull; Version Numbers for release (16.x.y, etc) updated \n &bull; Dependency Upgrades & Non-Security Fixes incorporated"]
    end

    B -->  stage2
    stage2 --> C

    subgraph stage4[" "]
    VERIFY2["<b>VERIFY</b> \n &bull; All links work \n &bull; Versions are correct \n &bull; *Critical* Designation included / excluded"] -.-> VERIFYCOMMS
    VERIFYCOMMS["<b>COMMS TASKS</b> \n Ping the team member assigned in comms issue to trigger the test email"]
    end

    E[Test Marketing Email]
    C --> stage4
    stage4 --> E
    E --> F[Marketing Email]
    F --> FINAL[Run <b>make_blogpost.rb</b> with <b>--terminate</b>]

For critical patch releases, follow the steps described in this link.

1. Create the AppSec release checklist task issue
2. Create blog post on security mirror
3. Create blog post on canonical
4. Prepare marketing email
5. Post release. Once completed, you have finished the patch release.

For all steps in the process, keep in mind that there might be a GitLab Runner security release going on at the same time and that it should be included in the process (blog post, CVE ID requests, notifications on H1 reports, etc.)

Remark: Workhorse, Gitaly, Pages and Omnibus are part of the general patch release process. Other products (e.g. gitlab-org/gitlab-runner) follow their own independent patch release process.

Create release checklist

Use the patch release tools to create the checklist task issue. Once this is done, verify that each of the individual CVE task items and the per-fix task items are completed.

Create blog post on security mirror

One day before the patch release due date, the blog post on the security mirror will be automatically created by the Delivery release tooling automation and assigned to AppSec release managers. On the patch release due date, after AppSec release managers have confirmed the blog post is completed, it will be automatically migrated and merged into the canonical handbook mirror

If required, the blog post can be manually prepared:

When all security Issues have an associated CVE request, it is possible to automatically generate the blog post with the make_blogpost.rb script located in the security-release-tools project:

ruby make_blogpost.rb -e <email address> -f "<firstname> <lastname>" -n <gitlab username>

The above command will print out the blog post contents in the terminal for review. First, we need to create a draft Merge Request. To do this, simply add the --draft flag to the command. Once the draft Blog Post is created, perform the required tasks and verifications outlined in the above mermaid diagram. Legal (@lasayers and @emccrann) and corporate communications (@gitlab-com/marketing/corporate_marketing/security-communications) stakeholders should be pinged on the draft MR.

General tasks

Make sure to verify that:

• Date in filename is correct
• Version Numbers for release (16.x.y, etc) are updated
• Dependency Upgrades & Non-Security Fixes are incorporated

Fixes that mitigate 3rd party vulnerabilities are typically added at the end of the blog post with a reference to the existing CVE ID that was assigned to that vulnerability by the 3rd party.

Example of previously used wording:

### Update Nokogiri dependency
The Nokogiri dependency has been upgraded to 1.10.8. This upgrade include a security fix for [CVE-2020-7595](https://github.com/advisories/GHSA-7553-jr98-vx47).

Verify

• Verify that MRs in Release Task issue match MRs included in Blog Post. You can run the check_merge_request.rb script to check the status of all the MRs for the fixes.
• Verify that canonical URI for Blog Post is correct
• Verify wording of each fix description

Create blog post on canonical

• The delivery team will move the reviewed blog post MR to canonical and will assign it to Primary and Secondary Appsec release managers. After the MR is created:

• The delivery automation will approve and merge the blog post MR.
• The delivery automation will close the draft Blog Post on security mirror.
• Ping #mktgops to intimate them about the blog post release

Prepare marketing email

Ensure that Legal and Corporate communications stakeholders are pinged in the communications issue by using the appropriate template when creating it. Ping the team member assigned in the communications issue to trigger the test email. Following this, ensure that:

• All links work
• Versions are correct
• Critical Designation included / excluded

In case the script does not work, follow the below steps to manually create the blog post:

Today we are releasing versions X.X.X, X.X.X, and X.X.X for GitLab Community Edition (CE) and Enterprise Edition (EE).

These versions contain a number of important security fixes, and we strongly recommend that all GitLab installations be upgraded to one of these versions immediately.

Please forward this alert to appropriate people at your organization and have them subscribe to [Security Notices](https://about.gitlab.com/contact/). You can also receive patch release blog updates by subscribing to our [RSS feed](https://about.gitlab.com/security-releases.xml).

For more details about this release, please visit our [blog](TODO)."

Post release

Run make_blogpost.rb with the --terminate flag

30 days after release

• Remove confidential from issues unless they have been labelled ~keep confidential.
  • Our bot will automatically remind us to remove confidential from the issues.

Critical patch releases

The issue must be prepared as soon as we are aware of the critical issue. Please ask @release-managers if the timing is appropriate for a release, if not all efforts should be put on finding a mitigation that doesn’t involve a new release until the agreed date. Also, notify GitLab Dedicated team as soon a critical issue is confirmed so that it will help to plan the dedicated patch process and comms.

Once a fix has been provided for a critical S1 labelled security issue:

1. Make sure that each fix included in the release has had a pre-assigned CVE requested. This should have been done by the person verifying the fix by following the steps in the Verifying Security Fixes runbook.

2. Create a security tracking issue with ~security and ~“upcoming security release” labels (e.g https://gitlab.com/gitlab-org/gitlab/-/issues/297419).

3. Associate the security issue to the security tracking issue.

4. Notify release managers of a need to prepare for the critical patch release:

   • Ping them in the security tracking issue, and
   • Ping them in the #f_upcoming_release channel.

5. If the fix is not ready yet, provide release managers with an estimate of when the fix is going to be ready so that they can create a release plan.

6. 30 days after the release is published, follow steps in Post Release.

View page source - Edit this page - please contribute.