General process for the application security team in security releases

The main difference is in who initiates the release process.

  • Critical security releases are initiated on-demand by the security team to resolve the most severe vulnerabilities that are labelled as S1. They are highly urgent and need to be released as soon as the patch is ready.
  • Regular security releases are scheduled monthly and are initiated by the release management team.

Cross-regional AppSec Release Managers

In order to make sure there is always someone available and ready to help get the release finished, the AppSec team assigns two team members to each release.

The ‘Primary’ team member is responsible for the steps described below in the overall process section. They are the main DRI for the security release and the core responsibilities associated with it.

A ‘Secondary’ team member located in a different region is also assigned to each release. As with other rotations, the Secondary should find someone to switch with if they are going to be out of office or otherwise unavailable during their assigned rotation.

The Secondary is expected to help with any tasks needed to get the release finalized such as:

  • Copying the reviewed blog post over to the canonical www-gitlab-com repository
  • Working with the Communications team to get the e-mail sent out to customers
  • Assisting the Delivery team as needed in order to get the release finalized

The Primary and Secondary should post handover messages in the #sec-appsec Slack channel at the end of their workday when appropriate leading up to release, and daily during release week. These messages should provide information such as:

  • The current status of the release
  • Links to any relevant Slack threads or merge requests
  • What still needs to be done
  • Any other information that might be helpful to the other team member or the rest of the team

Here is an example template:

Security release appsec handover to <username>
Status: (brief status statement)
Next: (brief description of what should happen next)
Links
- [Security release issue](https://gitlab.com/gitlab-org/gitlab/-/issues?sort=created_date&state=opened&label_name[]=upcoming+security+release) 
- Appsec security release checklist:
- Release draft blog post MR:
- Release blog post MR: 
- Release communications issue:

Overall process

The following guidance can be used while also refering to the documentation included in the security release tools. Use the tooling to create the checklist task issue, which is the single source of truth for security releases.

You can refer to the following mermaid diagram in accordance with the steps attached below:

flowchart TD
    A{MR for vulnerability fix on security mirror} -->|Vulnerability added to Release Task issue| B{"Generate AppSec Task Issue using \n <b>create_checklist_issue.rb</b>"}
    B --> C{Create Blog Post on security mirror}
    stage1 -.-> B
    stage2 -.-> C
    C --> D{Create Blog Post on Canonical}
    stage3 -.-> D
    D --> E{Test Marketing Email}
    stage4 -.-> E
    E --> F{Marketing Email}
    F --> FINAL{Run <b>make_blogpost.rb</b> with <b>--terminate</b>}


    subgraph stage1[" "]
    direction RL
    CVE["<b>CVE TASKS</b> \n &bull; Versions affected has been updated in the CVE issue and also the appsec checklist issue \n &bull; CVE description is updated \n &bull; CVSS and fixed version numbers are filled out"] -.-> PERFIX
    PERFIX["<b>PER FIX TASKS</b> \n &bull; MR is (still) included in the Release Task issue \n &bull; MR has been approved / merged \n &bull; CVE issue is complete \n"]
    end

    subgraph stage2[" "]
    direction RL
    VERIFY["<b>VERIFY</b> \n &bull; Verify that MRs in Release Task issue match MRs included in Blog Post, and run <b>check_merge_request.rb</b> to see the status of MRs for all the fixes \n &bull; Verify that canonical URI for Blog Post is correct \n &bull; Verify wording of each fix description"] 
    VERIFY -.-> GENERAL["<b>GENERAL TASKS</b> \n &bull; Date in filename is correct \n &bull; Version Numbers for release (16.x.y, etc) updated \n &bull; Dependency Upgrades & Non-Security Fixes incorporated"]
    end

    subgraph stage3[" "]
    direction RL
    GENERAL2["<b>GENERAL TASKS</b> \n &bull; Ping the release manager to get the Blog Post merged \n &bull; Close the draft Blog Post on security mirror"]
    end

    subgraph stage4[" "]
    direction RL
    VERIFY2["<b>VERIFY</b> \n &bull; All links work \n &bull; Versions are correct \n &bull; *Critical* Designation included / excluded"] -.-> VERIFYCOMMS
    VERIFYCOMMS["<b>COMMS TASKS</b> \n Ping the team member assigned in comms issue to trigger the test email"]
    end
    

For critical security releases, follow the steps described in this link.

  1. Create the AppSec release checklist task issue
  2. Create blog post on security mirror
  3. Create blog post on canonical
  4. Prepare marketing email
  5. Post release. Once completed, you have finished the security release.

For all steps in the process, keep in mind that there might be a GitLab Runner security release going on at the same time and that it should be included in the process (blog post, CVE ID requests, notifications on H1 reports, etc.)

Remark: Workhorse, Gitaly, Pages and Omnibus are part of the general security release process. Other products (e.g. gitlab-org/gitlab-runner) follow their own independent security release process.

Create release checklist

Use the security release tools to create the checklist task issue. Once this is done, verify that each of the individual CVE task items and the per-fix task items are completed.

Create blog post on security mirror

When all security Issues have an associated CVE request, it is possible to automatically generate the blog post with the make_blogpost.rb script located in the security-release-tools project:

ruby make_blogpost.rb -e <email address> -f "<firstname> <lastname>" -n <gitlab username>

The above command will print out the blog post contents in the terminal for review. First, we need to create a draft Merge Request. To do this, simply add the --draft flag to the command. Once the draft Blog Post is created, perform the required tasks and verifications outlined in the above mermaid diagram:

General tasks

Make sure to verify that:

  • Date in filename is correct
  • Version Numbers for release (16.x.y, etc) are updated
  • Dependency Upgrades & Non-Security Fixes are incorporated

Fixes that mitigate 3rd party vulnerabilities are typically added at the end of the blog post with a reference to the existing CVE ID that was assigned to that vulnerability by the 3rd party.

Example of previously used wording:

### Update Nokogiri dependency
The Nokogiri dependency has been upgraded to 1.10.8. This upgrade include a security fix for [CVE-2020-7595](https://github.com/advisories/GHSA-7553-jr98-vx47).

Verify

  • Verify that MRs in Release Task issue match MRs included in Blog Post. You can run the check_merge_request.rb script to check the status of all the MRs for the fixes.
  • Verify that canonical URI for Blog Post is correct
  • Verify wording of each fix description

Create blog post on canonical

When the draft has been approved, it’s possible to create the actual release blog post by adding the --release flag to the command. After the MR is created:

  • Ping the release manager to get the Blog Post merged
  • Ping #mktgops to intimate them about the blog post release
  • Close the draft Blog Post on security mirror

Prepare marketing email

Ping the team member assigned in the communications issue to trigger the test email. Following this, ensure that:

  • All links work
  • Versions are correct
  • Critical Designation included / excluded

In case the script does not work, follow the below steps to manually create the blog post:

Steps 1. :warning: The developer should have provided the range of affected versions: ask on the related security issues to provide it if not already provided.
• This is required to [request CVEs](https://about.gitlab.com/handbook/security/security-engineering/application-security/runbooks/verifying-security-fixes.html#requesting-cves). See also [GitLab's CVE Numbering Authority page](https://about.gitlab.com/security/cve/).
2. Create a branch for the Draft blog post merge request that will go in the [security www-gitlab-com repo](https://gitlab.com/gitlab-org/security/www-gitlab-com):
a. On the merge request branch in the [security www-gitlab-com repo](https://gitlab.com/gitlab-org/security/www-gitlab-com), make a file in the sites/uncategorized/source/releases/posts with this format: YYYY-MM-DD-security-release-gitlab-X-X-X-released.html.md.
b. Make sure that Draft: is prepended to the blog post merge request title until it has been reviewed, approved, and the packages have been built and ready for release.
c. Ensure to add the CVE IDs, vulnerability descriptions, affected versions, remediation info, and a thank you to the reporter.
• Much of this information can be copy and pasted from the [summary table] in the developer security issue created on [GitLab Security].
d. Feel free to use a past security release as a guide, for example [this blog post](https://about.gitlab.com/releases/2020/07/01/
security-release-13-1-2-release/).
3. Please make sure the tags section of the post header includes the security tag, like so: tags: security.
4. Please add /images/blogimages/security-cover-new.png to the image_title: field in the front matter.
5. Please include a "Receive Security Release Notifications" section at the very bottom of the blog post with links to our contact us page and RSS feed. See previous security release posts or [this issue](https://gitlab.com/gitlab-com/gl-security/security-communications/communications/issues/145#note_240450797) for examples.
6. Make sure that the blog post only mentions fixes that actually made it into the release. Consider the [48h deadline before the Security Release due date](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/process.md#security-release-deadlines), every issue associated (even after the deadline) will likely not be included and we should ensure it's not mentioned on the blog post. For anything that was removed, please comment on the related security issue with the blog post text that you have written.
7. Assign the blog post to another member of the appsec team for review.
8. Create a WIP issue using the Security Release Email Alert template in https://gitlab.com/gitlab-com/gl-security/security-communications/communications/issues for the communications team and request an email notification be sent to subscribers of the Security Alert segment on the day of the release. Include a note that this should be sent out after the blog post is live. Also mention that you'll include the link to the blog post MR once it is prepared. The content of this notification should be similar to the blog post introduction:
Today we are releasing versions X.X.X, X.X.X, and X.X.X for GitLab Community Edition (CE) and Enterprise Edition (EE).

These versions contain a number of important security fixes, and we strongly recommend that all GitLab installations be upgraded to one of these versions immediately.

Please forward this alert to appropriate people at your organization and have them subscribe to [Security Notices](https://about.gitlab.com/contact/). You can also receive security release blog updates by subscribing to our [RSS feed](https://about.gitlab.com/security-releases.xml).

For more details about this release, please visit our [blog](TODO)."

Post release

Run make_blogpost.rb with the --terminate flag

30 days after release

  • Open a new security release issue for the next security release. Identify the next release managers from the release managers page and assign the issue to them.
  • Remove confidential from issues unless they have been labelled ~keep confidential.
    • Our bot will automatically remind us to remove confidential from the issues.

Critical security releases

The issue must be prepared as soon as we are aware of the critical issue. Please ask @release-managers if the timing is appropriate for a release, if not all efforts should be put on finding a mitigation that doesn’t involve a new release until the agreed date.

Once a fix has been provided for a critical S1 labelled security issue:

  1. Make sure that each fix included in the release has had a pre-assigned CVE requested. This should have been done by the person verifying the fix by following the steps in the Verifying Security Fixes runbook.

  2. Create a security release tracking issue with ~security and ~“upcoming security release” labels (e.g https://gitlab.com/gitlab-org/gitlab/-/issues/297419).

  3. Associate the security issue to the security release tracking issue.

  4. Notify release managers of a need to prepare for the critical security release:

    • Ping them in the security release tracking issue, and
    • Ping them in the #f_upcoming_release channel.
  5. If the fix is not ready yet, provide release managers with an estimate of when the fix is going to be ready so that they can create a release plan.

  6. 30 days after the release is published, follow steps in Post Release.

Last modified September 6, 2023: Replace taps with spaces (69f17a79)