Application Security Runbooks

The following process is a supplement to the first few steps of the critical release process

Once a potential severity::1/priority::1 issue is made known. The appsec engineer steps are as follows:

Triage

1. Triage and verify the issue as you normally would triage a report.
2. Finalize the CVSS score of the security issue with team member votes on Bug Bounty Council (BBC) thread before engaging the SIRT team. Consider using a sync call or Slack for the discussion due to time sensitivity. Capture the outcome of the discussion in the BBC thread. If a sync call or a Slack discussion was not possible due to AppSec team members in the region being on PTO or timezone issues, trigger the SIRT workflow if 4 hours have passed since the issue was triaged.
3. Within the BBC thread, create a GitLab Dedicated specific CVSS score.
4. To help SecOps quickly determine impact and log analysis, comment in the security issue with the summarized reproduction steps (HTTP Requests, generated log messages, images, etc).
5. After escalating, do an investigation to try to determine if there are other immediately vulnerable components or other impacts.

Escalate

1. Engage the Security Engineer on-call with a link to the issue, a summary of what has happened, and an description of what SIRT may need to do.
2. Engage the appropriate engineering manager and product manager of the affected component in both the issue and in the appropriate Slack channels.
3. If help from the GitLab Dedicated team is needed, follow the runbook to escalate to their engineer on call.
4. Ping @appsec-leadership in the #sec-appsec Slack channel with a link to the issue. This will help team leadership and other engineers get up to speed, in case they need to step in.
5. Create a link to the Bug Bounty Council CVSS discussion in the SIRT incident GitLab issue.
6. Create a bookmark to the CVSS discussion in the incident specific Slack channel.

Evaluate Impact in Different Environments

Due to differences in settings, feature availability, and configuration between GitLab self-managed, GitLab Dedicated, and GitLab SaaS (GitLab.com), the CVSS for a single vulnerability may differ depending on environment.

This runbook is meant to help AppSec engineers who need to engage and work with SIRT to respond to a HackerOne report, discovered vulnerability, or other security incident.

Requirements for using /security to engage SIRT

If this is a P1S1, follow the P1S1 runbook and engage the security on-call.

When using /security to engage SIRT:

• Only use /security after we have validated the vulnerability or otherwise confirmed the incident
• Fill out the form as completely as you can
• Include a link to the issue or HackerOne report, a summary of what has occurred, and a description of what we are asking SIRT to help with
• Select the appropriate urgency (see below for some guidelines)

Situations where SIRT needs to be engaged

Note: This is not an exhaustive list, there may be other situations in which you need to engage SIRT.

When evaluating security issues or MRs, it can be useful to have a way to reproduce issues, dig in to root causes, look for further impacts. This can also be a great way to get familiar with GitLab during your first few weeks of onboarding. Here are some handy tips & tricks.

Get a version of GitLab to play with

Many AppSec engineers will opt to install GitLab locally using GDK. Here’s how to install GDK and how to use it. The UX team have a great summary of other ways to play with & test GitLab.

This runbook describes the process for times when the Application Security team has team members available during holidays, Friends and Family days, or other events where many team members are expected to be out of office.

Since Application Security is not on-call, we aim to provide this best-effort coverage that may or may not be available in every region.

If you need assistance:

• Post in #sec-appsec with details
• Find the AppSec engineer providing coverage - see Communicating the coverage below
• Reach out to the team member via Slack @ mention
• If urgent or the team member does not respond in a timely manner, reach out via text message or phone call after checking the contact preferences listed on their Slack profile
• If it is unclear who is providing coverage, ping the @appsec-team in Slack

Expectations for those doing coverage

• In the event that AppSec is needed to respond to an S1 incident or otherwise urgently assist another team, the team member providing coverage should be reachable by Slack or phone
• AppSec team members providing coverage are not expected to be at the keyboard and working, but they should be able to get online within one hour if needed
• If they are not actively working during coverage times, they should try to check Slack occasionally for pings
• They should occasionally check for potential S1s coming in through HackerOne and take action if it requires immediate attention
• If an S1 incident occurs or AppSec assistance is urgently needed by another team, the expectation is that the AppSec team member will attempt to triage the situation and escalate as needed to a security manager

Planning

When holidays or Friends and Family days are coming up, the AppSec team should try to find a team member who can swap their day off. Ideally a team member for each region would be available, but it is not necessarily required.

How can AppSec Engineers Contribute to the Secure Code Warrior Training Program?

If anyone from the AppSec team is interested in contributing to the Secure Code Warrior (SCW) training program and you don’t have access to the Secure Code Warrior training portal, please post a comment in the #sec-appsec Slack channel requesting access.

Once you have access to the Secure Code Warrior training portal, please do the following:

• Join the #security-secure-code-warrior Slack channel
• Watch Secure Code Warrior platform walkthrough for a quick overview of the platform
• Please read through SCW’s documentation on Getting Started
• Go through our old SCW Trial issue to get an idea on how we have organized Tournaments in SCW.

Once you have done the above, you will be in a great shape to start contributing to the SCW training program. Please feel free to post in #sec-appsec if you have any questions.

Bug Hunting Day Process

The Application Security Team has a bug hunting day on the last Friday of every month.

Goal

The objective is to have some time to investigate on something a team member has observed, discussed or noticed during a review but didn’t have the time to go into a detailed review or testing.

Bug hunting day is not mandatory. There may be months that are particularly busy and we may not be doing bug hunting days on months like that.

Please refer to the GitLab CVSS Calculator as the single-source-of-truth to determine CVSS scores on GitLab security issues.

Ruby Gems

When a new gem is added to our Gemfile or when versions are changed in Gemfile.lock, we ask developers to reach out to the AppSec team to request a review. As an AppSec engineer performing the review, please follow the steps mentioned below to perform reviews like these:

• First, look to see if the gem is well maintained.
  • Look closer at the gem’s code to see if there are any anomalies. This step can be time consuming, so, make sure you balance the time taken to review the gem’s code with the urgency of the MR/issue the GitLab developer pinged us on.
  • The diffgem helper function in gitlab-com/gl-security/product-security/appsec/tooling/appsec-command-line-utils was created to assist in reviewing gem version changes by showing only the diffs for Ruby files. (excluding changes to specs, json, yaml, and other noise)
  • Depscore if enabled in the project CI/CD will run checks against the new dependency and comments the result in the MR to help with the Gem review.
• Look to see if we are using the latest version of the gem in our code.
• Read the gem’s documentation to see if there are any configurations or usages that should be avoided.
• Check for security advisories and fixes to evaluate the quality of security fixes.

After you have done the above, please add a comment to the MR/issue the developer pinged us on mentioning that:

Certain customers scan containers that GitLab provides for known vulnerabilities and other security concerns. The GitLab Federal Application Security team is responsible for reviewing these results. In many cases, these findings are related to dependencies that need to be updated to address vulnerabilities.

The expectation is that the Federal Application Security Team will verify every finding in each scanned container and make a determination as to whether the finding is a true positive, false positive, or a duplicate. For each one of these determinations, a justification needs to be communicated. These justifications are then reviewed and used to decide whether or not that particular GitLab container is approved for use.

This content has moved to the internal handbook: https://internal.gitlab.com/handbook/security/product_security/application_security/runbooks/fedramp-scanner-triage-process/.

You can find the current month’s release managers (from AppSec) by utilizing our dynamically updated whos-on wiki.

The main difference is in who initiates the release process.

• Planned patch releases are scheduled on the Wednesdays of the weeks before and after the monthly release and are initiated by the release management team.
• Critical patch releases are initiated on-demand by the security team to resolve the most severe vulnerabilities that are labelled as S1. They are highly urgent and need to be released as soon as the patch is ready.

Cross-regional AppSec Release Managers

In order to make sure there is always someone available and ready to help get the release finished, the AppSec team assigns two team members to each release.

GitLab utilizes HackerOne for its bug bounty program. Security researchers can report vulnerabilities in GitLab applications or the GitLab infrastructure via the HackerOne website. Team members authorized to respond to HackerOne reports use procedures outlined here. The #hackerone-feed Slack channel receives notifications of report status changes and comments via HackerOne’s Slack integration.

Queues

• New contains all reports in the New state
• GitLab Team contains reports that have been validated by the HackerOne triage team, but are yet to be assigned to a specific GitLab team member
• H1 Triage are reports being triaged by the HackerOne triage team
• Pending Disclosure are reports that should be reviewed and disclosed

GitLab Team On-boarding

• New members of the GitLab security team are granted access to the GitLab HackerOne team via an access request issue using the appropriate role based entitlement template, which should be submitted by their manager during onboarding
• During onboarding, new GitLab security team members will be invited to join the HackerOne program if their role requires it.

Working the Queue

Namespace with Ultimate license for triaging

Please ensure that you use this namespace to create projects and groups required for testing vulnerabilities. This namespace is dedicated to reproduction of HackerOne issues.

Third parties

Sometimes the root cause of a vulnerability is in a third-party dependency and even the latest version of that dependency is vulnerable. This page describes how we process in those situations.

The first step is to disclose the vulnerability to the third party. See in the project’s README.md, SECURITY.md or on their website if they have official means of disclosing vulnerabilities. If they don’t, simply email the maintainer(s) of the project. This email should cc the Engineering Manager, the person from Application Security who was involved with the triage of the security issue, and any other relevant team members. For transparency, we should copy this communication in the security issue.

List of Package Hunter Findings

Any Package Hunter related finding can be found on this dashboard (internal link).

Network Connection Findings

We’re going to use this finding as an example. In this finding, Package Hunter detected that a package opened a network connection.

02:24:56.163600570: Notice Disallowed outbound connection destination (command=node scripts/install.js connection=172.17.0.2:36884->52.217.109.44:443 user=root container_id=fb38d63ef02a container_name=some-container-eae8d3ec-a482-441b-8262-78198b46fdfb.tgz image=maldep)

Investigation

• We can look at the destination IP address and see if gives any interesting information
  • In this example, it’s a an AWS IP address but it gives us no information about what package might have contacted this IP
• Attention: This step requries to checkout the branch for which the finding was made. Please proceed with caution when analyzing potentially malicious code on your local computer. We recommend to checkout the code into a dedicated VM. If there are any questions or you require assistance, please reach out to @gitlab-com/gl-security/product-security/appsec.
  • We know that the package ran the command scripts/install.js so we can search for that file (be sure to execute yarn install before running the search)
  • Running find . -name install.js leads us to node_modules/node-sass/scripts/install.js
• We can inspect the package.json files to find the command that triggered the detection
  • We see in node_modules/node-sass/package.json that scripts/install.js is indeed called

Completing the investigation in the example, we see that the install.js script calls getBinaryURL(). This function is defined in node_modules/node-sass/lib/extensions.js:

The Merge Monitor tool looks in public GitLab repositories that JiHu contributes to for merge requests that:

• Were merged
• Were labeled as a JiHu Contribution
• Were not labeled with the label that AppSec team members need to apply after conducting security reviews of JiHu contributions

Any findings will be included in reports that are created as issues in the jihu_merge_request_monitor_reports repository. The Federal AppSec team is pinged on each report that is created and the expectation is that they will review each finding.

Frequency: Daily

AppSec engineers are responsible for triaging the findings of the GitLab security tools. This role has two primary functions.

1. Validate the findings and handoff to engineering for correction
2. Provide feedback to the Secure team

Security Dashboard List

The following is a list of security dashboards that need to be reviewed:

• GitLab
• GitLab Omnibus
• Gitaly
• GitLab Pages
• GitLab Shell
• k8s-workloads
• Version
• UBI images
• release-cli
• vscode-extension
• Customers
• elastic-search-indexer
• GitLab DAST
• GitLab Terminal
• GitLab Agent
• Compensation Calculator

Validation

For each finding:

Application Security team members are alphabetically assigned as the responsible individual (DRI) for incoming requests to the Application Security team, typically for a weekly or fortnighly period.

Who is on rotation?

Automation manages the scheduling and assignment of rotations:

• “Who is on rotation now?”
• Rotation Management FAQ & README
• Holiday Coverage

What are the rotations?

The following rotations are defined:

• (Weekly Assignment) HackerOne + Security Dashboard Review
  • Point of contact for “New” HackerOne reports during that week.
  • Responsible to escalating to other team members and management if the size of the either queue spikes.
  • Responsible for reviewing security dashboards on a best-effort level
• (Weekly Assignment) Triage Rotation (mentions and issues), by order of priority:
  • Daily triage of FedRAMP vulnerabilities reported by our scanners
    • This is a mandatory daily check to make sure we do triage of vulnerabilities.
    • If it is not possible to do a daily check and triage of reported vulnerabilities, ask on for help in the #sec-appsec channel.
  • First responder to JiHu Contribution pings that come into the #sec-appsec Slack channel
  • First responder to automated messages posted in the #public_merge_requests_referencing_confidential_issues Slack channel
    • Add a check mark emoji if the merge request can be public
    • If the merge request references a legitimate security issue
      • If the issue has a ~security-fix-in-public label, indicating it has been approved by an AppSec team member to be fixed in public, link to the comment granting approval or include a message in Slack denoting that the ~security-fix-in-public label was added.
      • Decide if it can be public anyway, and apply the ~security-fix-in-public label retrospectively
      • Otherwise contact SIRT and the merge request author to get the merge request removed.
      • Use the Urgent - SEOC should be paged right away option if waiting up to 24 hours for a resolution would be too long.
  • First responder to mentions of the following group aliases:
    • @gitlab-com/gl-security/product-security/appsec on GitLab.com
    • @appsec-team in Slack
  • First responder for issues created needing triage: ~security-triage-appsec issue search
    • Refer to [this page]({{ ref “engaging-with-security#reproducibility-on-security-issues” }}) to learn about the different labels that we can apply to issues when they’re not vulnerabilities
• (~Fortnightly Assignment) Security Engineer for Security & Patch Releases
• (Fortnightly Assignment, Federal AppSec only) Release Certifications
  • Responsible for the release certification process
  • This applies to any release that might have JiHu contributions, including monthly and patch releases
• (Quarterly Assignment) Bug Bounty/AppSec Blog Post

If the Application Security team member has a conflict for the assigned week they may swap rotation weeks with another team member. This may be done for any reason including time off or the need for time to focus on a particular task.

The review of a fix by an application security engineer is triggered by the engineer implementing the fix. As the engineer engaging in the verification, these are the steps that should be followed:

1. Doing a code review from the security perspective.

2. Validating the security fix using the Docker image generated by package-and-qa, see the section below for more details.

3. Once the validation is done, update the security issue with a link to the CVE.