The following process is a supplement to the first few steps of the critical release process
Once a potential severity::1/priority::1 issue is made known. The appsec engineer steps are as follows:
Triage Triage and verify the issue as you normally would triage a report. To help SecOps quickly determine impact and log analysis, comment in the security issue with the summarized reproduction steps (HTTP Requests, generated log messages, images, etc). After escalating, do an investigation to try to determine if there are other immediately vulnerable components or other impacts.
This runbook is meant to help AppSec engineers who need to engage and work with SIRT to respond to a HackerOne report, discovered vulnerability, or other security incident.
Requirements for using /security to engage SIRT If this is a P1S1, follow the P1S1 runbook and engage the security on-call.
When using /security to engage SIRT:
Only use /security after we have validated the vulnerability or otherwise confirmed the incident Fill out the form as completely as you can Include a link to the issue or HackerOne report, a summary of what has occurred, and a description of what we are asking SIRT to help with Select the appropriate urgency (see below for some guidelines) Urgency guidelines These are some guidelines for selecting the urgency in the /security form:
When evaluating security issues or MRs, it can be useful to have a way to reproduce issues, dig in to root causes, look for further impacts. This can also be a great way to get familiar with GitLab during your first few weeks of onboarding. Here are some handy tips & tricks.
Get a version of GitLab to play with Many AppSec engineers will opt to install GitLab locally using GDK.
A curated list of the most frequently asked AppSec related questions
This runbook describes the process for times when the Application Security team has team members available during holidays, Friends and Family days, or other events where many team members are expected to be out of office.
Since Application Security is not on-call, we aim to provide this best-effort coverage that may or may not be available in every region.
If you need assistance:
Post in #sec-appsec with details Find the AppSec engineer providing coverage - see Communicating the coverage below Reach out to the team member via Slack @ mention If urgent or the team member does not respond in a timely manner, reach out via text message or phone call after checking the contact preferences listed on their Slack profile If it is unclear who is providing coverage, ping the @appsec-team in Slack Expectations for those doing coverage In the event that AppSec is needed to respond to an S1 incident or otherwise urgently assist another team, the team member providing coverage should be reachable by Slack or phone AppSec team members providing coverage are not expected to be at the keyboard and working, but they should be able to get online within one hour if needed If they are not actively working during coverage times, they should try to check Slack occasionally for pings They should occasionally check for potential S1s coming in through HackerOne and take action if it requires immediate attention If an S1 incident occurs or AppSec assistance is urgently needed by another team, the expectation is that the AppSec team member will attempt to triage the situation and escalate as needed to a security manager Planning When holidays or Friends and Family days are coming up, the AppSec team should try to find a team member who can swap their day off.
This review template is tailored to application security reviews of GitLab features. Parts of it might be applicable to other software, other parts might not.
This threat modeling process is tailored to GitLab features.
How can AppSec Engineers Contribute to the Secure Code Warrior Training Program? If anyone from the AppSec team is interested in contributing to the Secure Code Warrior (SCW) training program and you don’t have access to the Secure Code Warrior training portal, please post a comment in the #sec-appsec Slack channel requesting access.
Once you have access to the Secure Code Warrior training portal, please do the following:
Join the #security-secure-code-warrior Slack channel Watch Secure Code Warrior platform walkthrough for a quick overview of the platform Please read through SCW’s documentation on Getting Started Go through our old SCW Trial issue to get an idea on how we have organized Tournaments in SCW.
Bug Hunting Day Process The Application Security Team has a bug hunting day on the last Friday of every month.
Goal The objective is to have some time to investigate on something a team member has observed, discussed or noticed during a review but didn’t have the time to go into a detailed review or testing.
Bug hunting day is not mandatory. There may be months that are particularly busy and we may not be doing bug hunting days on months like that.
Please refer to the GitLab CVSS Calculator as the single-source-of-truth to determine CVSS scores on GitLab security issues.
The following chart is intended for ~"severity::1" issues. Other issues should follow the regular security release process.
flowchart TD Start(S1 SIRT incident) --> Mitigated{Is the incident mitigated<br>by a Cloudflare rule,<br>a feature flag, a<br>configuration change, etc.?} Mitigated -->|No| QuickFix{Can a fix be developed quickly?} QuickFix --> |Yes| RegularRelease QuickFix --> |No| HotPatch(Deploy a Hot Patch with a 'hacky' mitigation while the long term<br>fix is being figured out by the team that owns the feature.
Certain customers scan containers that GitLab provides for known vulnerabilities and other security concerns. The GitLab Federal Application Security team is responsible for reviewing these results. In many cases, these findings are related to dependencies that need to be updated to address vulnerabilities.
The expectation is that the Federal Application Security Team will verify every finding in each scanned container and make a determination as to whether the finding is a true positive, false positive, or a duplicate.
This content has moved to the internal handbook: https://internal.gitlab.com/handbook/security/application_security/runbooks/fedramp-scanner-triage-process/.
When a new gem is added to our Gemfile or when versions are changed in Gemfile.lock, we ask developers to reach out to the AppSec team to request a review. As an AppSec engineer performing the review, please follow the steps mentioned below to perform reviews like these:
First, look to see if the gem is well maintained. Look closer at the gem’s code to see if there are any anomalies.
The main difference is in who initiates the release process.
Critical patch releases are initiated on-demand by the security team to resolve the most severe vulnerabilities that are labelled as S1. They are highly urgent and need to be released as soon as the patch is ready. Planned patch releases are scheduled on second and fourth Wednesday of a month and are initiated by the release management team. Cross-regional AppSec Release Managers In order to make sure there is always someone available and ready to help get the release finished, the AppSec team assigns two team members to each release.
GitLab utilizes HackerOne for its bug bounty program. Security researchers can report vulnerabilities in GitLab applications or the GitLab infrastructure via the HackerOne website. Team members authorized to respond to HackerOne reports use procedures outlined here. The #hackerone-feed Slack channel receives notifications of report status changes and comments via HackerOne’s Slack integration.
Queues New contains all reports in the New state GitLab Team contains reports that have been validated by the HackerOne triage team, but are yet to be assigned to a specific GitLab team member H1 Triage are reports being triaged by the HackerOne triage team Pending Disclosure are reports that should be reviewed and disclosed GitLab Team On-boarding New members of the GitLab security team are granted access to the GitLab HackerOne team via an access request issue using the appropriate role based entitlement template, which should be submitted by their manager during onboarding During onboarding, new GitLab security team members will be invited to join the HackerOne program if their role requires it.
The runbook for handling different scenarios of unintended vulnerability disclosures.
Third parties Sometimes the root cause of a vulnerability is in a third-party dependency and even the latest version of that dependency is vulnerable. This page describes how we process in those situations.
The first step is to disclose the vulnerability to the third party. See in the project’s README.md, SECURITY.md or on their website if they have official means of disclosing vulnerabilities. If they don’t, simply email the maintainer(s) of the project.
List of Package Hunter Findings Any Package Hunter related finding can be found on this dashboard (internal link).
Network Connection Findings We’re going to use this finding as an example. In this finding, Package Hunter detected that a package opened a network connection.
02:24:56.163600570: Notice Disallowed outbound connection destination (command=node scripts/install.js connection=172.17.0.2:36884->52.217.109.44:443 user=root container_id=fb38d63ef02a container_name=some-container-eae8d3ec-a482-441b-8262-78198b46fdfb.tgz image=maldep)
Investigation We can look at the destination IP address and see if gives any interesting information In this example, it’s a an AWS IP address but it gives us no information about what package might have contacted this IP Attention: This step requries to checkout the branch for which the finding was made.
The Merge Monitor tool looks in public GitLab repositories that JiHu contributes to for merge requests that:
Were merged Were labeled as a JiHu Contribution Were not labeled with the label that AppSec team members need to apply after conducting security reviews of JiHu contributions Any findings will be included in reports that are created as issues in the jihu_merge_request_monitor_reports repository. The Federal AppSec team is pinged on each report that is created and the expectation is that they will review each finding.
Frequency: Daily
AppSec engineers are responsible for triaging the findings of the GitLab security tools. This role has two primary functions.
Validate the findings and handoff to engineering for correction Provide feedback to the Secure team Security Dashboard List The following is a list of security dashboards that need to be reviewed:
GitLab GitLab Omnibus Gitaly GitLab Pages GitLab Shell k8s-workloads Version UBI images release-cli vscode-extension Customers elastic-search-indexer GitLab DAST GitLab Terminal GitLab Agent Compensation Calculator Validation For each finding:
Application Security team members are alphabetically assigned as the responsible individual (DRI) for incoming requests to the Application Security team for a given calendar week in the Triage Rotation Google Sheet in the Security Team Drive.
One application security engineer is assigned this task each week and can be found at Triage Rotation.
The following rotations are defined:
(Weekly Assignment) HackerOne + Security Dashboard Review Point of contact for “New” HackerOne reports during that week.
The review of a fix by an application security engineer is triggered by the engineer implementing the fix. As the engineer engaging in the verification, these are the steps that should be followed:
Doing a code review from the security perspective. Validating the security fix using the Docker image generated by package-and-qa, see the section below for more details. Once the validation is done, update the security issue with a link to the CVE.
