Secure Sub-Department
The Secure engineering sub-department is responsible for the Secure Stage of the product.
Vision
To provide content and tools to support the best possible assessment at the earliest possible moment.
Following our single application paradigm,
we integrate and build scanning tools to supply security and compliance assessment data to the main GitLab application
where we develop our vulnerability management system and other features.
While it might be technically feasible, we do not aim at building standalone products that provide this data independently from the GitLab application.
For more details about the vision for this area of the product, see the Secure stage page.
Mission
To support the success of GitLab by developing highly usable, hiqh quality tools for customers to build more secure software.
The Secure team works on GitLab’s Secure stage.
Sub-department development people leaders
To contact Secure stage development people leaders, use the following aliases:
- GitLab:
@gitlab-org/secure/managers
- Slack:
@s_secure_managers
Team Members
The following people are permanent members of the Secure Sub-Department:
Composition Analysis
Team Page: Composition Analysis
| Name |
Role |
 Fabien Catteau
|
Staff Backend Engineer, Secure:Composition Analysis |
 Igor Frenkel
|
Senior Backend Engineer, Secure:Composition Analysis |
 Joey Khabie
|
Backend Engineer |
|
Miki Amos
|
Senior Backend Engineer, Secure:Composition Analysis |
 Nick Ilieskou
|
Senior Backend Engineer, Secure:Composition Analysis |
 Olivier Gonzalez
|
Staff Backend Engineer, Secure:Composition Analysis |
|
Orin Naaman
|
Backend Engineer, Secure:Composition Analysis |
 Oscar Tovar
|
Senior Backend Engineer, Secure:Composition Analysis |
|
Tal Kopel
|
Manager, Software Engineering, Secure:Composition Analysis |
 Thiago Figueiró
|
Manager, Software Engineering, Secure:Composition Analysis |
|
Yasha Rise
|
Senior Backend Engineer, Secure:Composition Analysis |
 Zamir Martins Filho
|
Senior Fullstack Engineer, Secure:Composition Analysis |
Static Analysis
Team Page: Static Analysis
Dynamic Analysis
Team Page: Dynamic Analysis
Vulnerability Research
Team Page: Vulnerability Research
Stable Counterparts
The following members of other functional teams are our stable counterparts:
| Name |
Role |
 Alexander Dietrich
|
Senior Backend Engineer, Secure:Dynamic Analysis |
 Connor Gilbert
|
Principal Product Manager, Secure:Static Analysis |
 Dheeraj Joshi
|
Senior Frontend Engineer, Secure:Secret Detection |
|
Fabien Catteau
|
Staff Backend Engineer, Secure:Composition Analysis |
|
Fernando Cardenas
|
Senior Frontend Engineer, Secure:Composition Analysis |
 Lucas Charles
|
Principal Engineer, Secure & Govern |
|
Miki Amos
|
Senior Backend Engineer, Secure:Composition Analysis |
 Nikhil George
|
Senior Security Engineer, Application Security, Secure (Static Analysis, Dynamic Analysis, Composition Analysis, Vulnerability Research), Growth (Acquisition, Activation). |
 Ross Fuhrman
|
Senior Backend Engineer, Secure:Secret Detection |
 Russell Dickenson
|
Senior Technical Writer, Secure |
 Sarah Waldner
|
Group Manager, Product Management, Secure |
 Sara Meadzinger
|
Senior Product Manager, Secure:Dynamic Analysis |
 Thomas Woodham
|
Senior Engineering Manager, Secure |
 Will Meek
|
Senior Software Engineer in Test, Secure |
Secure Team
The Secure Team (previously known as the Security Products Team) is responsible for the security checks features in the GitLab platform, and maps to the secure transversal stage.
You can learn more about our approach on the Secure Vision page.
The features provided by the Secure Team are mostly present at the pipeline level, and mostly available as Docker images.
This particularity shapes our processes and QA, which differs a bit from the other backend teams.
Security Products
We still refer to “Security Products” as the tools developed by the Secure Team. Hence the home of our projects in GitLab: https://gitlab.com/gitlab-org/security-products/.
We strive to maintain a consistent User Experience across our Security Products but we do not enforce consistency at the implementation level.
Each group faces its own challenges and is in the best position to make the technical choices it deems are the most suitable to achieve its goals.
While UX inconsistencies are considered as bugs,
we rely on individual teams to make smart decisions about when consistency is important and when divergence makes more sense
— either because the divergence itself creates a better experience or because of velocity considerations.
Domains of Expertise
SAST
SAST (Static Application Security Testing) refers to static code analysis.
GitLab leverages the power of various opensource tools to provide a wide range of checks for many languages and support.
These tools are wrapped inside docker images which ensure we get a standard output from there.
An orchestrator, developed by GitLab, is in charge of running these images, and gathering all the data needed to generate the final report.
DAST
DAST (Dynamic Application Security Testing) is used to hit a live application.
Because some vulnerabilities can only be detected once all the code is actually running, this method complements the static code analysis.
DAST is relying on OWASP Zed Attack Proxy Project, modified by GitLab to enable authentication.
Dependency Scanning
Dependency Scanning is used to detect vulnerabilities introduced by external dependencies in the application.
Because a large portion of the code shipped to production is actually coming from third-party libraries, it’s important to monitor them as well.
Dependency Scanning is relying mostly on the Gemnasium engine.
Fuzz Testing
Coverage-guided fuzzing and API fuzzing are used to automatically input data into applications or web apis that has the potential to cause crashes or bugs. Coverage-guided fuzzing relies on open-sourced language-specific fuzzers. API Fuzzing is based on a proprietary GitLab engine.
License Compliance
License Compliance helps with the licenses introduced by third-party libraries in the application.
Licence management relies on the LicenseFinder gem.
Vulnerability Research
The Vulnerability Research team’s purpose is
to perform research and develop proofs of concepts that increase the
capabilities and effectiveness of the
Secure stage.
Label Usage
If you are submitting an issue about a Secure Stage feature, use ~devops::secure and one of the following group labels to get the issue in front of the most appropriate team members.
| Label |
Use |
~devops::secure |
All issues related to the Secure Stage |
Additional labels should be added according to the Workflow Labels Documentation.
Skills
Because we have a wide range of domains to cover, it requires a lot of different expertises and skills:
| Technology skills |
Areas of interest |
| Ruby on Rails |
Backend development |
| Go |
SAST, Dependency Scanning, DAST |
| Python |
DAST |
| SQL (PostgreSQL) |
Dependency Scanning / all |
| Docker |
Container Scanning / all |
| C# |
API Security |
Our team also must have a good sense of security, with at least basic skills in application security.
We provide tools for many different languages (ex: sast, dependency scanning, license compliance). It means our team is able to understand the basics of each of these languages, including their package managers. We maintain tests projects to ensure our features are working release after release for each of them.
Release process
See Versioning and release process.
QA process
See QA Process for more info.
Vulnerability Management process
Automation
We use the security-triage-automation tool in conjunction with scheduled pipelines in the release project to handle the following tasks:
- Create security issues for FedRAMP vulnerabilities (Container Scanning results only) still detected on the default branch, executed at least once, on the first day of the month to match with FedRAMP compliance report cadence.
Note that we do not yet automatically create security issues for non-FedRAMP vulnerabilities. Please see the Non-FedRAMP vulnerabilities section for more details.
- Resolve all vulnerabilities (both FedRAMP and non-FedRAMP) no longer detected on the default branch and close their issues, executed every 2 days.
The Vulnmapper tool also provides some automation to vulnerability management like:
- Adding labels to security issues to further classify the fix availability (fix_available, fix_unavailable, will_not_be_fixed, etc.).
- Creating Deviation Request issues for FedRAMP related security issues that should have one.
Note: Our goal is to centralize automation for vulnerability management in the Vulnmapper tool in the nearest future and standardize our processes across the company. However, so far we’re following the existing process based on the security-triage-automation tool.
Automation failures
It’s possible that our security automation tooling may fail.
If this occurs, and the issue cannot be immediately resolved, open an issue to
track the error. Then, announce the failure in #s_secure to raise awareness,
and follow the manual security triage process outlined below.
View manual process fallback when automation fails
Manually reviewing and resolving vulnerabilities
On a weekly basis: review the vulnerability report to resolve no longer detected ones and close related issues. Note: It is not necessary to investigate vulnerabilities that are no longer detected.
- Visit
Vulnerability Report Dashboards to verify that there are vulnerabilities that can be resolved.
- Execute the
security-triage-automation tool to resolve vulnerabilities and close their issues. This tool must be executed separately for each project that have vulnerabilities to resolve.
- Verify in
Vulnerability Report Dashboards that vulnerabilities have been resolved.
Manually creating security issues for FedRAMP vulnerabilities
Last working day before the 1st of the month, create security issues
for FedRAMP vulnerabilities of the CONTAINER_SCANNING type, and CRITICAL, HIGH,
MEDIUM, LOW, and UNKNOWN severity levels by executing the security-triage-automation
tool to process vulnerabilities for a given project
(please make sure to adjust CLI options accordingly). This tool must be executed
separately for each project.
Manually creating deviation requests for FedRAMP vulnerabilities
Vulnmapper automatically creates Deviation Requests but may fail for various reasons, such as the absence of analysis from NVD.
In cases where automation fails, you must create the Deviation Requests manually before the issues reach SLA.
To do so, use the following procedure.
-
Open a DR issue with the operational requirement template.
- Update the
Vulnerability Details section with a link to the advisory (RedHat tracker usually), CVE ID, severity, and CVSS score.
- Update the
Justification Section with:
The OS vendor has published an updated advisory for <CVE_ID>, indicating that package <PACKAGE_NAME> has not yet had a fix released for this vulnerability. Until a fix is available for the package, this vulnerability cannot practically be remediated.
- Update the
Attached Evidence section with:
As this operational requirement represents a dependency on a vendor-published package to address this vulnerability, no additional evidence has been supplied. Please refer to the linked vendor advisory in the above justification.
- Link it to the security issue:
/relate <issue_id>
-
Update the security issue accordingly
/label ~"FedRAMP::Vulnerability" ~"FedRAMP::DR Status::Open"
/milestone %Backlog
Troubleshoothing
GITLAB_ACCESS_TOKEN has expired. The automation relies on API requests to manage vulnerabilities and issues on various projects. This requires specific permissions and authentication is achieved with a Private Access Token generated on the service account gl-service-security-triage (credentials available in 1Password). If the token is expired, a new one (with api scope) must be generated by signing in with this account on gitlab.com and then the new value must be configured in the settings of the release project.
FedRAMP vulnerabilities
To ensure compliance, the management of FedRAMP vulnerabilities is handled by automation. Please check the manual process fallback for additional details.
Non-FedRAMP vulnerabilities
We do not yet have the same automation in place for non-FedRAMP vulnerabilities since it represents a too important volume to manage for our teams and some necessary improvements in the vulnmapper tool are required prior to enabling this.
In the meantime, we favor a more specialized approach for these vulnerabilities and there is no standardized process across the groups.
Error Monitoring
500 errors on gitlab.com are reported to Sentry. Below are some quick links to pull up Sentry errors related to Secure.
Brainstorming sessions
Our team occasionally schedules synchronous brainstorming sessions as a method of deep-diving on a specific topic.
This approach can be useful in breaking down complexity and deriving actionable steps for problems that lack
definition.
These are purposefully freeform to allow for creative problem solving.
When possible, time should be reserved for a list of actions to be taken from the open discussion.
Brainstorming Sessions Doc (Internal): https://docs.google.com/document/d/179JL5RzbgSIz2XZewbYn79cuX7_vUtte_TcoLwUUC5o/edit#
Examples of previous brainstorming topics:
Resources
Product Documentation
As the product evolves, it is important to maintain accurate and up to date documentation for our users. If it is not documented, customers may not know a feature exists.
To update the documentation, the following process should be followed:
- When an issue has been identified as needing documentation, add the
~Documentation label, outline in the description of the issue what documentation is needed, and assign a Backend Engineer and Technical Writer(TW) to the issue (find the appropriate TW by searching the product categories).
- If the task is documentation only, apply a
~Px label.
- For documentation around features or bugs, a backend engineer should write the documentation and work with the technical writer for editing. If the documentation only needs styling cleanup, clarification, or reorganization, this work should be lead by the Technical Writer with support from a BE as necessary. The availability of a technical writer should in no way hold up work on the documentation.
Further information on the documentation process.
Async Daily Standups
Since we are a remote company, having daily standup meetings would not make any sense, since we’re not all in the same timezone.
That’s why we have async daily standups, where everyone can give some insights into what they did yesterday, what they plan to do today, etc.
For that, we rely on the geekbot slack plugin to automate the process.
- Use the “
description in backquote + [link to issue](#)” format when mentioning issues in your standup report.
- Prepend CI status icons to the answer lines for
What did you do since yesterday? to denote the current state:
for successfully accomplished tasks (:ci_passing: emoji)
for tasks that were due on some period of time but were not accomplished (:ci_failing: emoji)
for tasks currently in progress (:ci_running: emoji)
for paused or postponed tasks (:ci_pending: emoji)- any other
:ci_... icon you find applicable
Example:
What did you do since yesterday?
Slack Channels:
As our teams focus on different areas, we have Geekbot configured to broadcast to separate channels in addition to our common one at [#s_secure-standup].
- Composition Analysis: #g_secure-composition-analysis-standup
- Static Analysis: #g_secure-static-analysis-standup
- Dynamic Analysis: #g_secure-dynamic-analysis-standup
- Frontend: #s_secure-frontend-standup
Recorded meetings
Our important meetings are recorded and published on YouTube, in the GitLab Secure Stage playlist.
They give a good overview of the decision process, which is often a discussion with all the stakeholders. As we are a remote company, these video meetings help to synchronize and take decisions faster than commenting on issues. We prefer asynchronous work, but for large features and when the timing is tight, we can detail a lot of specifications. This will make the asynchronous work easier, since we have evaluated all edge cases.
Calendar
We welcome team members to join meetings that are on our shared calendar. The Secure Calendar is available to all logged in GitLab team members.
GitLab is an extremely active organization which generates a lot of news and activity each week. Everyone in Secure are encouraged to keep themselves informed as to what is happening in the larger organzation. Everyone is also
encouraged to contribute to these channels and communication paradigms when you have information to share.
In addition to this, each group in Secure conducts a weekly synchronous meeting. These meetings are publicized on the Secure Calendar mentioned above. As always at GitLab, we strive to make meeting attendance optional.
In addition to keeping yourself informed, team members are encouraged to keep others informed as well. Secure groups have adopted a practice of including the following topics as standing agenda items in their weekly meetings, with example
topics for each bullet point.
- Current status
- Work recently achieved against top priorities for that milestone.
- Pre-recorded demos are appreciated and encouraged as part of these updates.
- Newly discovered scope or dependencies.
- Risks
- Issues which are blocked or slowed, impacting whether they can be delivered in the desired timeframe.
- Help wanted
- Issues or topics on which the team or individuals on the team are getting stuck and could use some help.
- Praise
- Anyone doing a great job and you want to give them kudos?
- Any bit of work which has been delivered that’s exceptional?
Engineering Managers are responsible for populating this section of weekly group meetings, though everyone can contribute. In addition to helping the group keep itself informed about what’s happening each week, the SEM for Secure will collect
this information weekly and broadcast a curated list to the section.
Technical onboarding
New hires should go through these steps and read the corresponding documentation when onboarding in the Secure Team.
Every new hire will have an assigned onboarding issue that will guide them through the whole process.
Workflow and Refinement
Secure largely follows our Product Development Flow
See Issue Refinement to learn how we evaluate complexity, level of effort, our implementation plan and assign issue weights.
Shared pool of Frontend work
From time to time, the demand for frontend engineering tasks does not match the capacity available in Secure groups. If a group could use frontend support, they are encouraged to add the
~secure::frontend backlog label to issues which are ready to be picked up. Additionally,
Engineering Managers are encouraged to look at this queue of work during milestone planning if there is frontend capacity available.
Coding standards and style guidelines
The Secure Team follows the coding standards and style guidelines outlined in the company-wide Contributor and Development Docs, however, please consult the following guidelines which are specific to the Secure Team:
Cross group collaboration
Some components of the architecture that support Secure features are shared between multiple groups like the common Go library,
the Security Report Schemas, the rails parsers, etc.
Modifying these shared pieces might impact other groups so we should rely as much as possible on approval rules to ensure
such changes are reviewed by the relevant teams before being merged.
Impactless two-way door changes could skip the approval process, please use sound judgement and common sense in such situations.
The author of changes should announce broadly the changes made on these components to raise awareness (weekly meeting agenda, slack channel).
Development of new analyzers
For a complete guide about developing a new analyzer please refer to our user documentation
Technical Documentation
As our product evolves, the engineering teams are researching ways to achieve new functionality and improve our architecture.
The outcome of this research can be found in our Technical Documentation section.
Data Sources
We keep a list of data sources in our internal wiki. This includes advisory databases, package license information, and related data.
Retrospectives
The Secure sub-department conducts retrospectives at the group level that follow our engineering workflow.
Each group’s DRI is responsible to prepare and schedule the retrospective sync sessions and the async retrospective issues can be found in the corresponding project.
After all groups have completed their retrospective, we conduct a Section Retrospective.
NB: we use to have a sub-department wide retrospective whose issues are still accessible in the deprecated project.
Analytics
The Secure group reviews analytics to help understand customers and their usage of the tools. This data helps drive product and technical decisions. The following links show usage of Secure functionality.
Secure Scan Metrics Dashboard
Cross-functional Backlog
We also track our backlog of issues, including past due security and infradev issues, and total open SUS-impacting issues and bugs.
Merged Merge Request Types
MR Type labels help us report what we’re working on to industry analysts in a way that’s consistent across the engineering department. The dashboard below shows the trend of MR Types over time and a list of merged MRs.
Common Links
API Security
The API Security team is a standalone team which is part of the Dynamic Analysis group at GitLab. It is charged with developing solutions which perform Fuzzing.
| Repo |
Purpose |
| API Fuzzer - Private |
GitLab’s API Fuzzing scanner. |
Important Fuzzing repositories
- Slack channel: #g_ast-dynamic-analysis
- Slack alias: @secure_dynamic_analysis_be
- Google groups: dynamic-analysis-be@gitlab.com
- GitLab mention: @gitlab-org/secure/dynamic-analysis-be
How We Work
The Dynamic Analysis group largely follows GitLab’s Product Development Flow.
The Composition Analysis group at GitLab is charged with developing solutions which perform Container and Dependency Scanning and License Compliance.
Vision
Facilitate the Application Security Testing stage achieving lovable maturity and supporting the best possible security assessment at the earliest possible moment.
Customer outcomes we are driving for GitLab
Vulnerability Research sits at the crossroads between the Application Security Testing stage itself, and customers of the stage. We provide enhancements to our products or processes to ensure our products and services are more effective for GitLab’s customers.
We strive to enhance our customer experience with regards to providing correct and accurate results from our services.
Dynamic Analysis
The Dynamic Analysis group at GitLab is charged with developing solutions which perform Dynamic Analysis Software Testing (DAST) and Fuzzing. Our work is a mix of open and closed source code.
Mission
To support the success of GitLab by developing highly usable, hiqh quality tools for customers to build more secure software. The Dynamic Analysis group at GitLab is charged with developing solutions which perform API Security Testing, Dynamic Analysis Software Testing (DAST) and Fuzzing.
Secret Detection
The Secret Detection group was split from Static Analysis on 2024-03-05.
The Secret Detection group maintains the Secret Detection feature category for customer software repositories.
Software Delivery
For software delivery, we generally follow a similar process to the one used by Static Analysis group.
Refinement
We have recently experimented with a new refinement process when refining the issues for the Pre-receive SD beta/.com epic. This process draws a lot of inspiration from other sections/stages but also aligns with the current Secure Engineering Refinement.
Overview
Engineering refinement is the most important step to ensure an issue is ready to move into development and that the issue will match
everyone’s expectations when the work is delivered.
The goal of the refinement process is to
- Identify and resolve outstanding questions or discussions.
- Raise any questions, concerns or alternative approaches.
- Outline an implementation plan.
- Ensure issue is ready to be worked on.
- Identify code boundaries, for example, does the issue change code maintained by another team.
- Notify other teams if the issue is relevant to them in some way.
- Assign a weight to the issue.
The refinement process can break down the issue into technical subtasks by following the sub-issue convention but we should avoid redefining the scope of an implementation issue as this should have already been done during the Planning Breakdown with UX and PM.
Everything starts with a Merge Request
We expect and require all contributions to our products to go a merge request with a formal review. As such, we follow the Merge Request workflow and code review guidelines articulated in GitLab’s developer documentation. We would, however, like to highlight a few items from these documents and add a few additional considerations for reviewers and authors.
Additional considerations for Merge Request reviewers
- The best way to unblock a peer or community member is to provide feedback in a timely manner. If you are at capacity and cannot facilitate a review in the SLO to which we aspire, please let folks know in the merge request so another reviewer may be found.
Additional considerations for Merge Request authors
- Being a globally distributed organization can, and frequently does, add latency to back-and-forth communication between folks. Don’t take it personally if it’s taking longer than you expected to get feedback on your changes.
Secure QA Process
The secure analyzers verify merge requests by running a new commit against downstream test projects for their supported languages/frameworks (i.e. the gemnasium analyzer of Dependency Scanning will trigger tests against php, go, and several other test projects). The verification is done by comparing the generated report output against an expected report committed to the analyzer’s repository. If analyzer behavior has changed, then the pipeline will fail because the contents of the expected and generated reports will no longer match.
Architecture
Overview
The architecture supporting the Secure features is split into two main parts.
flowchart LR
subgraph G1[Scanning]
Scanner
Analyzer
CI[CI Jobs]
end
subgraph G2[Processing, visualization, and management]
Parsers
Database
Views
Interactions
end
G1 --Report Artifact--> G2
Scanning
The scanning part is responsible for finding vulnerabilities in given resources and exporting results.
The scans are executed in CI jobs via several small projects called Analyzers which can be found in our Analyzers sub-group.
The Analyzers are small wrappers around in-house or external security tools called Scanners to integrate them into GitLab.
The Analyzers are mainly written in Go and rely on our Common Go library.
Static Analysis
The Static Analysis group at GitLab is charged with developing the Static Application Security Testing (SAST) feature category for customer software repositories.
Links
How We Work
The Static Analysis group is largely aligned with GitLab’s Product Development Flow,
however there are some notable differences in how we seek to deliver software. The engineering team
predominantly concerns itself with the delivery of software, which is the portion of the workflow
states where we deviate the most. What follows is how we manage the handoff from product management
to engineering to deliver software.
Bhavya Kaushal
Gal Katz
Or Gal
Senior Backend Engineer
Daniel Abeles
Isaac Dawson
Michael Henriksen