The Secure engineering sub-department is responsible for the Secure Stage of the product.
Vision
To provide content and tools to support the best possible assessment at the earliest possible moment.
Following our single application paradigm,
we integrate and build scanning tools to supply security and compliance assessment data to the main GitLab application
where we develop our vulnerability management system and other features.
While it might be technically feasible, we do not aim at building standalone products that provide this data independently from the GitLab application.
For more details about the vision for this area of the product, see the Secure stage page.
Mission
To support the success of GitLab by developing highly usable, hiqh quality tools for customers to build more secure software.
The Secure Team (previously known as the Security Products Team) is responsible for the security checks features in the GitLab platform, and maps to the secure transversal stage.
You can learn more about our approach on the Secure Vision page.
The features provided by the Secure Team are mostly present at the pipeline level, and mostly available as Docker images.
This particularity shapes our processes and QA, which differs a bit from the other backend teams.
We strive to maintain a consistent User Experience across our Security Products but we do not enforce consistency at the implementation level.
Each group faces its own challenges and is in the best position to make the technical choices it deems are the most suitable to achieve its goals.
While UX inconsistencies are considered as bugs,
we rely on individual teams to make smart decisions about when consistency is important and when divergence makes more sense
— either because the divergence itself creates a better experience or because of velocity considerations.
Domains of Expertise
SAST
SAST (Static Application Security Testing) refers to static code analysis.
GitLab leverages the power of various opensource tools to provide a wide range of checks for many languages and support.
These tools are wrapped inside docker images which ensure we get a standard output from there.
An orchestrator, developed by GitLab, is in charge of running these images, and gathering all the data needed to generate the final report.
DAST
DAST (Dynamic Application Security Testing) is used to hit a live application.
Because some vulnerabilities can only be detected once all the code is actually running, this method complements the static code analysis.
DAST is relying on OWASP Zed Attack Proxy Project, modified by GitLab to enable authentication.
Dependency Scanning
Dependency Scanning is used to detect vulnerabilities introduced by external dependencies in the application.
Because a large portion of the code shipped to production is actually coming from third-party libraries, it’s important to monitor them as well.
Dependency Scanning is relying mostly on the Gemnasium engine.
Fuzz Testing
Coverage-guided fuzzing and API fuzzing are used to automatically input data into applications or web apis that has the potential to cause crashes or bugs. Coverage-guided fuzzing relies on open-sourced language-specific fuzzers. API Fuzzing is based on a proprietary GitLab engine.
License Compliance
License Compliance helps with the licenses introduced by third-party libraries in the application.
Licence management relies on the LicenseFinder gem.
Vulnerability Research
The Vulnerability Research team’s purpose is
to perform research and develop proofs of concepts that increase the
capabilities and effectiveness of the
Secure stage.
Label Usage
If you are submitting an issue about a Secure Stage feature, use ~devops::secure and one of the following group labels to get the issue in front of the most appropriate team members.
Because we have a wide range of domains to cover, it requires a lot of different expertises and skills:
Technology skills
Areas of interest
Ruby on Rails
Backend development
Go
SAST, Dependency Scanning, DAST
Python
DAST
SQL (PostgreSQL)
Dependency Scanning / all
Docker
Container Scanning / all
C#
API Security
Our team also must have a good sense of security, with at least basic skills in application security.
We provide tools for many different languages (ex: sast, dependency scanning, license compliance). It means our team is able to understand the basics of each of these languages, including their package managers. We maintain tests projects to ensure our features are working release after release for each of them.
It’s possible that our security automation tooling may fail.
If this occurs, and the issue cannot be immediately resolved, open an issue to
track the error. Then, announce the failure in #s_secure to raise awareness,
and follow the manual security triage process outlined below.
View manual process fallback when automation fails
Manually reviewing and resolving vulnerabilities
On a weekly basis: review the vulnerability report to resolve no longer detected ones and close related issues. Note: It is not necessary to investigate vulnerabilities that are no longer detected.
Visit Vulnerability Report Dashboards to verify that there are vulnerabilities that can be resolved.
Execute the security-triage-automation tool to resolve vulnerabilities and close their issues. This tool must be executed separately for each project that have vulnerabilities to resolve.
Verify in Vulnerability Report Dashboards that vulnerabilities have been resolved.
Manually creating security issues for FedRAMP vulnerabilities
Last working day before the 1st of the month, create security issues
for FedRAMP vulnerabilities of the CONTAINER_SCANNING type, and CRITICAL, HIGH,
MEDIUM, LOW, and UNKNOWN severity levels by executing the security-triage-automation
tool to process vulnerabilities for a given project
(please make sure to adjust CLI options accordingly). This tool must be executed
separately for each project.
Manually creating deviation requests for FedRAMP vulnerabilities
Vulnmapper automatically creates Deviation Requests but may fail for various reasons, such as the absence of analysis from NVD.
In cases where automation fails, you must create the Deviation Requests manually before the issues reach SLA.
To do so, use the following procedure.
Update the Vulnerability Details section with a link to the advisory (RedHat tracker usually), CVE ID, severity, and CVSS score.
Update the Justification Section with:
The OS vendor has published an updated advisory for <CVE_ID>, indicating that package <PACKAGE_NAME> has not yet had a fix released for this vulnerability. Until a fix is available for the package, this vulnerability cannot practically be remediated.
Update the Attached Evidence section with:
As this operational requirement represents a dependency on a vendor-published package to address this vulnerability, no additional evidence has been supplied. Please refer to the linked vendor advisory in the above justification.
GITLAB_ACCESS_TOKEN has expired. The automation relies on API requests to manage vulnerabilities and issues on various projects. This requires specific permissions and authentication is achieved with a Private Access Token generated on the service account gl-service-security-triage (credentials available in 1Password). If the token is expired, a new one (with api scope) must be generated by signing in with this account on gitlab.com and then the new value must be configured in the settings of the release project.
FedRAMP vulnerabilities
To ensure compliance, the management of FedRAMP vulnerabilities is handled by automation. Please check the manual process fallback for additional details.
Non-FedRAMP vulnerabilities
We do not yet have the same automation in place for non-FedRAMP vulnerabilities since it represents a too important volume to manage for our teams and some necessary improvements in the vulnmapper tool are required prior to enabling this.
In the meantime, we favor a more specialized approach for these vulnerabilities and there is no standardized process across the groups.
Error Monitoring
500 errors on gitlab.com are reported to Sentry. Below are some quick links to pull up Sentry errors related to Secure.
Our team occasionally schedules synchronous brainstorming sessions as a method of deep-diving on a specific topic.
This approach can be useful in breaking down complexity and deriving actionable steps for problems that lack
definition.
These are purposefully freeform to allow for creative problem solving.
When possible, time should be reserved for a list of actions to be taken from the open discussion.
As the product evolves, it is important to maintain accurate and up to date documentation for our users. If it is not documented, customers may not know a feature exists.
To update the documentation, the following process should be followed:
When an issue has been identified as needing documentation, add the ~Documentation label, outline in the description of the issue what documentation is needed, and assign a Backend Engineer and Technical Writer(TW) to the issue (find the appropriate TW by searching the product categories).
If the task is documentation only, apply a ~Px label.
For documentation around features or bugs, a backend engineer should write the documentation and work with the technical writer for editing. If the documentation only needs styling cleanup, clarification, or reorganization, this work should be lead by the Technical Writer with support from a BE as necessary. The availability of a technical writer should in no way hold up work on the documentation.
Further information on the documentation process.
Async Daily Standups
Since we are a remote company, having daily standup meetings would not make any sense, since we’re not all in the same timezone.
That’s why we have async daily standups, where everyone can give some insights into what they did yesterday, what they plan to do today, etc.
For that, we rely on the geekbot slack plugin to automate the process.
Standup messages format
Use the “description in backquote + [link to issue](#)” format when mentioning issues in your standup report.
Prepend CI status icons to the answer lines for What did you do since yesterday? to denote the current state:
for successfully accomplished tasks (:ci_passing: emoji)
for tasks that were due on some period of time but were not accomplished (:ci_failing: emoji)
for tasks currently in progress (:ci_running: emoji)
for paused or postponed tasks (:ci_pending: emoji)
Catch-up on all emails and threads after the vacation
Slack Channels:
As our teams focus on different areas, we have Geekbot configured to broadcast to separate channels in addition to our common one at [#s_secure-standup].
Our important meetings are recorded and published on YouTube, in the GitLab Secure Stage playlist.
They give a good overview of the decision process, which is often a discussion with all the stakeholders. As we are a remote company, these video meetings help to synchronize and take decisions faster than commenting on issues. We prefer asynchronous work, but for large features and when the timing is tight, we can detail a lot of specifications. This will make the asynchronous work easier, since we have evaluated all edge cases.
Calendar
We welcome team members to join meetings that are on our shared calendar. The Secure Calendar is available to all logged in GitLab team members.
Staying informed
GitLab is an extremely active organization which generates a lot of news and activity each week. Everyone in Secure are encouraged to keep themselves informed as to what is happening in the larger organzation. Everyone is also
encouraged to contribute to these channels and communication paradigms when you have information to share.
In addition to this, each group in Secure conducts a weekly synchronous meeting. These meetings are publicized on the Secure Calendar mentioned above. As always at GitLab, we strive to make meeting attendance optional.
Keeping others informed
In addition to keeping yourself informed, team members are encouraged to keep others informed as well. Secure groups have adopted a practice of including the following topics as standing agenda items in their weekly meetings, with example
topics for each bullet point.
Current status
Work recently achieved against top priorities for that milestone.
Pre-recorded demos are appreciated and encouraged as part of these updates.
Newly discovered scope or dependencies.
Risks
Issues which are blocked or slowed, impacting whether they can be delivered in the desired timeframe.
Help wanted
Issues or topics on which the team or individuals on the team are getting stuck and could use some help.
Praise
Anyone doing a great job and you want to give them kudos?
Any bit of work which has been delivered that’s exceptional?
Engineering Managers are responsible for populating this section of weekly group meetings, though everyone can contribute. In addition to helping the group keep itself informed about what’s happening each week, the SEM for Secure will collect
this information weekly and broadcast a curated list to the section.
Technical onboarding
New hires should go through these steps and read the corresponding documentation when onboarding in the Secure Team.
Every new hire will have an assigned onboarding issue that will guide them through the whole process.
See Issue Refinement to learn how we evaluate complexity, level of effort, our implementation plan and assign issue weights.
Shared pool of Frontend work
From time to time, the demand for frontend engineering tasks does not match the capacity available in Secure groups. If a group could use frontend support, they are encouraged to add the
~secure::frontend backlog label to issues which are ready to be picked up. Additionally,
Engineering Managers are encouraged to look at this queue of work during milestone planning if there is frontend capacity available.
Coding standards and style guidelines
The Secure Team follows the coding standards and style guidelines outlined in the company-wide Contributor and Development Docs, however, please consult the following guidelines which are specific to the Secure Team:
Some components of the architecture that support Secure features are shared between multiple groups like the common Go library,
the Security Report Schemas, the rails parsers, etc.
Modifying these shared pieces might impact other groups so we should rely as much as possible on approval rules to ensure
such changes are reviewed by the relevant teams before being merged.
Impactless two-way door changes could skip the approval process, please use sound judgement and common sense in such situations.
The author of changes should announce broadly the changes made on these components to raise awareness (weekly meeting agenda, slack channel).
Development of new analyzers
For a complete guide about developing a new analyzer please refer to our user documentation
Technical Documentation
As our product evolves, the engineering teams are researching ways to achieve new functionality and improve our architecture.
The Secure sub-department conducts retrospectives at the group level that follow our engineering workflow.
Each group’s DRI is responsible to prepare and schedule the retrospective sync sessions and the async retrospective issues can be found in the corresponding project.
After all groups have completed their retrospective, we conduct a Section Retrospective.
NB: we use to have a sub-department wide retrospective whose issues are still accessible in the deprecated project.
Analytics
The Secure group reviews analytics to help understand customers and their usage of the tools. This data helps drive product and technical decisions. The following links show usage of Secure functionality.
We also track our backlog of issues, including past due security and infradev issues, and total open SUS-impacting issues and bugs.
Merged Merge Request Types
MR Type labels help us report what we’re working on to industry analysts in a way that’s consistent across the engineering department. The dashboard below shows the trend of MR Types over time and a list of merged MRs.
The API Security team is a standalone team which is part of the Dynamic Analysis group at GitLab. It is charged with developing solutions which perform Fuzzing.
Vulnerability Research sits at the crossroads between the Application Security Testing stage itself, and customers of the stage. We provide enhancements to our products or processes to ensure our products and services are more effective for GitLab’s customers.
We strive to enhance our customer experience with regards to providing correct and accurate results from our services.
The Dynamic Analysis group at GitLab is charged with developing solutions which perform Dynamic Analysis Software Testing (DAST) and Fuzzing. Our work is a mix of open and closed source code.
Mission
To support the success of GitLab by developing highly usable, hiqh quality tools for customers to build more secure software. The Dynamic Analysis group at GitLab is charged with developing solutions which perform API Security Testing, Dynamic Analysis Software Testing (DAST) and Fuzzing.
Engineering refinement is the most important step to ensure an issue is ready to move into development and that the issue will match
everyone’s expectations when the work is delivered.
The goal of the refinement process is to
Identify and resolve outstanding questions or discussions.
Raise any questions, concerns or alternative approaches.
Outline an implementation plan.
Ensure issue is ready to be worked on.
Identify code boundaries, for example, does the issue change code maintained by another team.
Notify other teams if the issue is relevant to them in some way.
Assign a weight to the issue.
The refinement process can break down the issue into technical subtasks by following the sub-issue convention but we should avoid redefining the scope of an implementation issue as this should have already been done during the Planning Breakdown with UX and PM.
We expect and require all contributions to our products to go a merge request with a formal review. As such, we follow the Merge Request workflow and code review guidelines articulated in GitLab’s developer documentation. We would, however, like to highlight a few items from these documents and add a few additional considerations for reviewers and authors.
Additional considerations for Merge Request reviewers
The best way to unblock a peer or community member is to provide feedback in a timely manner. If you are at capacity and cannot facilitate a review in the SLO to which we aspire, please let folks know in the merge request so another reviewer may be found.
Additional considerations for Merge Request authors
Being a globally distributed organization can, and frequently does, add latency to back-and-forth communication between folks. Don’t take it personally if it’s taking longer than you expected to get feedback on your changes.
Secure QA Process
The secure analyzers verify merge requests by running a new commit against downstream test projects for their supported languages/frameworks (i.e. the gemnasium analyzer of Dependency Scanning will trigger tests against php, go, and several other test projects). The verification is done by comparing the generated report output against an expected report committed to the analyzer’s repository. If analyzer behavior has changed, then the pipeline will fail because the contents of the expected and generated reports will no longer match.
Feedback(Dismiss, create an issue or a Merge Request)
Overview
The architecture supporting the Secure features is split into two main parts.
flowchart LR
subgraph G1[Scanning]
Scanner
Analyzer
CI[CI Jobs]
end
subgraph G2[Processing, visualization, and management]
Parsers
Database
Views
Interactions
end
G1 --Report Artifact--> G2
Scanning
The scanning part is responsible for finding vulnerabilities in given resources and exporting results.
The scans are executed in CI jobs via several small projects called Analyzers which can be found in our Analyzers sub-group.
The Analyzers are small wrappers around in-house or external security tools called Scanners to integrate them into GitLab.
The Analyzers are mainly written in Go and rely on our Common Go library.
The Static Analysis group is largely aligned with GitLab’s Product Development Flow,
however there are some notable differences in how we seek to deliver software. The engineering team
predominantly concerns itself with the delivery of software, which is the portion of the workflow
states where we deviate the most. What follows is how we manage the handoff from product management
to engineering to deliver software.
When you visit any website, it may store or retrieve information on your browser, mostly in the form of cookies. This information might be about you, your preferences or your device and is mostly used to make the site work as you expect it to. The information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies. Click on the different category headings to find out more and change our default settings. However, blocking some types of cookies may impact your experience of the site and the services we are able to offer.
Cookie Policy
User ID: b19ba099-b74d-47b0-9b66-8e5120d76bb2
This User ID will be used as a unique identifier while storing and accessing your preferences for future.
Timestamp: --
Strictly Necessary Cookies
Always Active
These cookies are necessary for the website to function and cannot be switched off in our systems. They are usually only set in response to actions made by you which amount to a request for services, such as setting your privacy preferences, enabling you to securely log into the site, filling in forms, or using the customer checkout. GitLab processes any personal data collected through these cookies on the basis of our legitimate interest.
Functionality Cookies
These cookies enable helpful but non-essential website functions that improve your website experience. By recognizing you when you return to our website, they may, for example, allow us to personalize our content for you or remember your preferences. If you do not allow these cookies then some or all of these services may not function properly. GitLab processes any personal data collected through these cookies on the basis of your consent
Performance and Analytics Cookies
These cookies allow us and our third-party service providers to recognize and count the number of visitors on our websites and to see how visitors move around our websites when they are using it. This helps us improve our products and ensures that users can easily find what they need on our websites. These cookies usually generate aggregate statistics that are not associated with an individual. To the extent any personal data is collected through these cookies, GitLab processes that data on the basis of your consent.
Targeting and Advertising Cookies
These cookies enable different advertising related functions. They may allow us to record information about your visit to our websites, such as pages visited, links followed, and videos viewed so we can make our websites and the advertising displayed on it more relevant to your interests. They may be set through our website by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant advertisements on other websites. GitLab processes any personal data collected through these cookies on the basis of your consent.