Product Security Engineering

The Product Security Engineering team’s mission is to create proactive and preventative controls which will scale with the organization and result in improved product security.

Product Security Engineering Mission

As part of the Product Security department, and sibling to the Application Security sub-department, our mission is to:

  • Enhance security along the software development lifecycle by creating “paved roads”
  • Contribute product-first code that enhances the security of GitLab’s software assets
  • Reduce the manual burden of the Application Security team by building automation and product improvements

What we do

Product Security Engineering will take potential work from several areas:

  • Existing security enhancement issues in GitLab
  • Automation requirements from AppSec
  • Previously uncaptured efforts identified by Product Security Engineering as being high impact security improvement work
  • Product needs identified by other teams (ex SIRT, Trust & Safety, Security leadership)
  • The Key risks/areas (“Security Focus” areas)

Contacting us

To reach the Product Security Engineering team, team members can:

Workflow

Backlog building / requests

The ~ProdSecEng Candidate label identifies a particular issue as potentially being work that the Product Security Engineering team can take on. This label can be added by anyone, the issue will be reviewed and potentially pulled into the backlog by the Product Security Engineering team members.

Depending on the nature of the work it is added either to:

When we take on work:

  1. Add the ~"team::Product Security Engineering" label
  2. For internal issues: ensure it meets the criteria defined in Automation Request template for automation work, or
  3. For product issues:
    1. Identify the relevant PM/EM are based on group:: labels. If there are no group:: labels, make a best effort to figure out what group it would be relevant to.
    2. Ping the group’s PM/EMs. Say that we’re working on this issue, do your best to align with any existing efforts, and highlight that after release it will belong to their team (similar to a community contribution).
    3. If we can’t figure it out an owner, don’t ping anybody.

Removing work items from the backlog

If at any point during the refinement process it is determined that something is not work the Product Security Engineering team will take on, a member of the Product Security Engineering team will:

  • Remove the ~"team::Product Security Engineering" or ~ProdSecEng Candidate label
  • Make a comment explaining the reasoning as to why the Product Security Engineering team has decided not to commit to this work
  • Make a best-effort to @-mention the appropriate Engineering Manager, Product Manager, or teams and apply the relevant group labels
  • Consider applying the ~Seeking community contributions label, if the issue is public and a potential fit for a community contribution

Refinement, Design, and Build

Like Single Engineer groups, each Product Security Engineer will “encompass all of product development (product management, engineering, design, and quality) at the smallest scale. They are free to learn from, and collaborate with, those larger departments at GitLab but not at the expense of slowing down unnecessarily”.

  • Our build boards are organized into workflow columns
  • We use the labels, outcomes, and activities described Product Development Flow, but have the flexibility to skip the process where it’s not needed
  • All Product Security Engineering team members can contribute to validation, refinement, and solution design
  • All Product Security Engineering team members can contribute to the prioritization, but the Security Engineering Manager is DRI
  • New projects should follow the “Creating a new project” engineering guidance
  • Unless the effort is AppSec automation, the workflow ends by handing over the feature to a Product team

Refinement Cadence

Product Security Engineering team members are responsible for performing refinement on issues that we may potentially take on. We aim to do refinement on a regular basis, which helps to ensure our Milestone Planning process is smooth and efficient.

It is expected that Product Security Engineering team members will do refinement tasks throughout the milestone. It is up to the individual to decide how they want to accomplish this, some ideas include:

  • Setting aside a specific amount of time per week on the calendar to perform refinement
  • Refining issues in-between major context switches, for example after submitting a merge request for review but before picking up the next piece of work

Step-by-step refinement process

Below is a step-by-step process for team members to walk through when refining backlog issues. We try our best to adhere to existing GitLab development team standards, so that the work can be picked up by anyone.

  1. Choose an issue to refine
    1. Unrefined issues are labeled ~workflow::validation backlog (or perhaps have no ~workflow:: label)
    2. You may also consider refining an issue labeled ~ProdSecEng Candidate
    3. If possible, timebox refinement to at most 1 hour per issue
  2. Get an understanding of what the issue is trying to accomplish
    1. You may need to ask questions of the person who created the issue or the relevant teams
    2. Ensure there is a clear definition of done for this particular set of work
    3. Consider breaking the issue down into separate pieces or, if needed, making an epic
  3. Add additional details to the appropriate sections such that someone can easily understand the goals and requirements
  4. Investigate what an ideal solution might look like and add potential solution information to that issue
    1. Consider timeboxing this effort
    2. If needed, consider applying the ~workflow::solution validation label and engaging with the relevant product, engineering, or security teams to determine if the proposed solution addresses the requirements
  5. Add a weight based on how much effort this will take to accomplish
  6. Add the ~workflow::ready for development label to indicate that the issue has been refined

Weights

We use a lightweight system of issue weighting with the knowledge that things take longer than you think. It’s OK if an issue takes longer than the weight indicates. The weights are intended to be used in aggregate, and what takes one person a day might take another person a week, depending on their level of background knowledge about the issue. That’s explicitly OK and expected.

These weights we use are:

Weight Meaning
1 Trivial, does not need any testing
2 Small, needs some testing but nothing involved
3 Medium, will take some time and collaboration
4 Substantial, will take significant time and collaboration to finish
5 Large, will take a major portion of the milestone to finish

Anything larger than 5 should be broken down if possible.

Choosing what to work on

Product Security Engineering should always have at least one AppSec-related issue in flight. This rule’s intention is to make sure we achieve our mission of reducing AppSec’s manual work burden.

If a Product Security Engineer completes all their work in a Milestone, they should take an item from the top of the backlog and assign themselves to it. If they need to stop working on something they should unassign themselves, @ mention the team, and apply the correct workflow label (e.g. ~workflow::blocked).

Merge Request Reviews

When contributing to a project that is owned or maintained by another team or an official GitLab asset, we follow that project’s established review conventions, rules, and requirements.

When contributing to a project owned and primarily maintained by Product Security Engineering:

  • We default to asking for other Product Security Engineering team members to review our merge requests
    • We strive to review eachother’s contributions in order to encourage collaboration, facilitate knowledge sharing, and reduce silos
    • We must acknowledge that we are a small team and that sometimes a thorough review isn’t going to happen in a timely manner, impacting velocity
    • We evaluate the tradeoff between knowledge sharing and velocity on a case-by-case basis, with each team member empowered to make decisions on foregoing a review
  • We can skip a formal review if something is blocking, time-sensitive, and/or resolving an urgent high-impact need
  • We try to pick up issues in tooling other team members have written

Project namespaces

The Product Security Engineering team defaults to using the namespaces of the stakeholder that we’re performing the work for:

Milestone Planning

The Product Security Engineering team plans its work on a cadence based around GitLab Product Milestones. This section describes the planning process that we use to determine what work will be completed for each Milestone.

Milestone Planning Issue

For each Milestone, a [Milestone Planning issue](https://gitlab.com/gitlab-com/gl-security/product-security-engineering/product-security-engineering-team/-/issues/?label_name[]=Milestone Planning) is created in the Product Security Engineering team repository. The purpose of this issue is to:

  • Identify potential work to perform
  • Identify refinement gaps and determine how to address them
  • Determine what work we’re committing to for the upcoming Milestone
  • Set and communicate priority for the work we’ve decided to take on

This issue is the single source of truth for all planning related discussions and decisions related to the upcoming Milestone.

Milestone Planning Process

  1. An issue will be created using the Milestone Planning issue template
    1. The issue will be created immediately after planning is finished for the current Milestone
    2. Throughout the current Milestone, Product Security Engineering team members can add work to the upcoming Milestone’s Parking Lot
  2. The Product Security Engineering manager will be responsible for completing the checklist items in the Planning Checklist section of the Milestone Planning issue
  3. Product Security Engineering team members will add any work being carried over from the previous Milestone into the Milestone Work table
  4. The Product Security Engineering team will add potential work items to the Parking Lot section, with a brief explanation of why it would be good to include in the Milestone
    1. Team members can add discussion threads to discuss potential work to pull into the Milestone
    2. Both individual team members and the Product Security Engineering manager can add items to this list
  5. The Product Security Engineering team will work together to add new items to the Milestone Work table
    1. Each item being added must be refined before it can be formally committed to
    2. The team member likely to take on the work should review and agree with the Weight, if it wasn’t them who refined the issue.
    3. Once we have refined and committed to the work, the relevant issue needs to be updated with the Milestone and Assignee(s)
  6. The Milestone Planning issue should be finalized at least 3 days before the Milestone Start Date
    1. The Product Security Manager will use threads in the Milestone Planning issue to work with each Product Security Engineering team member to finalize their workload
    2. Once finalized, the Planning Issue should be closed and an issue for the next Milestone should be opened

Milestone Planning responsibilities

Product Security Engineering team members are responsible for:

  • Evaluating and communicating their capacity for the Milestone (based on PTO and other factors)
  • Adding work that is being carried over into the Milestone Work table
  • Adding potential work items to the Parking Lot and being involved in discussions around what work we should pull into the Milestone
  • Collaborating with the Product Security Engineering manager to finalize the set of work being committed to for the Milestone

The Product Security Engineering manager is responsible for:

  • Creating, updating, and maintaining the Milestone Planning issue
  • Collaborating with Product Security Engineering team members to discuss potential work, identify refinement gaps, and assemble the Milestone Work table
  • Coordinating the finalization of the Milestone Planning issue