Create:Source Code BE Team

The Create:Source Code BE team focuses on GitLab’s Source Code Management (SCM) tools and is responsible for all backend aspects of the Source Code group’s product categories in the Create stage of the DevOps lifecycle. For information on our product direction, visit the Category Direction - Source Code Management page.

We interface with the Gitaly and Code Review teams, and work closely with the Create:Source Code Frontend team. The features we work with are listed on the Features by Group Page and technical documentation is available on the Create: Source Code Backend page.

Team members

The following people are permanent members of the Create:Source Code BE Team:

Name	Role
Devi Poosarla	Engineering Manager, Create:Source Code Backend
Emma Park	Backend Engineer, Create:Source Code Backend
Fred Reinink	Backend Engineer, Create:Source Code Backend
Gavin Hinfey	Backend Engineer, Create:Source Code
Hunter Stewart	Senior Backend Engineer,Create:Source Code Backend
Jerry Seto	Senior Backend Engineer, Create:Source Code
Javiera Tapia	Backend Engineer, Create:Source Code
Joe Woodward	Senior Backend Engineer, Create:Source Code
Pranav Jain	Associate Backend Engineer, Create:Source Code
Olaoluwa Oluro	Senior Backend Engineer, Create:Source Code
Vasilii Iakliushin	Staff Backend Engineer, Create:Source Code, Systems:Gitaly API

Stable counterparts

The following people of other functional teams are our stable counterparts:

Name	Role
Brendan Lynch	Technical Writer, Create (Source Code)
Costel Maxim	Senior Security Engineer, Application Security, Plan (Project Management, Product Planning, Certify), Create:Source Code, Growth, Fulfillment:Purchase, Fulfillment:Provision, Fulfillment:Utilization, Systems:Gitaly
Jay McCure	Senior Software Engineer in Test, Dev:Create
Jacques Erasmus	Senior Frontend Engineer, Create:Source Code
Marie-Christine Babin	Senior Product Manager, Create:Source Code
Chaoyue Zhao	Frontend Engineer, Create:Source Code
Paulina Sedlak-Jakubowska	Senior Frontend Engineer, Create:Source Code

Sisense and KPIs

To help us stay on track with Development KPIs, we use a metrics dashboard. This dashboard includes security MRs from production, but doesn’t include security MRs from dev.gitlab.org. For team-specific data and metrics, ensure you filter by our team.

Workflow

We use the standard GitLab engineering workflow. To create an issue for the Create:Source Code BE team, add these labels:

• ~backend
• ~“devops::create”
• ~“Category:Source Code Management”
• ~“group::source code”

For more urgent items, use the #g_create_source_code Slack channel.

Take a look at the features we support per category here.

Working with the Product Team

Weekly calls between the Product Manager and Engineering Managers (frontend and backend) are listed in the “Source Code Group” calendar. Everyone is welcome to join and these calls are used to discuss any roadblocks, concerns, status updates, deliverables, or other thoughts that impact the group.

Issue refinement

1. Once we have validated the problem, product, UX, and engineering will collaborate to propose a solution and decide on what’s technically feasible. The proposed solution will be shared with users to validate it solves the problem.
   1. Issues that require design work are marked with UX and workflow::ready for design.
   2. Once designs are ready and the proposed solution is viable then the label workflow::planning breakdown will be applied.
2. Once we have confirmed the proposed solution is viable, we will move to break it down as much as possible. When issues are ready for this stage, PM will mark issues with workflow::refinement label to signal next step.
3. EM will work with engineers to decide distribution of work and assign issues for breakdown.
4. Engineers or EM will evaluate the issue, work with PM, UX, and other engineering counterparts where necessary to address questions and concerns.
5. If the planned implementation of the issue can be further broken down, the engineer/EM will work with the PM to reduce scope and create new issues until this is the case (either PM or engineer/EM can create new work items).
6. When the planned implementation of the issue is in its smallest form, the engineer/EM will provide a weight. EM or PM will mark as workflow::needs issue review.

Note: if an issue receives a weight > 3 after this process, it may indicate the IC may not have a full idea of what is needed and further research is needed.

Convention over configuration

As stated in our direction, we must place special emphasis on our convention over configuration principle. As the feature set within Create:Source Code grows, it may feel natural to solve problems with configuration. To ensure this is not the case, we must intentionally challenge MVC and new feature issues to check for this. Let’s consider the following steps for best results:

1. Once issues have been labeled as workflow::needs issue review PM will share the proposal with either a peer or their manager as well as engineering (EM or IC) and product designer.

2. Peers in product and engineering who review the issue should look for opportunities to eliminate configuration where possible. If opportunities are identified, the issue is moved back to workflow::refinement.

3. If PM and peers are satisfied with the proposal and it follows our convention over configuration principle as much as possible, those who reviewed the issue indicate their agreement with the proposal (with either 👍 or a comment in the issue). Finally, PM or EM will label issue workflow:: ready for development.

Collaborating with counterparts

You are encouraged to work as closely as needed with stable counterparts outside of the PM. We specifically include quality engineering and application security counterparts prior to a release kickoff and as-needed during code reviews or issue concerns.

Quality engineering is included in our workflow via the Quad Planning Process.

Application Security will be involved in our workflow at the same time that kickoff emails are sent to the team, so that they are able to review the upcoming milestone work, and notate any concerns or potential risks that we should be aware of.

Communication

As a team we strive to be responsive and accommodating when we communicate. When consuming information, we use the following emoji to definition table as a shorthand glossary:

Emoji	Definition
👀	I’m looking at this now
🤔	I need to think about this before responding
📍	I don’t have time, will come back later
🤷🏼	I don’t know
👌	I understand
👍	Agree (and often won’t give additional comment)
✅	Task is complete
⏭	I’ve seen this but I don’t think I’m the best person for the job. Ping me if no one else responds and you need help

Merge Request reviews

Conventional Comments

When giving Merge Request review feedback, we aim to provide additional context by utilizing the conventional: comments standard.

Triage process

The weekly Triage Report is generated automatically by the GitLab bot and this report is reviewed by the EM. Here is an example of a previous report.

The Triage Report can be quite long, and it important to deal with it efficiently. An effective way to approach it is:

• Open every issue in a separate browser tab and use “edit issue” to mark then as checked once review, then close the tab.
• Verify if the issue belongs to ~“group::source code” and change group label if needed. The Features by Group page is a good starting point for this assessment.
• Apply ~frontend if it is a frontend issue.
• Perform a brief search to assess if is a duplicate, close with a ~Duplicate label if this is the case.
• Is it a ~“support request” ? Does it ~“needs investigation” ? Apply labels if so.
• Apply ~regression label if it is one, consider bumping severity numbers if recent.
• Apply ~“severity::4”, ~“priority::4”, %Backlog if a smaller issue with no significant impact.
• If an uncontroversial problem with a clear solution, consider applying ~“Seeking community contributions”
• If also an easier issue which might interest a newer community contributor, consider applying ~“quick win”.
• Apply ~“priority::3” ~“severity::3” if a bug with a workaround.
• Anything causing data loss, severe performance impact or security apply a ~“severity::1” and ~“priority::1” or ~“priority::2” and assign to a team member.
• Unassign yourself from the Triage Report

Engineering cycle

The engineering cycle is centered around the GitLab Release Date every month. This is the only fixed date in the month, and the table below indicates how the other dates can be determined in a given month.

Iteration documents

These documents comprise everything that is documented during the release planning and execution.

Issue boards

Create Source Code BE planning takes inputs from the following sources:

• Performance Board
• Infradev Board
• Application Limits Board
• Security Board
• Missed SLO Board
• Bugs
• New features

Create Source Code UX planning takes inputs from the following sources:

• SCM UX Planning Board
• SCM UX Build Board

Planning issue

Each month a planning issue is created by one of the EMs, using automated tools based on the Source Code issue template.

Planning board

The Planning Board is created for each release by the PM, and is a curated list of issues by category. The EM requests engineers to allocate weights to all issues on this board via the Needs weight issue

Capacity planning spreadsheet

The EM maintains a Google Sheet for calculating team capacity, and the same Spreadsheet is also used to perform the process of assigning issues to the release based on weight and priority.

Build board

The EM selects issues from the Planning Board based on:

• slipped issues
• weight
• priority
• PM preference

The EM then applies the ~Deliverable label to each issue in the Release and assigns then to an engineer. The issues are tracked through the release via the Build Board.

Candidate issues

Urgent issues are tentatively assigned to a release to ensure other teams have visibility.

At this point the issues are Candidate issues, and the milestone does not confirm that they will be definitely scheduled. Issues move from Candidate status to confirmed during the Issue selection process.

Key dates

Date	Event
The Monday of the week the milestone ends	PM creates planning board and pings EMs in the Planning Issue for review & weighting. EMs calculate capacity, add to Planning Issue. PM submits RPIs for reviews.
Monday to Friday of the week the milestone ends	EMs & ICs add weights to issues in the planning board
The Friday the milestone ends	EMs add ~Deliverable labels to issues so that they appear on the Build board as a draft Release Post: EMs, PMs, and PDs contribute to MRs for Usability, Performance Improvements, and Bug Fixes
The Friday the milestone ends	EMs adjust ~Deliverable labels for slippage and make final assignments PMs review final plan for milestone on Build board EMs merge RPI MRs for features that have been merged.
The third Thursday of the month	Release

Weighting issues

Based on the issues on the Planning Board, the EM will create a Needs Weight issue to request an estimation of work by the engineers. In general no more than 4 issues should be assigned to an engineer for weighting.

1. Give issue a weight if there’s none yet or update if the existing weight isn’t appropriate anymore. Optionally leave a comment describing why a certain weight is given.
2. It’s strongly encouraged to spend no more than 1 hour per issue. Give it your best guess and move on if you run out of time.
3. Label the issue as ~“workflow::ready for development” if you feel we can make progress on it in the next milestone.

If you would like to be assigned to work on this issue in the upcoming release, add a comment and ping the EM.

Weight categories

The weights we use are:

Weight	Description
1: Trivial	The problem is very well understood, no extra investigation is required, the exact solution is already known and just needs to be implemented, no surprises are expected, and no coordination with other teams or people is required. Examples are documentation updates, simple regressions, and other bugs that have already been investigated and discussed and can be fixed with a few lines of code, or technical debt that we know exactly how to address, but just haven’t found time for yet.
2: Small	The problem is well understood and a solution is outlined, but a little bit of extra investigation will probably still be required to realize the solution. Few surprises are expected, if any, and no coordination with other teams or people is required. Examples are simple features, like a new API endpoint to expose existing data or functionality, or regular bugs or performance issues where some investigation has already taken place.
3: Medium	Features that are well understood and relatively straightforward. A solution will be outlined, and most edge cases will be considered, but some extra investigation will be required to realize the solution. Some surprises are expected, and coordination with other teams or people may be required. Bugs that are relatively poorly understood and may not yet have a suggested solution. Significant investigation will definitely be required, but the expectation is that once the problem is found, a solution should be relatively straightforward. Examples are regular features, potentially with a backend and frontend component, or most bugs or performance issues.
4: Large	Features that are well understood, but known to be hard. A solution will be outlined, and major edge cases will be considered, but extra investigation will definitely be required to realize the solution. Many surprises are expected, and coordination with other teams or people is likely required. Bugs that are very poorly understood, and will not have a suggested solution. Significant investigation will be required, and once the problem is found, a solution may not be straightforward. Examples are large features with a backend and frontend component, or bugs or performance issues that have seen some initial investigation but have not yet been reproduced or otherwise “figured out”.
5: Unknown	A feature that is weight 5 will not be scheduled and instead should be broken down or a spike scheduled

A weight of 5 generally indicates the problem is not clear or a solution should be instead converted to an Epic with sub-issues.

If the issue should be broken down

If the problem is well-defined but too large (weight 5 or greater), either:

• Promote the issue to an Epic and break the work into sub-issues. Weight the individual issues if possible.
• Ping the EM and PM and outline the reason the issue needed to be promoted to an Epic.

If the issue SSOT is not clear

• Don’t assign a weight, instead add a comment indicating what needs clarification and ping the EM and PM.

If the issue needs a spike

• Don’t assign a weight, instead add a comment about the need for a spike (and possibly what would be investigated) and ping the EM or PM.
• Spikes are scheduled with a weight of 2.
• Spikes are scheduled with a weight of 2 (timeboxed).

See the spike issues section for more details about these issues.

Security issues

Security issues are typically weighted one level higher than they would normally appear from the table above. This is to account for additional work and backports in the patch release process.

Planning issue review

The Source Code stable counterparts (BE, FE, PM, UX) meet and propose issues to be worked on in the upcoming release. Using the Mural visual collaboration tool, candidate issues are voted on by the group.

Capacity planning

Capacity planning is a collaborative effort involving all Source Code team members and stable counterparts from Frontend, UX and Product. An initial list of issues is tracked in the Source Code Group Planning issue example for each month.

Team availability

Approximately 5-10 business days before the start of a new release, the EM will begin determining how “available” the team will be. Some of the things that will be taken into account when determining availability are:

• Upcoming training
• Upcoming time off / holidays
• Upcoming on-call slots
• Potential time spent on another teams deliverables

Availability is a percentage calculated by (work days available / work days in release) * 100.

All individual contributors start with a “weight budget” of 10, meaning they are capable (based on historical data) of completing a maximum number of issues worth 10 weight points total (IE: 2 issues which are weighted at 5 and 5, or 10 issues weighted at 1 each, etc.) Then, based on their availability percentage, weight budgets are reduced individually. For example, if you are 80% available, your weight budget becomes 8.

Product will prioritize issues based on the teams total weight budget. Our planning rotation will help assign weights to issues that product intends on prioritizing, to help gauge the amount of work prioritized versus the amount we can handle prior to a kickoff.

Source Code issue pipeline

The Source Code issue pipeline is broad, and the PM and EM work together throughout the planning process and the final list is chosen during the Issue Selection meeting. The issue pipeline includes:

• Features
• Bugs
• Security issues
• Infradev board issues
• Performance board issues
• Application limit board issues

Issue selection

On or around the 16th, the PM and EM meet once more to finalize the list of issues in the release. The issue board for that release is then updated, and any issues with an candidate milestone that are not selected will be moved to Backlog or added as a candidate for a future release.

Issues scheduled for the release are then marked ~“workflow::ready for development”.

Issue assignments

Once availability has been determined, weights have been assigned, and the PM/EM finalize a list of prioritized issues for the upcoming release, kickoff emails will be sent. The intent of this email is to notify you of the work we intend to assign for the upcoming release. This email will be sent before the release begins. The kickoff email will include:

• Your availability, weight budget, and how it was calculated
• A list of the issues you will most probably be assigned as an individual
• A reasoning behind why you have been assigned more than your weight budget, if applicable
• A list of the issues the team is working on that are deemed “note-worthy,” in case you’d like to offer help on those issues as time allows

You will begin to collect follow-up issues when you’ve worked on something in a release but have tasks leftover, such as technical debt, feature flag rollouts or removals, or non-blocking work for the issue. For these, you can address them in at least 2 ways:

• Add an appropriate future milestone to the follow-up issue(s) with a weight and good description on the importance of working this issue
• Add the issue(s) to the relevant planning issue

You should generally take on follow-up work that is part of our definition of done, preferably in the same milestone as the original work, or the one immediately following. If this represents a substantial amount of work, bring it to your manager’s attention, as it may affect scheduling decisions.

If there are many follow-up issues, consider creating an epic.

Spike issues

The primary purpose of a spike issue is to understand the feasibility of the request and perform a study, research, analysis, or PoC rather than producing a shippable product. The outcome of the spike is to finalize the design of a solution and further plan and break down the work into multiple issues that are weighted depending on their complexity.

To recognize and improve the readability of these issues when they become deliverables, we should:

• Use the label ~"spike" and the [SPIKE] identifier at the beginning of the spike issue title.
• Link the created issues (outcome of the spike) to the spike issue. The labels are copied from the original issue.
• Close the spike issue with a summary of the next steps we will perform to achieve the goal of the spike.

When dealing with a feature request that needs further investigation, we can open a spike issue that assesses the research and is linked to the original issue.

Exceptions

There are some exceptions to the rule. For example, when the spike issue is highly discussed and visible to customers, closing the original issue if scheduled (using the ~"Deliverable" label) as a spike will send the wrong signals to customers. That is when we keep the issue open and remove the ~"spike" label and the [SPIKE] identifier from the spike issue title. For more details see this example.

Double-assign for overly-complex or time-sensitive issues

As discussed in a previous retrospective, in addition to breaking down issues, we should assign two engineers to each task instead of just one for overly-complex or time-sensitive issues. This co-ownership will help parallelize efforts in multiple-MR tasks, speed up immediate code reviews, and ultimately lead to faster delivery of results.

Backend and Frontend issues

Many issues require work on both the backend and frontend, but the weight of that work may not be the same. Since an issue can only have a single weight set on it, we use scoped labels instead when this is the case: ~backend-weight::<number> and ~frontend-weight::<number>.

Workflow labels

The easiest way for engineering managers, product managers, and other stakeholders to get a high-level overview of the status of all issues in the current milestone, or all issues assigned to specific person, is through the Development issue board, which has columns for each of the workflow labels described on Engineering Workflow handbook page under Updating Issues Throughout Development.

As owners of the issues assigned to them, engineers are expected to keep the workflow labels on their issues up to date, either by manually assigning the new label, or by dragging the issue from one column on the board to the next.

Retrospectives

We have 1 regularly scheduled “Per Milestone” retrospective, and can have ad-hoc “Per Project” retrospectives.

Per Milestone

The Create: Source Code group conducts monthly retrospectives in GitLab issues. These include

the backend team, plus any people from frontend, UX, and PM who have worked with that team during the release being retrospected.

These are confidential during the initial discussion, then made public in time for each month’s GitLab retrospective. For more information, see team retrospectives.

Retrospective Coordinator

For each retrospective we assign a retrospective coordinator whose responsibilities include:

• Invite team members to provide asynchronous feedback
• Provide updates on action points from previous retrospective
• Capture action points for the current retrospective
• Ensure a retrospective coordinator is assigned for the next retrospective

Per Project

If a particular issue, feature, or other sort of project turns into a particularly useful learning experience, we may hold a synchronous or asynchronous retrospective to learn from it. If you feel like something you’re working on deserves a retrospective:

1. Create an issue explaining why you want to have a retrospective and indicate whether this should be synchronous or asynchronous
2. Include your EM and anyone else who should be involved (PM, counterparts, etc)
3. Coordinate a synchronous meeting if applicable

All feedback from the retrospective should ultimately end up in the issue for reference purposes.

Deep dives

The groups in the Create stage organize regular Deep Dive sessions to share our domain specific knowledge with others in the stage, the organization, and the wider community. All existing Deep Dives can be found on GitLab Unfiltered with slides and recordings in the descriptions. For Create specific Deep Dives, there is a different playlist. To find more information on upcoming sessions, or to propose a new topic, please see the epic.

Career development

Career development conversations in the Create:Source CodeBE team are centered around a Career Development Sheet that is based on the Engineering Career Matrix for Individual Contributors. The sheet lists the expected current level behaviors on the left, the next level behaviors on the right, and uses colored columns in between to visually represent the extent to which the individual has shown to have grown from the current level to the next. Columns to the right of a next level behavior are used to collect specific examples of that behavior, which serve as evidence of the individual’s growth.

Both the matrix and the sheet are Works in Progress; the development of the career matrix is tracked in an epic, and as the matrix evolves, the sheet will be updated to match.

Performance monitoring

The Create:Source Code BE team is responsible for keeping some API endpoints and controller actions performant (e.g. below our target speed index).

Here are some Kibana visualizations that give a quick overview on how they perform:

• Create::Source Code: Controller Actions
• Create::Source Code: Endpoints

These tables are filtered by the endpoints and controller actions that the group handles and sorted by P90 (slowest first) for the last 7 days by default.