Engineering

Complexity at Scale

As GitLab grows, through the introduction of new features and improvements on existing ones, so does its complexity. This effect is compounded by the care and feeding of a single codebase that supports the wide variety of environments in which it runs, from small self-managed instances to large installations such as GitLab.com. The company itself adds to this complexity from an organizational perspective: hundreds employees worldwide contribute in one way or another to both the product and the company, using GitLab.com on a daily basis to do their job. Teams members in Engineering are directly responsible for the codebase and its operation, for the infrastructure powering GitLab.com, and for the support of customers running self-managed instances. Likewise, team members in the Product organization chart the future of the product.

Vision

Our goal is not merely to launch features, but to ensure they land successfully and provide real value to our customers. We strive to develop a best-in-class product that exceeds expectations across all user groups by meeting high-quality standards while ensuring reliability and maintaining an ease of operation and scalability to meet diverse customer needs. All team members should remain mindful of our target customers and the multiple platforms we support in everything we do.

Overview

The Cross-Functional Prioritization framework exists to give everyone a voice within the product development quad (PM, Development, Quality, and UX). By doing this, we are able to achieve and maintain an optimal balance of new features, security fixes, availability work, performance improvements, bug fixes, technical debt, etc. while providing transparency into prioritization and work status to internal and external stakeholders so they can advocate for their work items. Through this framework, team members will be able to drive conversations about what’s best for their quad and ensure there is alignment within each milestone.

The CTO Leadership Team is composed of the CTO’s direct reports and the Office of the CTO (OCTO).

Office of the CTO (OCTO)

The OCTO is composed of the CTO, the Engineering EBAs, the CTO’s People Business Partners, and the CTO’s Director of Strategy and Operations. This team works to amplify the CTO’s reach, vision, and mission. They work together to deliver programs and results across the entire Engineering Division.

Overview and terminology

This page describes the deployment and release approach used to deliver changes to users. The overall process consists of two significant parts:

1. Monthly self-managed release: GitLab version (XX.YY.0) published every month. From this monthly release, planned patches are scheduled twice a month and unplanned critical patches are created as needed.
2. GitLab.com deployment: A Continous Delivery process to deploy branches created from master branch, on regular intervals.

For more details on the individual processes and how to use them please see the Deployments page for GitLab.com changes and the Releases page for changes for self-managed users.

The Three Components of Career Development

There are three important components of developing one’s career:

Structure

Team members who are (or want to be) on track for promotion should be engaged in a career coaching conversation with their manager. Some basic information about this process can be found in the People Ops handbook. Specific coaching plan templates are listed here to help start the conversation:

• Senior Engineer
• (Others to come)

We want to build these documents around the career matrix for Engineering. Since this career matrix is still being developed, these documents are currently based on the job family requirements.

Communication

GitLab Engineering values clear, concise, transparent, asynchronous, and frequent communication. Here are our most important modes of communication:

As part of a fully-distributed organization such as GitLab, it is important to stay informed about engineering-led initiatives. We employ multimodal communication, which describes the minimum set of communication channels we’ll broadcast to.

The Engineering Divison has a Google Group, engineering@gitlab.com (internal only), that all members of the division should become members as part of the onboarding process. If this is not the case for you, reach out to your manager. As GitLab, the company, primarily communicates via Slack, use this list mainly for Access Control to Google Drive/Docs/Sheets/Slides.

Occasionally, it is useful to set up a demo on a regular cadence to ensure cross-functional iterative alignment. This is helpful for high-impact deliverables that require integration across multiple functional teams. This is in-line with the seventh principle of the Agile Manifesto: “Working Software is the best measure of progress”.

Demo script

For multi-person groups or critical projects, we use a heavier weight grading process:

1. The demo owner identifies the outcome of the demo based on the business criteria. This can be an engineering manager, a product manager or someone who is a business stakeholder of the outcome.
2. The demo owner breaks down the outcome into smaller pieces, aligning with functional areas (tracks) and structured in procedural flow. This will later be captured as demo steps.
   • List each step, however small it might look to expose implicit dependencies.
3. The demo owner identifies a functional team leader as a DRI for each demo track. The DRI for each track is responsible for demoing each track to completion.
4. The demo owner collaborates with functional team leaders to populate the demo steps in a scorecard. Here is the demo scorecard template. To use this template:
   • Copy the template and rename to the initative/deliverable.
   • Clear the scores in the scorecard sheet.
   • Populate the demo tracks and demo steps.
   • Note: Here is an example of a populated demo scorecard.
5. The demo owner identifies a demo grader to hold grading accountability. This can be the demo owner or someone who is familiar with the product domain and customers’ usecase. It is important that the demo grader is someone who can advocate for the success of our end users.

Demo scheduling

1. Once the script is finalized, the demo owner schedules a recurring recorded meeting for the demo with target end date.
2. Demo owner & demo grader must be present in every demo to ensure accoutablility. Assign delegates appropriately for one-off un-avaliability.
3. Create an agenda document where each participant can take notes in, in addition to the scorecard.
4. The audience is the key business stakeholder of the demo deliverables & the product group team (Development, UX, Quality, Product).
5. Meeting should be kept to 30 minutes. The emphasis should be on the product requirements & acceptance criteria.
6. The demo gets kicked off and each demo tracks iterate each week on the progress until completion.
7. Live streaming or uploading to GitLab Unfiltered channel is optional. Please abide by our SAFE guidelines if choosen to do so.

Demo grading

The demo master grades each step during the demo meeting. To make it less subjective, we use a scale that is widely understood and communicated. Our scoring definitions are as follows:

Executive Summary

Key Performance Indicators

Engineering Handbook MR Rate

The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This is measured by Merge Requests that update the handbook contents relate to the Engineering Division overtime.

Overview

Hiring is a cornerstone of success for our engineering organization, contributing to our growth and our ability to drive results for our customers. As such, it’s not just a responsibility but fundamental to every engineer’s contribution to GitLab. It should be deeply ingrained in every engineer’s role at GitLab, regardless of their seniority.

By actively participating in recruitment efforts, engineers help shape their team culture, elevate technical standards, and ensure a continuous influx of diverse perspectives and skillsets. Contributing to hiring efforts allows GitLab to grow responsibly and affects our collective success within Engineering.

Engineering IC Leadership at GitLab: going beyond Senior level

At GitLab, it is expected that everyone is a manager of one. For Individual Contributors (IC) a new type of challenge begins with the Staff Engineer role. Engineering IC Leadership is an alternative career path to Engineering Management.

Just like moving into management, also moving from Senior to Staff changes the day-to-day work and expectations placed on ICs.

Engineering IC Leaders exert technical leverage in their scope of influence. Like any other leadership role, the focus should be on helping others to improve. Their impact multiplies with every person they help grow, and the company gets more value when they’re not investing time in doing things themselves.

How Engineering Management Works at GitLab

At GitLab, we promote two paths for leadership in Engineering. While there is a healthy degree of overlap between these two ideas, it is helpful and efficient for us to specialize training and responsibility for each of:

• Technical leadership, as represented by Staff and higher-level engineers.
• Professional leadership, as represented by Engineering management.

While technical leadership tends to come naturally to software engineers, professional leadership can be more difficult to master.

Mentorship, Coaching and Engineering Programs

Line Managers and Senior Individual Contributors

The PlatoHQ Program has a total of 10 Engineering Managers/Senior IC’s participating. The program exists of both self-learning via an online portal and 1-1 sessions with a mentor.

Senior Leaders in Engineering

The 7CTOs Program is run with 4 Senior leaders in Engineering. The program exists of peer mentoring sessions (forums) and effective network building.

AI Gateway

AI Gateway for GitLab Duo features.

Vision

Scale and develop our diverse, global team to drive results that support our product and customer growth, while maintaining our values and unique way of working.

Mission

GitLab’s unique way of working asynchronously, handbook first, using the product we develop, and with clear focus on our values enables very high productivity. In delivering on growth, we maintain our values and ways of working while developing team members and increasing the diversity of our team. We focus on constantly improving usability and reliability of our product to reach maximum customer satisfaction. Community contributions and customer interactions rely on efficient and effective communication. We are a data-driven, customer experience first, open core organization delivering one secure, reliable, world leading DevOps platform.

A Fast Boot is an event that gathers the members of a team or group in one physical location to work together and bond in order to accelerate the formation of the team or group so that they reach maximum productivity as early as possible.

History of the Fast Boot

• The first Fast Boot took place in December 2018. The 13 members of Monitor Group gathered for 3 days to work and bond in Berlin. You can learn more by reading the original planning issue.
• The second Fast Boot took place in April 2019. The 5 members of Delivery team gathered in Utrecht to bond but also work on finalising auto-deployment process. Planning issue contains the proposal for Fast boot, and the Delivery Fast boot epic contains issues and links to recordings created during the Fast Boot.
• The third Fast Boot took place in Vancouver in September 2019. It included 18 people from Product, Engineering, UX and Data from the Acquisition, Conversion, Expansion and Retention teams. The planning issue contains the proposal for Fast Boot, and outcomes are available in the Growth Fast Boot Page.

Why should you have a Fast Boot?

Right now, the fast boot is intended for new teams or for teams with a majority of new members who need to build their culture of shipping work. If your team fits this description, you can propose holding a Fast Boot to reduce ramp up time and establish and strengthen relationships between team members.

Teams

• Configure
• Create
• Manage
• Monitor
• Plan
• Secure
• Verify and Release

Frontend domain experts

You can find engineers with expertise in various frontend domains on the engineering projects page under the following sections:

• GitLab maintainers with domain expertise
• GitLab reviewers with domain expertise

You can reach out to these experts to get help on:

• discussing and defining the architecture of complex frontend features.
• frontend technical topics like Vue, GraphQL, CSS, testing, tooling, etc.
• proposing changes to the cross-domain frontend architecture via an RFC.
• Questions about the frontend for a product area like design management, merge requests, pipelines, etc.

Frontend group calls

The frontend group has scheduled weekly calls every Tuesday. Since 2021-06-01, these occur at three staggered, time zone friendly times, repeating every three weeks. During these calls, team members are encouraged to share information that may be relevant to share with other members synchronously (Eg. new documentation change, new breaking changes added to master).

Background

As part of the FY25-Q2 Engagement Survey Results & Action Planning, we identified Team Member Development & Engagement as being an area to focus on. One of the actions we took was to identify a way to provide Engineering get-togethers for increased sense of belonging

After looking at different possibilities based on budget we were able to provide a subsidy in FY25 to facilitate these get-togethers, both in an in-person format as well as virtually.

Program Overview

GitLab has partnered with Plato HQ for an external Mentoring Program. In this program GitLab team members select Mentors external to GitLab. Some of the other Mentoring programs we have here at GitLab are internal to GitLab. Minorities in Tech and Women in Sales are both made up of GitLab Mentors and GitLab Mentees. The external mentoring is what makes this approach to GitLab unique.

For more information on mentoring best practice, visit Mentoring.

GitLab consists of many subprojects. A curated list of GitLab projects can be found at the GitLab Engineering projects page.

Creating a new project

When creating a new project, please follow these steps:

1. Read and familiarize yourself with our stance on Dogfooding. Be aware that as part of a product development organization that builds a tool for people like us, that our default is to add features and tooling to the GitLab project. This is still true when the effort to do so is 2-5x. Despite this, if you still feel you need to create a project outside of GitLab, you must follow this process to document the decision

Definition of an Incident

The definition of “incident” can vary widely among companies and industries. Here at GitLab, incidents are anomalous conditions that result in — or may lead to — service degradation, outages, or other disruptions. These events require human intervention to avert disruptions, communicate status, restore normal service, and identify future improvements. Incidents are always given immediate attention.

Incident Management

Incident Management is the process of responding to, mitigating, and documenting an incident. At GitLab, we approach Incident Management as a feedback loop with the following steps, with different teams adjusting them as needed:

Vision

Our vision is to be a world-class Infrastructure & Tools department that enables GitLab to meet & exceed our customers’ needs.

We:

1. Build critical infrastructure, metrics & tools that enable GitLab Engineering & Product teams to do their best work efficiently and ship high-quality & reliable products to our customers.
2. Are customer focused. We have an ambitious drive to attain high availability & reliability for SaaS platforms and self-managed customers.
3. Provide and maintain best practice tools and methodologies that create a platform for engineering teams to do their work productively.
4. Enable GitLab Engineering & Product teams to run services effectively using our tools, to meet business needs & SLOs.

Direction

Direction is set within the Infrastructure, and the Quality direction pages. With the ongoing consolidation of the departments, separate direction pages will become obsolete.

R&D OKR Overview

This page provides an overview of the joint R&D OKR workflow. All departments within R&D, which includes the Product and Engineering Divisions, collaborate by following this guidance. For clarifications on the OKR process, team members can post in Slack #product or #engineering-fyi.

Timeline and process for OKRs

The OKR process is designed to tie in to the overall OKR process the company uses. That process is driven largely off of the date of the Key Review meetings, so the Product process keys off of that date as well. Dates will not necessarily align with the start of a fiscal quarter as a result.

GitLab.com Service Availability

The calculation methodology for GitLab.com Service Availability definition is in the monitoring policy.

More details on definitions of outage, and degradation are on the incident-management page

Historical Service Availability

Related Pages

• Production Architecture
• GitLab.com Settings
• GitLab Performance Monitoring Documentation
• Performance of the Application

Related Videos

These videos provide examples of how to quickly identify failures, defects, and problems related to servers, networks, databases, security, and performance.

Expectations for On-Call

• If you are on call, then you are expected to be available and ready to respond to PagerDuty pages as soon as possible, and within any response times set by our Service Level Agreements in the case of Customer Emergencies. If you have plans outside of your workspace during your on-call shift, this may require that you bring a laptop and reliable internet connection with you.
• We take on-call seriously. There are escalation policies in place so that if a first responder does not respond in time, another team member is alerted. Such policies are not expected to be triggered under normal operations, and are intended to cover extreme and unforeseeable circumstances.
• Because GitLab is an asynchronous workflow company, @mentions of On-Call individuals in Slack will be treated like normal messages, and no SLA for response will be associated with them.
• Provide support to the release managers in the release process.
• As noted in the main handbook, after being on-call, make sure that you take time off. Being available for issues and outages can be taxing, even if you had no pages. Resting after your on-call shift is critical for preventing burnout. Be sure to inform your team of the time you plan to take for time off.
  • Team members in Australia should review the Australia time in lieu policy.
• During on-call duties, it is the team member’s responsibility to act in compliance with local rules and regulations. If ever in doubt, please reach out to your manager and/or aligned People Business Partner.

Customer Emergency On-Call Rotation

• We do 7 days of 8-hour shifts in a follow-the-sun style, based on your location.
• After 10 minutes, if the alert has not been acknowledged, support management is alerted. After a further 5 minutes, everyone on the customer on-call rotation is alerted.
• All tickets that are raised as emergencies will receive the emergency SLA. The on-call engineer’s first action will be to determine if the situation qualifies as an emergency and work with the customer to find the best path forward.
• After 30 minutes, if the customer has not responded to our initial contact with them, let them know that the emergency ticket will be closed and that you are opening a normal priority ticket on their behalf. Also let them know that they are welcome to open a new emergency ticket if necessary.
• You can view the schedule and the escalation policy on PagerDuty. You can also opt to subscribe to your on-call schedule, which is updated daily.
• After each shift, if there was an alert / incident, the on call person will send a hand off email to the next on call explaining what happened and what’s ongoing, pointing at the right issues with the progress.
• If you need to reach the current on-call engineer and they’re not accessible on Slack (e.g. it’s a weekend, or the end of a shift), you can manually trigger a PagerDuty incident to get their attention, selecting Customer Support as the Impacted Service and assigning it to the relevant Support Engineer.
• See the GitLab Support On-Call Guide for a more comprehensive guide to handling customer emergencies.

GitLab.com Reliability On-Call Rotation

Infrastructure Engineer On-Call

The Infrastructure department’s SREs provide 24x7 on-call coverage for the production environment. For details, please see incident-management.

We believe in Open Source

As a company, GitLab is dedicated to open source. Not only do we believe in it, but we use it, and we give back to it. Not just through GitLab, but through contributions to other open source projects.

The purpose of this page is to document how a GitLab employee can:

• Create an open source project on behalf of GitLab
• Contribute to a third-party open source project on behalf of GitLab
• Use a third-party open source code in a GitLab’s project

Growth Strategy

As an open source project, we want to stay healthy and be open for growth, but also ready to accommodate a 10x factor of our community. In order to achieve that, we’ve outlined a strategy that is a collaboration between multiple departments.

Performance Facets

We categorize performance into 3 facets

1. Backend
2. Frontend
3. Infrastructure

Backend performance

Backend performance is scoped to response time of API, Controllers and command line interfaces (e.g. git).

DRI: Tim Zallman, VP of Engineering, Core Development.

Performance Indicators:

• Memory Utilization (backlog)

Frontend performance

Frontend performance is scoped to response time of the visible pages and UI components of GitLab.

DRI: Tim Zallman, VP of Engineering, Core Development

The handbook pages nested under “policies” directory are controlled documents, and follow a specific set of requirements to satisfy various regulatory obligations.

Avoid nesting non-controlled documentation at this location.

GitLab submits applications for R&D Tax Credits in a number of jurisdictions that implement reimbursement schemes for research and development. A subject-matter expert (SME) from engineering is appointed to each application to assist with data collection. A third-party tax agent prepares and submits the report. SMEs are usually Engineering Managers or Directors and located in, or with reasonable knowledge of, the jurisdiction under application.

Role of the SME

The role of the SME is twofold:

Engineering Quarterly Achievers

Quarterly, CTO Leadership will recognize Engineering team members who have excelled in a given quarter. Recognition includes:

• an invitation to the Engineering Quarterly Achievers Chat
• participation in the Engineering Quarterly Achiever’s Recognition Dinner - an expensed meal for yourself, friends and family to celebrate your work, the meal must occur before the last day of the quarter following the announcement. Winners each quarter have until the last day of the quarter to submit for reimbursement. Winners may submit their receipt for the meal for reimbursement via Navan. Please see the instructions below.

In Navan, click Add Transaction Select Upload receipt (or select Type in details) Under Expense Type field, please select “Team events & meals” Under Classification, please select “FY25 Team Building” Under Description field, please include this link: https://handbook.gitlab.com/handbook/engineering/recognition/#engineering-quarterly-achievers-recognition-dinner Click Submit (or Save & close if you need to come back to add more information).

Overview and terminology

This page describes the processes used to release packages to self-managed users.

Monthly self-managed release

GitLab version (XX.YY.0) is published every month. From this monthly release, planned, and unplanned critical patch releases are created as needed.

Our maintenance policy describes in detail the cadence of our major, minor and patch releases for self-managed users. The major release yearly cadence was defined after an all stakeholder discussion.

Self-managed overview

The self-managed release is a semver versioned package containing changes from many successful deployments on GitLab.com. Users on GitLab.com, therefore, receive features and bug fixes earlier than users of self-managed installations.

At GitLab transparency is one of our core values, as it helps create an open and honest working environment and service, which in turn accelerates growth and innovation. We treat a root cause analysis (RCA) as an opportunity to be transparent amongst our organization and community by investigating what went well and what didn’t after working on a project, incident, or issue. This page defines an RCA, the benefits of completing them, and how to complete a successful RCA here at GitLab.

Starting new teams

Our product offering is growing rapidly. Occasionally we start new teams. Backend teams should map to our product categories. Backend teams also map 1:1 to product managers.

A dedicated team needs certain skills and a minimum size to be successful. But that doesn’t block us from taking on new work. This is how we iterate our team size and structure as a feature set grows:

1. Existing Team: The existing PM schedules issues for most appropriate existing engineering team
   • If there is a second PM for this new feature, they work through the first PM to preserve the 1:1 interface
2. Shared Manager Team: Dedicated engineer(s) are identified on existing teams and given a specialty
   • The manager must do double-duty
   • Their title can reflect both specialties of their engineers e.g. Engineering Manager, Distribution & Package
   • Even if temporary, managing two teams is a valuable career opportunity for a manager looking to develop director-level skills
   • Each specialty can have its own process, for example: Capitalized team label, Planning meetings, Standups
3. New Dedicated Team:
   • Engineering Manager
   • Senior/Staff Engineer
   • Two approved fulltime vacancies
   • A dedicated PM

Team Construction

Generally engineering teams at GitLab are fullstack, they are made up of Frontend, Backend, and Fullstack individual contributors with a single Engineering Manager.

An unplanned upgrade stop is disruptive for customers as it requires to perform rollback and additional maintenance work for performing the upgrade. Unplanned stops should be treated as incidents. The process below outlines the different stages of the incident resolution process and the steps to be taken by the corresponding teams and Directly Responsible Individuals (DRIs).

High-level workflow:

1. Detect unplanned upgrade stop: Identify instances of unplanned upgrade stops.
2. Resolve upgrade bug: Backport the fix or update Upgrade path to include new stop.
3. Perform Unplanned Upgrade Stop Root Cause Analysis: Understand why the stop occurred and prevent future incidents.

What is unplanned upgrade stop?

An unplanned upgrade stop happens when we fail to communicate the necessity of this upgrade stop in our upgrade path. For more information, read what an unplanned upgrade stop is.

Pilot Program Overview

This program allows team members at GitLab to volunteer and donate their time and technical skills (such as programming or Linux administration) to provide knowledge, support, and coaching to members of underrepresented groups (URGs) in the technology industry. The hope is we can help people who have been denied opportunity for whatever reason, and desire to get their first job in the technology industry.

This program is in pilot as of November 1, 2020. Please reach out to the contacts below if you are interested in taking part.