Production Engineering Foundations Team

Mission

The mission of the Production Engineering Foundations team at GitLab is to own the lifecycle of the core infrastructure for GitLab Saas instances and services. We seek to reduce the effort required to provide our core infrastructure services, and to enable other teams to self-serve core infrastructure that allows them to more efficiently and effectively run their services across all GitLab offerings.

Vision

The Foundations teams North Star consists of two pieces:

1. Creating a platform that drives vertical ownership of engineering services. We will do this by providing the rest of engineering the resources and tools needed for them to manage their infrastructure and core services in a way that makes it easy for other teams to adopt the best practices and conventions defined by our team. By moving towards centralized sets of tooling for all GitLab instances and all services to work from, we will create a culture of scalability for the future of GitLab infrastructure.
2. Sustainable toil. Toil is a given in SRE work. We will adopt processes and policies that create a balance between what is automated and is done manually so as GitLab grows, the toil involved in running it is maintained at a minimal level for the team.

Responsibilities

Ownership

There are two primary areas can be considered our flagship services that we actively work to improve and expand:

• Networking (From edge, to ingress. Including, but not limited to: load balancing, WAF, DNS, VPC)
• Rate limiting

Services

Foundations is responsible for several services related to GitLab SaaS Platforms. The Services that the Foundations team is responsible for fall into two general categories: Core and Edge.

Core

Core services are services that are within GitLab.com’s ecosystem, generally not talking directly to incoming traffic.

Edge

Edge services are services that sit at the network edge, or more generally, services that process inbound traffic.

Getting Assistance

Should you require assistance from the Foundations team, please open an issue in the Production engineering tracker

Apply one of the following templates:

• For issues pertaining to Teleport connections, use the teleport-troubleshooting template

• For issues pertaining to Cloudflare, use the Cloudflare Troubleshooting template

• For anything else, use the request-foundations template

• We also have team handles that ping the full team

  • Slack: @infra-foundations
  • GitLab: @gitlab-org/production-engineering/foundations

We rotate triage duties weekly between team members as part of our interrupt rotation.

Priority Levels for Requests

Note: Due to the complexity of our systems, we cannot commit to a resolution-based SLA when you open an issue with a given priority level, because we don’t know how long something will take to resolve, and there is often more information we need from the stakeholder or external customer, which takes time to obtain. Therefore, we can only commit to an SLA for how quickly we will engage with an issue and prioritize the work.

We endeavor to triage incoming requests twice per week. If you have an issue that is very urgent and requires immediate attention (i.e. Priority 1), please also send us a link to the request issue in our Slack channel: #g_foundations.

Considerations for tooling and maintaining our services

• We support dogfooding when it makes sense to do so.
  • We consider the scale at which we run gitlab.com and the additional engineering effort required to dogfood, weighed against the capacity and other priorities of the team.
• When considering external tooling, we look first to other open source projects that are part of CNCF
• Further evaluation criteria include:
  • Technical Fit - How well does it solve the problem we need it to solve?
  • Service Maturity - How well developed is the tool, does it have a proven track record of being used at scale?
  • Maintainability - Does it have strong community engagement, which will ensure it is well maintained for the foreseeable future? Complexity of keeping it up to date?
  • Cost - Licensing costs when applicable, weighed against the cost of building and maintaining the tooling ourselves. Engineering time required to implement new tooling and maintain it, weighed against the benefit it brings.
  • License - Identify the specific license under which the open-source project is distributed (GNU, MIT, ISC, etc.). This allows us to ensure compatibility with other tools’ licensing, contribute to the project if needed in the future, and understand the restrictions and risks associated with a specific license.

Key Performance Indicators

KPIs for the team are currently under development in https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/24928.

We also internally track team and internal customer satisfaction with regular surveys.

Internal survey results

• Team Satisfaction - current status: green
• Customer Satisfaction - current status: green (4.25/5.00)

For more context, see the related discussion issue.

Team Members

Name	Role
Steve Abrams	Engineering Manager, Saas Platforms:Foundations
Adeline Yeung	Site Reliability Engineer
Donna Alexandra	Senior Site Reliability Engineer
Jack Stephenson	Senior Site Reliability Engineer
Marcel Chacon	Senior Site Reliability Engineer
Pierre Guinoiseau	Senior Site Reliability Engineer
Site Reliability Engineer	Site Reliability Engineer
Tarun Khandelwal	Site Reliability Engineer Saas Platforms:Foundations

Key Technical Skills

The Foundations Team must maintain a broad and diverse set of technical skills while also maintaining the ability to switch contexts frequently. Some of these technical skills include:

• Cloudnative Engineering - Proficiency in Kubernetes and the associated ecosystem of running cloudnative services.
• Infrastructure as Code - Proficiency in Chef and Terraform
• Network Systems - Understanding of network concepts and experience with our Edge stack (see Edge services above)

Common Links

• Foundations Top Level Epic
• Foundations team meeting agenda
• Foundations OKRs
• GitLab Issues capture smaller, concrete pieces of work, and those labeled ~workflow-infra::In Progress should be updated weekly or whenever a portion of work has been completed.
• #g_foundations - work related discussions, external requests, etc
• #g_foundations_social - socializing and standups
• #g_foundations_notifications - automated MR notifications
• #g_foundations_alerts - pipeline failures and service alerts

How We Work

Team Sync meetings

• We have weekly synchronous meetings in two different time zones to encourage discussion, give status updates, triage incoming requests, and connect as a team.
  • Agenda
  • Youtube playlist.

Calendar

• We have a internal team calendar for regular team meetings.
  • To subscribe to the calendar you can use Other Calendars > Subscribe to calendar with c_fbda056c7fecb09a0d0d0801527838384fe862b80f2e36cdc24bca7b0d9d274f@group.calendar.google.com as the ID.

Standup

We have Geekbot automated checkins on Mondays and Fridays in the #g_foundations channel.

• Monday questions include:
  • What are your top priorities this week?
  • What did you accomplish last week?
  • What else is on your todo list for the week?
  • Any blockers or requests for pairing?
• Friday questions include:
  • How did your week go?
  • What were your wins this week?

Retros

We have a quarterly async retro that aligns with the company fiscal quarters and OKRs. In addition to reflecting on our quarter in general, we take a set of actions to try the next quarter and revisit how successful they were in the next retro.

Picking up work

We have three buckets of work:

1. OKRs (project work)
2. External requests
3. General operations, also known as Keep the lights on (KTLO)

While we rely on the workflow-infra::* labels to denote the current state of any given issue, we rely on the Build Board to understand what specific issues are ready to be picked up.

In short, anyone on the team can find issues to work on in the Build Board.

When something is prioritized to be worked on, the Foundations::Build label is applied so it is included on the board. With the exception of some external requests, issues being added should be in workflow-infra::Ready (clearly defined and ready for any team member to pick up).

We want the build board to be relatively small in size so it is easy to traverse and there is no confusion around what needs to be worked on. It can provide some options for people to find work, but also prevent key work from being missed.

Different people are responsible for adding the Foundations::Build label to issues for each bucket of work:

• OKRs - DRIs of each OKR uses the Foundations::Build label to communicate what the next issues needed to be picked up are.
• External requests - the Engineering manager will add Foundations::Build to issues that need attention. These will also include the unblocks others label to make it clear to people on the interrupt rotation which issues are external requests. The person on interrupt rotation can also actively triage incoming issues, adding the Foundations::Build label when new issues are opened.
• KTLO - These issues will generally be added according to a general combination of due date/priority/severity/weight. Any team member should feel empowered to add KTLO issues to the board when they fit the criteria (prioritization makes sense based on the due date/priority/severity/weight).

From here, people have options when looking for what’s next:

1. Pick up issues related to the OKR they are involved with
2. Pick up KTLO work they are interested in or to create a break between other project work
3. Help out with external requests if the interrupt rotation has higher volume on a given week

We trust everyone to be a Manager of One - owning not just their own work, but the team’s work - taking responsibility to define their own workflow that provides the right balance of work for them to be most impactful to the team.

Prioritization of work

We use priority labels to prioritize our work. OKR work is updated to Reliability::2 when it is ready to be worked on. As such it is given higher priority than most other work. This means that external requests will be worked in based on their own priority and impact, where only P1 and P2 issues will regularly interrupt OKR work.

Interrupt Rotation

The goal of the team member on interrupt rotation is to purposely plan to have interrupt work for a week so that the rest of the team can have less interrupt work. The schedule is available in pager duty.

If a team member is unavailable for more than 2 days of the week they are on triage, they should seek to trade weeks or find coverage for the days they are out.

The team member on rotation is responsible for:

• Checking the Foundations Build Board throughout the week and ensuring new issues related to Foundations:
  • Have the correct labels (priority, workflow, any other applicable category labels)
  • Have a weight.
  • Are assigned to an appropriate epic if applicable.
• Asking issue authors for more information when needed to understand the scope of work.
• Raising any Reliability::1 or Reliability::2 issues and ensuring they have a DRI and are being actively worked on.
• Completing any issues that come up that can be completed during the week. Weight 5 issues can be considered depending on the priority.
  • Anything weight 8 or larger (more than 1 week of work) should be made clear that it will need to be scheduled around other work.
• Responding to alerts in #g_foundations_alerts.
• Working on Renovate MRs posted in #g_foundations_notifications.
  • Patch updates with passing pipelines should be merged.
  • Minor updates should have their changelogs be reviewed before merging.
  • Major updates should be looked at closely to understand breaking changes before considering merging.
• Responding to Terraform drifts related to Foundations services posted in #infra-terraform-alerts.
• Respond to any Access Requests that need Foundations provisioning.

Project Management

The Foundations team top level Epic can be found here We follow Platforms Project Management practices as outlined in the Handbook. Below builds on top of those guidelines.

OKRs

• For Objectives and Key Results, we align with Platforms guidance for creation and structure.

Epics

In addition to the format described in the platforms project management page, these sections may be helpful

## Administrative

<!-- A copy paste section for creating child epics/issues, ensuring that they relate to the current epic and have the correct labels -->

\`\`\`
/epic [current epic]
/labels ~"group::Production Engineering" ~"Sub-Department::SaaS Platforms" ~"team::Foundations" ~"workflow-infra::Triage" ~"Reliability::P2"
\`\`\`

## References

<!-- Links to related OKRs, Epics or issues, external resources etc -->

## Demos

| Demo Date | Demo Link | Highlights |
|-----------|-----------|------------|

## Decision log
<!-- A collapsible section to aggregate any decisions made along the way. Be sure to include "why" in addition to "what". -->

<details  >
  <summary>Log</summary>
  
<details  >
  <summary>date</summary>
  <p>[decision taken and why]</p>
</details>

</details>

• Apply any applicable service labels.
• When changing the health status to anything other than “On track”, give context for why in the weekly status update, including the plan to get back on track when possible.

Issues

Open planned work for our team is located in the Production Engineering project. Issues should be updated whenever significant work occurs. New issues are expected to:

• Link to a related Epic.

• Include the following Labels (update the priority as needed):

  /labels ~"group::Production Engineering" ~"Sub-Department::SaaS Platforms" ~"team::Foundations" ~"workflow-infra::Triage" ~"Reliability::P4"
  

• If there is a service label that is applicable, also apply that.

Issue weighting

We dogfood the weight feature in issues. We use a modified Fibonacci sequence (the numbers match roughly with the number of days) and align our weights to t-shirt sizes to make them easy to talk about:

• Fib 1 (XS) - A few hours of work. Consider if the work should just be completed now depending on competing priorities.
• Fib 2 (S) - A day or two of work.
• Fib 5 (M) - Up to a week of work.
• Fib 8 (L) - Over a week of work. This issue should likely be broken down into smaller pieces.
• Fib 13 (XL) - Several weeks or more of work. This issue will need to be broken down into smaller pieces or promoted to an epic.

View page source - Edit this page - please contribute.