Switchboard team

Summary

Switchboard is the control plane for GitLab Dedicated. It is built by two closely linked functional teams within the Dedicated Group: Switchboard Orchestration API and Switchboard Customer Features. Together, our mission is to empower GitLab Dedicated customers to see and change everything they need in their tenant environments, and to give internal operators and automation a single, safe API surface for operating tenants, so that the GitLab Dedicated offering can scale without a matching increase in operational toil.

Because the two teams share a single codebase and set of working processes, both team charters live on this page. We follow the same processes as listed on the Dedicated Group page, unless a difference exists which is explicitly noted here.

Resources

• Switchboard handbook page
• Switchboard Demo Library
• Dedicated Group functional team charters (internal)

Team Charters

Switchboard follows the Dedicated Group functional team model: small teams in neighbouring time zones; one tech lead, one manager, and one product manager per team; clear ownership of artifacts; and a persistent problem area that defines each team. The sections below describe the mission, scope, people, and success criteria for each Switchboard team. The two teams are split along the boundary between the API and automated tooling used to maintain the Dedicated fleet and the customer-facing UI built on top of it - though some features built by the feature team also serve internal customers, and the feature team maintains an API for Dedicated customers to manage their tenants programatically as well.

Switchboard Orchestration API

Mission

Provide a well-governed API surface and set of tooling / services for everything needed to operate a GitLab Dedicated tenant, so that internal operators, automation, and customer-facing features all act on a tenant through the same safe and consistent interface.

Charter

The Switchboard Orchestration API team owns the Switchboard API and the orchestration interactions behind it. The team consolidates all of the interactions needed to operate a Dedicated tenant, defines ownership of and controls over the slices of the tenant model that other teams depend on, and moves business logic out of the frontend and into the API so that every consumer, human or automated, gets the same behaviour.

• In scope: the Switchboard Orchestration API and the consolidated set of tenant operation tooling used by operators and automation.
• Out of scope: internal or external customer-facing UI and workflows (owned by Switchboard Customer Features); Amp and Instrumentor (owned by DevOps Dedicated Orchestration Automation).

FY27 Priorities

• Complete adoption of API-first approach so Orchestration API is the single source of behaviour for tenant operations. By the end of FY27, there should be nothing left that is not represented in the API - hosted runners is a prime example of a large piece of functionality that had to move.
• Define API and artifact ownership, and the controls over tenant model slices, so that other Dedicated teams can self-serve safely
• Treat the DevOps Dedicated Orchestration Automation and Switchboard Customer Features teams as first-class customers of the API

Team members

Name	Role
Oriol Lluch	Manager, Infrastructure
Andrey Ruzmanov	Senior Backend Engineer
Graeme Gillies	Senior Site Reliability Engineer, Dedicated:Switchboard
Sean Arnold	Staff Backend Engineer, Dedicated:Switchboard

Responsibilities

The team serves:

• DevOps Dedicated Orchestration Automation, which owns the orchestration platform the API drives
• Switchboard Customer Features, which builds workflows on top of the API
• Internal operators and automation that need to act on Dedicated tenants

The team aligns its boundaries with those teams and maintains an ongoing conversation with the Service Delivery Platform and Cells operations teams.

Artifacts owned:

• The Switchboard Orchestration API
• Tenant model slice ownership, schema, and access controls
• Consolidated tenant operation interactions
• Workflow engine

What success looks like

• The Switchboard API responds to all orchestration needs in Dedicated
• Dedicated functional teams are empowered to use Switchboard’s APIs and functionality independently as established use cases evolve - without needing to wait for a customized Switchboard implementation.

Impact is also tracked via the Switchboard Performance Indicator (PI) Dashboard.

Switchboard Features

Mission

Give GitLab Dedicated customers self-service control for common tenant configurations and high-level visibility, removing human effort from routine change so that Dedicated can scale tenants without scaling SRE toil and manual processes.

Charter

The Switchboard Customer Features team owns the customer-facing Switchboard product: a self-service change-management console for Dedicated tenants that automates configuration and workflows. It removes human effort from routine change and is successful when the number of tenants can grow without a corresponding increase in operational toil.

• In scope: the Tenant Management Console UI; customer-facing self-service configuration; customer-facing status and SLA information; the customer experience of self-service migrations; Tenant Admin API for programmatic SLA and configuration access.
• Out of scope: the Orchestration API itself and tenant model governance (owned by Switchboard Orchestration API); internal responder tooling (owned by DevOps Dedicated Responder Automation). Also, any functionality or features for configuring the GitLab application or modular features (vs the underlying infrastructure) should be added to the GitLab Admin UI by default, not Switchboard.

FY27 Priorities

• Self-service Geo migrations
• SLA data and a customer-facing status page
• Expanding the set of tenant configuration changes customers can make without a GitLab team member involved

Team members

Name	Role
Matt Kirkevold	Engineering Manager

Responsibilities

The team serves:

• GitLab Dedicated customers, in everything they need to see or change in a Dedicated tenant
• GitLab Dedicated customer account teams (Support Engineering, Customer Success, Solution Architects, Professional Services) who partner with Dedicated engineering teams and Dedicated customers

The team aligns closely with Switchboard Orchestration API and with DevOps Dedicated Responder Automation the Orchestration Automation team - especially as more modular features are introduced in FY28.

Artifacts owned:

• Tenant Management Console UI (Switchboard)
• Customer-facing self-service configuration workflows (built on Temporal)
• Customer-facing status page and SLA data
• Internal operational workflows (e.g. maintenance, also built on Temporal)
• Tenant Admin API
• Authentication and user management
• Availability of Switchboard data in BigQuery

What success looks like

Dedicated scales tenants without increasing toil: routine change is initiated and completed by customers through Switchboard, with no human intervention from GitLab for BAU configuration changes.

Success metrics

Tracked via the Switchboard Performance Indicator (PI) Dashboard.

Working with us

To engage with the Switchboard team:

• Create an issue in the GitLab Dedicated team issue tracker

• Label the issue with:

  • component::Switchboard
  • workflow-infra::Triage
  • group::switchboard

• When creating an issue, it is not necessary to @mention anyone

• In case you want to get attention please use the specific team handle, @gitlab-dedicated/switchboard, as defined in Dedicated group hierarchy

• As a cross-functional team Switchboard uses @gitlab-dedicated/switchboard/frontend-engineers and @gitlab-dedicated/switchboard/backend-engineers internally to seek input from engineers with specific expertise

• Switchboard issue board tracks all of the team’s current work

• Slack channels

  • For Switchboard specific questions, you can find us in #g_dedicated-switchboard-team
  • Our Slack group handle is @dedicated-switchboard-team
  • Issues relevant to the wider Dedicated Group may be raised #g_dedicated-team
  • Other teams in Dedicated group have their own work channels for team work discussions:
    • #g_dedicated-environment-automation-team
    • #g_dedicated-us-pubsec

Requesting Access to the Switchboard application

Access provisioning differs by environment:

Test environment

Access to the Test environment is provisioned through Lumos. The Okta group previously used for Test access no longer exists.

1. Open a Lumos access request for Switchboard Test.
2. Specify the Role required (see Role description below).
3. For Permissions, select Other (unless you specifically need Dedicated Team Users).
4. Provide a justification for the access.

Beta and Production environments

Access to the Beta and Production environments continues to use the existing Google/Okta group process:

• Create an Access Request specifying
  • the specific environment ( Beta / Production )
  • Role required (see Role description below)
  • justification for the access
• Access & Provision Details for the application can be found in the Switchboard - GitLab Dedicated section of the Tech Stack

How we work

Meetings and Scheduled Calls

Our preference is to work asynchronously, within our project issue tracker as described in the project management section.

The team does have a set of regular synchronous calls:

• Switchboard Sync - During this call, we are sharing important information for team-members day-to-day, as well as project items requiring a sync discussion
• 1-1s between the Individual Contributors and Engineering Managers

Impromptu Zoom meetings for discussing Switchboard work between individuals are created as needed. It is expected that these meetings are private streamed, or recorded(1*), and then uploaded to GitLab Unfiltered playlist. The outcome of the call is shared in a persistent location (Slack is not persistent). This is especially important as the team grows, because any decisions that are made in the early stage have will be questioned in the later stages when the team is larger.

1* Exceptions to the recording rule are: 1-1 calls, discussions around non-project work, and in cases where parties do not feel comfortable with recording or we cannot record due to the nature of content discussed. However, even with the exceptions, outcome of project related discussions need to be logged in a persistent location, such as the main issue tracker.

Tracking, Planning & Delivering Projects

Resources

• Switchboard team roadmap
• Switchboard team top-level epic
• Switchboard team issue board
• Switchboard technical writing board
• Dedicated Issue Tracker

Quarterly Planning

Quarterly planning is owned and driven by the Switchboard EM and PM.

1. EM or PM creates a quarterly planning issue using the Dedicated quarterly_planning issue template.

   • Set the issue title to Switchboard - FYXXQY planning where XX is the year and Y is the quarter
   • Ideally planning should be kicked off at least a quarter in advance. The planning issue for the upcoming quarter should be created on or before the first day of the second month of the current quarter

2. EM & PM brainstorm asynchronously on the issue

3. During month 3 of the current quarter EM or PM

   • Reach out to the team for input
   • Ensure all dependencies are communicated with the relevant teams
   • Prioritise the objectives
   • Document the associated epics

4. EM creates a delivery epic for the quarter that will be used in Grand Reviews and updates the epic-summaries bot to reflect this change

Directly Responsible Individuals (DRIs)

Each and every project is owned and driven by a Directly Responsible Individual or DRI. Though Switchboard is a cross-functional team the DRI is responsible for delivering the project output including UX design, product decisions, frontend and backend implementation along with any other deliverables, regardless of their expertise. For example if the DRI is a backend engineer, while they may not directly implement every issue, they are still responsible for checking progress to ensure other team members have access to the information needed to fulfil the project. The DRI will not directly implement every piece of the project but they are responsible for ensuring it is delivered. This involves collaborating with the EM to ensure issues are prioritised within the engineering team, collaborating with the various functions within the team to ensure they have access to the information needed to fulfil the project (UX, frontend, backend, product etc), highlighting possible risks and ensuring alignment across the team.

We use issues to collaborate on ideas and solve problems. Everyone working on issues is responsible for keeping its description (SSoT) up to date with the latest decisions, and to ensure any follow-up resulting work is being tracked. The DRI is responsible for ensuring discussion threads reach a decision. DRIs should also drive a pivot to synchronous discussion when a thread is not yielding a timely conclusion. In the case where the UX design has been agreed upon prior to implementation, the DRI can accelerate the MR review process and reduce the workload on the PM and UX designer by ensuring the MR deliverable matches the design provided in Figma and approving accordingly.

The Epic Refinement process followed by the DRI at the beginning of the project is described below in the Epic Refinement section. The DRI is also responsible for weekly status updates (further details) and ensuring the epic has a demo link attached (see Switchboard Demos). The instructions for delivering the final status update and closing an epic can be found here.

Epic Refinement

Switchboard team process to refine epics:

1. Identify a DRI for the epic
   • Team members can volunteer or the EM may ask team ask specific team members to act as DRI
2. Identify any missing requirements
   • All team members ask questions in the comments of this issue to drive out any edge cases
3. DRI labels the epic as ~“workflow-infra::Ready”
4. DRI ensures Epic Kick-Off is complete - see Epic Template
5. Assign Due Date & Start Date
   • DRI, EM & PM work together to assign due date based on team capacity, external deadlines and amount of work involved
6. DRI identifies at least one demo that will be delivered with the epic and adds a brief outline to the epic description (see Switchboard Demos).
7. For substantial code modifications or architectural additions, the DRI creates and shares draft technical documentation with team early, if possible, before development work starts.
   1. Draft documentation should contain a high level technical explanation of the new system and how it is meant to work
   2. Illustrative diagrams like sequence diagrams are encouraged
   3. Technical documentations can be added to the switchboard project where they currently live under ./docs directory
8. EM or DRI labels individual issues as ~“workflow-infra::Triage”
9. DRI enables issues to be worked on in parallel where possible so that multiple engineers can contribute to a single epic
10. If the epic involves both Frontend and Backend implementation the issues should be labeled accordingly
11. Team members pick up issues and start working on them
12. Team members use Progress Threads to track progress in individual issues
13. Team checks in on progress during Switchboard Sync
    • Any epics with no issues to raise can be read-only
    • Priority given to discussions of epics based on soonest due date
14. If due date is not realistic team member comments on the issue as early as possible so that the team can work together to address this

Note 1, 2 & 4 can be carried out in parallel

Issue Refinement

Switchboard team process to refine issues:

1. When an issue is created and ready to be refined it is labeled ~“workflow-infra::Triage”
2. PM and EM ensure that the Open and ~“workflow-infra::Ready” columns are prioritised
3. Team members look at issues in the Open column of the issue board and ask questions on the issue to drive clarity
4. When there are no outstanding questions on the issue it can be labeled ~“workflow-infra::Ready” and it will automatically move into the Ready column
5. If the issue exposes text to users in any way the technical writing label should be added. For example if the issue changes UI text, shows an error message, adds a field etc
6. If the issue requires Frontend implementation the frontend label should be used
7. DRI enables issues to be worked on in parallel where possible so that multiple engineers can contribute to a single epic
8. The default is to keep both frontend and backend implementation for a single piece of functionality on the same issue so that discussions are centralised, implementation is carried out in parallel and frontend and backend engineers are in sync
9. If the implementation requires modification or creation of an API endpoint, a plan on endpoint, params structure and return data structure should be agreed upon between Frontend and Backend as early as possible to avoid re-work.
10. Frontend and backend implementation should be delivered in separate MRs
11. If the implementation cannot be done in parallel, or there is a likely to be a meaningful delay between backend and frontend implementations, or if the backend can deliver value independently the issue should be split and the relationship clearly identified by linking the issues
12. SRE dependencies should be called out as early as possible with requirements clearly documented in issues (example SRE dependency issue)

Issue & Epic tracking

1. Engineers use Progress Threads to share progress in an async fashion
2. At the beginning of the Switchboard Sync the team will check in on epics labeled ~“workflow-infra::In Progress” or ~“workflow-infra::Triage” to ensure due dates are appropriate and highlight any blockers
3. Epic DRIs update the status in the Epic Description every Wednesday in preparation for the Grand Review
4. Epic DRIs review the due date weekly. The epic status update should include the DRI’s confidence level in the due date and any risks to delivery

Picking up work / What to work on next

1. ~“workflow-infra::Ready” column on the issue board

   1. Pick an issue from the ~“workflow-infra::Ready” column on the issue board
   2. Assign the issue to yourself and set to ~“workflow-infra::In Progress”
   3. Update the issue description with an Implementation Plan where relevant
   4. Create a Progress Thread on the issue and update daily

2. Switchboard team top-level epic

   1. Look at the Switchboard top level epic and offer to work on issues with the nearest due date
   2. Use the Switchboard team roadmap for guidance

3. Open column on the issue board

   1. Look at the issues at the top of the Open column
   2. For each issue ask questions and drive a discussion to identify any missing information
   3. Mark the issue as ~“workflow-infra::Ready” if it is ready to be worked on or @mention the EM if you are not confident to do so

Switchboard Demos

A demo enables a team member to share progress or the final output of an issue or epic. The focus is on sharing information, not creating an oscar winning documentary, so as a team we use the boring solution of either a screen recording, a recorded Zoom meeting or Loom. A link to the recording is added to the epic description.

The epic DRI is responsible for identifying at least one demo that will be delivered with the epic when the epic is being kicked off. For example When the functionality x is delivered (or issue y is Done) we will demo functionality x. Team members are encouraged to time demos to be delivered shortly before the fortnightly Switchboard Team Syncs whenever possible so that any synchronous Q&A can happen during already reserved time. Team members may choose to create additional demos to share progress or delivery milestones.

Issue Templates

Switchboard maintains the following issue templates:

Merge Request Review Guidelines

We specifically adhere to the GitLab Code Review Guidelines and follow the Dedicated group principles when requesting merge request reviews.

Merge request review process

As the Switchboard team is currently small, we use an ‘Approve and Merge’ approach:

1. When you’re ready to have your merge request reviewed, select one or more Switchboard reviewers.
   • If you’re not certain about who to choose, you can use the reviewer roulette to randomly select a reviewer.
   • If the issue is labeled technical writing add the Switchboard technical writer as a reviewer
2. Reviewers will perform a review based on reviewing a merge request guidelines.
3. If satisfied, a reviewer will approve and merge unless other reviewers have questions or suggestions that are not addressed.
4. If the merge request contains the required approvals, the reviewer will trigger a pipeline and set auto-merge.
   • If the reviewer does not have merge permission, they should seek out a maintainer for merging.

Additional UI Review Process

In addition to the above when a change is being proposed to the UI the following extra steps should be followed:

UI changes visible to internal GitLab users:

1. MR author cc’s PM & UX Designer on the MR but they are not reviewers or blockers of the merge
2. If they have any suggestions they can be dealt with on the MR or on a later MR at the MR author’s discretion
3. Eventually the PM & UX Designer will be reviewers on the internally visible UI updates but our process is not there yet nor is their capacity
4. If you require help or guidance with the UX or copy please ask before implementation begins on the issue

UI changes visible to external customers:

1. It is highly important to figure out any outstanding questions on the issue, including design changes, copy changes, and product requirements so that we can avoid ambiguity at the MR stage. This must translate to a concrete conclusion of design discussions in the issue before UI code changes are started.
2. In the event of a missing UX/product piece at the MR stage, the DRI and issue assignee will collaborate to make a decision about whether the missing piece can be resolved on the MR or if it needs to be moved back to the issue for resolution. This is in accordance to the DRI definition.
3. The PM and Designer will deal with these requests as a top priority.
4. Add the PM as reviewer on the MR to keep them updated. This review will not impede the MR progress, and the PM will handle it as a high priority. Ideally, product requirements should be finalized in the issue.
5. If there is a new copy change, add the Technical Writer as a reviewer to keep them informed and this will be non-blocking (copy should ideally be agreed on in the issue). Additionally add the Technical Writing tag to the MR.
6. Cc the UX Designer on the MR and when they are ready to be core reviewer this will be communicated to the team

Note: If significant discussion ends up being needed for a UI change (internal or customer facing) after the MR has been opened, that discussion should be moved back to the issue to resolve and the MR marked as blocked. These discussions will be high priority to resolve and the issue should be assigned to the PM and Designer until progress on the MR can resume. Notes:

• It is our intention to move towards a typical ‘reviewers and maintainers’ approach which would require two reviews as soon as we have the team members to support this.
• Merge requests should be approved based on the approval guidelines.
• As per the GitLab Review Guidelines there are scenarios where it is appropriate for the author to merge the merge request: If there are no blocking comments, and the merge request has all the required approvals, the author or maintainer can merge.
• Switchboard project is configured to use pipelines for merged results which means that reviewers need to run a pipeline pre-merge to guarantee that updates are compatible with the latest main branch.
• When reviewing merge requests, reviewers should use the Conventional Comment labels to convey your intent.
  • For the avoidance of doubt Suggestion:, Issue: and Chore: comments are all blocking, unless decorated with a (non-blocking) statement.
• We label merge requests using the Specialization labels found in the GitLab documentation. MRs should be labeled ~“frontend”, ~“backend” or ~“documentation”

Approval guidelines

Reviewer roulette

Reviewer roulette is an internal tool for use on GitLab.com projects that randomly picks a maintainer for each area of the codebase. To select a maintainer:

1. Go to the reviewer roulette page.
2. Select the Switchboard project.
3. Choose the desire role: ~maintainer::frontend, ~maintainer::backend, ~reviewer::backend, ~reviewer::frontend.
4. Click on Spin the wheel.

GitLab Code Review Guidelines

• Having your merge request reviewed
• Reviewing a merge request
• The Right Balance
• Quality
• Performance, reliability and availability
• Merge request performance guidelines

Reviewers and maintainers

There are two groups for Switchboard, Reviewers and Maintainers:

• All Switchboard team members are included in the Reviewer group.
• When a team member is fully onboarded and feel confident in their knowledge of the codebase they are invited to the Maintainer group.

Feature Flags

Purpose

Feature flags in Switchboard enable safe, incremental rollouts of new functionality and provide kill switches for rapid rollback when issues arise.

Definition of Done

Feature flag code must be removed after successful rollout. A feature is not considered complete until:

• The feature has been rolled out to 100% of applicable tenants
• The feature has been stable for a previously agreed upon amount of time (usually determined at epic planning level), with a minimum of 1 week.
• All feature flag conditionals have been removed from the codebase

Working with Feature Flags

• Assign an owner - Every feature flag must have a designated owner(s) responsible for its removal. The owner(s) will be set as the assigned individual(s) in the removal issue.
• Create cleanup issue upfront - When introducing a feature flag, immediately create a follow-up issue for flag removal and link it to the implementation issue
• Document removal plan in epic - The epic introducing the feature flag must include the expected removal timeline and criteria for cleanup in the feature flag section
• Default to short-lived - Flags should typically be removed within 1-2 weeks of 100% rollout

Stale feature flags create technical debt and obscure the intended behaviour of our system. Timely cleanup is essential to maintaining code quality.

Epic Template

### :levitate: DRI
- TBD

### :busts_in_silhouette: Participants 

- Frontend Engineer:
- Backend Engineer:
- SRE:
- Product Designer:
- PM:
- Technical Writer:
- EM:


### :thinking: Problem to solve
<!-- Describe the problem or opportunity this feature addresses. What is broken, missing, or inefficient? Focus on the "why" — what user or business need is driving this work? -->



### :bust_in_silhouette: Target users
<!-- Who will benefit from this change? Describe the Switchboard user groups affected (e.g., customer tenant admins, internal operators, internal support users, etc). Include any relevant context about their workflows.-->



### :bulb: Proposal
<!--Describe the proposed solution at a high level.-->



### :crystal_ball: Design spec

* Spec issue:
* Walkthrough video:
* Prototype:
* Figma:

### :link: Dependencies

### :book: Documentation

* Public docs:
* Internal docs:

### :key: User permissions

| User role | Visible | Notes |
|-----------|---------|-------|
| [Internal - Operator / Migration Operator](/handbook/engineering/infrastructure-platforms/gitlab-dedicated/switchboard/#roles) |  |  |
| [Internal - Hosted Runner Operator](/handbook/engineering/infrastructure-platforms/gitlab-dedicated/switchboard/#roles) |  |  |
| [Internal - Provisioner](/handbook/engineering/infrastructure-platforms/gitlab-dedicated/switchboard/#roles) |  |  |
| [Internal - Support](/handbook/engineering/infrastructure-platforms/gitlab-dedicated/switchboard/#roles) |  |  |
| [Internal - Read Only](/handbook/engineering/infrastructure-platforms/gitlab-dedicated/switchboard/#roles) |  |  |
| [External - Tenant Admin](https://docs.gitlab.com/ee/administration/dedicated/configure_instance.html#add-users-to-an-instance) |  |  |
| [External - Read Only](https://docs.gitlab.com/ee/administration/dedicated/configure_instance.html#add-users-to-an-instance) |  |  |

| Functionality | Roles with Access | Fleet-Level (Y/N) | Permission Name (set by Engineering) |
|---------------|-------------------|-------------------|--------------------------------------|
| e.g. View runner |  |  |  |
| e.g. Create runner |  |  |  |

### :flag_white: Feature flags
<!--This table should document any feature flags that were added or removed by the epic.-->

| Feature flag | Details | Removal Issue |
|--------------|---------|---------------|
|  |  |  |

### :arrow_right: Rollout and customer communications (optional)

_Use this section if the epic needs a rollout or customer communications after the build is complete. The DRI and the PM do this work together. Remove this section if it is not necessary._

- [ ] Make the rollout plan with the PM, for example internal users first and then all tenants.
- [ ] Tell the account teams before the release.
- [ ] Write the customer communications with the PM. Post them in #dedicated-customer-hub.
- [ ] If the change can disrupt customers, get approval from the account teams before the release.

### :books: Resources

### :arrow_forward: Demo

https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/gitlab-dedicated/switchboard/#switchboard-demos

---

<!-- STATUS NOTE START -->

<!-- STATUS NOTE END -->

/label ~"group::switchboard" ~"workflow-infra::Triage"
/confidential

View page source - Edit this page - please contribute.