The GitLab Dedicated team’s mission is to create a fully managed, single-tenant GitLab environment, served through a GitLab Dedicated platform. It is developed to remove any manual interactions with customer tenant installations, and to ensure that the customer tenants are fully focused on unlocking the power of The One DevOps Platform.
The GitLab Dedicated group is a customer facing team, with team members focused on a high level of infrastructure automation, and enabling customer interactions with the GitLab Dedicated platform.
Team mission is to:
For GitLab Dedicated Architecture documentation please refer to the Architecture Page
Team performance indicators are not fully defined. We are going to consider a Provisioning SLO to start with, possibly followed by DORA 4 metrics.
Engineering team members of GitLab Dedicated are publicly referenced with specialty for Environment Automation, Switchboard or US Public Sector Services team, based on their primary task.
The following people are members of the Dedicated:Environment Automation Team:
The following people are members of the Dedicated:US Public Sector Services Team:
| Name | Role |
|---|
The following people are members of the Dedicated:Switchboard Team:
To engage with the GitLab Dedicated teams:
Oriol Lluch
Andy Knight
Ermia Qasemi
Muhamed Huseinbašić
Samir Hafez
Nick Skoretz
Tania Roblot
Stephen Denham
Brendan McKitrick
Corey Cross
Harpratap Singh
Konst Tchernov
Riccardo Trivellato
Stephan Breitrainer
Veronica Mondo
Amy Shiel
Andrey Ruzmanov
Grace Lin
Graeme Gillies
Marina Mosti
Sean Arnold@mention anyoneCustomers require the ability to customize the configuration of their Dedicated instance before they are able to use it in a production setting. These customizations involve configuration changes to functionality already supported in Dedicated including infra-level settings like IP Allowlists and Cloud-Specific Private Networking configuration as well as GitLab application settings that cannot currently be self-served through the admin UI like SAML configuration changes.
To request functionality that is not currently supported within Dedicated, customers must open a feature request using the feature request issue template. To request functionality for the broader GitLab Application, customers can use the feature proposal template.
While in the long term, customer admins will be able to self-serve configuration changes using the Switchboard customer portal. In the short term, SREs will need to make the change and deploy it to the customer’s environment. This process is documented below.
start date specified in the customer contract) and make any needed configuration changes to the environment.@ mention the SRE in question. The SRE will assign the issue to themselves and perform the change.start date will be batched and handled in the next available weekly maintenance window. The cut off to get changes in is 2 business days before the maintenance window begins.While changes we make are rigorously tested and carefully deployed, it is a good practice to temporarily halt production changes during certain events such as GitLab Summit, major global holidays, and other times where GitLab Team Member availability is substantially reduced.
Risks of making a production environment change during these periods includes immediate customer impact and/or reduced engineering team availability in case an incident occurs. Therefore, we have introduced a mechanism called Production Change Lock (PCL) to GitLab Dedicated.
The GitLab Dedicated Production Change Lock is greatly inspired by the PCL for GitLab.com, but there are some differences worth noting.
A PCL is manually enforced once the following requirements are met:
The following dates are currently scheduled PCLs.
| Dates | Type | Reason |
|---|---|---|
| 2024-12-23 03:00 UTC -> 2025-01-06 03:00 UTC | Hard | Year End Holidays (Low team member availability) |
Times for the dates without a time specified begin at 09:00 UTC and end the next day at 09:00 UTC.
As opposed to GitLab.com PCL, for GitLab Dedicated we only consider a Hard PCL type.
Hard PCLs include:
New customers will not be onboarded during Hard PCLs.
In case of an active S1/S2 incident, it is at the EOC (Engineer on Call) discretion to make the decision to apply the changes necessary to mitigate or resolve the incident in order to keep service availability. Any action during an incident while in a PCL must be associated to an issue and the EOC should inform the GitLab Dedicated engineering Leadership about the action taken.
Changes not associated to any incident must have an exemption approval by the GitLab Dedicated engineering Leadership.
When it comes to escalating customer support issues, we follow the same definitions of severity as provided by support since Dedicated customers receive priority support. Only in cases where there is an availability or security sev-1 event it can be escalated to Sev-1 at the Support level. ‘Business as usual’ configuration changes cannot be escalated to sev-1. In sev-1 cases we will involve our on-call, as these incidents may affect our Availability SLA commitments to the customer. Sev-2 and below will be handled by the team during normal business hours. Any fixes identified as part of a support ticket that must go out immediately will be considered “emergency maintenance” and can be done outside the normal maintenance window. All other fixes will be done during the next available maintenance window.
GitLab Dedicated comes with strict access controls for tenant environments. By default, GitLab Dedicated logs are only accessible by members of the GitLab Dedicated and Support Engineering teams. In cases where GitLab Dedicated customers are impacted by issues that require additional team members to review logs, access can be granted on a case-by-case basis. Examples include, but are not limited to: Backend Engineers or Security Engineers in other departments.
Access will only be provided to:
To gain access, please create:
GitLab Dedicated Logs (Production) as the system.Log rotation access template.If you need to urgently contact one or more GitLab Dedicated customers, engage the GitLab Dedicated Communications Manager On-Call (CMOC).
Non-urgent communication should be handled through the customer’s Customer Success Manager (CSM).
Sometimes, a product fix is introduced to resolve a GitLab Dedicated incident or customer escalation. There can be a significant delay between when the product fix is merged and when it is deployed to GitLab Dedicated environments due to our upgrade policy.
In such cases, we should evaluate the impact of the delay and, if justified, use the backport request process to request that the product fix be backported to a GitLab version that can be deployed to GitLab Dedicated environments in an acceptable time frame.
GitLab Dedicated is highly visible within GitLab and the broader market. It is also complex service offering. Consequently, the Dedicated team needs to manage its own work and stakeholder expectations - both external and internal. The following sections describe processes and tools that the team has adopted to work efficiently and effectively.
These processes matter because they provide a clear way to understand the state of the product and ongoing work as well as enable team members to fulfill their role.
A main goal in designing these processes is to make it possible for every team member working on any part of GitLab Dedicated to understand why their work matters and how it contributes to customer results.
It is critical that the following processes are understood by all team members and that we hold ourselves accountable.
Our preference is to work asynchronously, within our project issue tracker as described in the project management section.
The team does have a set of regular synchronous calls:
Demo call - This call is scheduled every two weeks, rotating across timezones. During this call, team members show off their progress, and engage with other team members on topics related to GitLab Dedicated platform. Demo calls are supposed to be rough around the edges and unpolished. In fact, if the demo looks polished, we will discuss whether we are being ambitious enough with our goalsTeam SyncThe group has a set of regular synchronous calls for PMs and EMs to ensure alignment:
GitLab Dedicated Product <> Eng Sync - This call is weekly on Mondays and Thursdays for PMs and EMs to alignDedicated Managers Sync - This call is every two weeks for Dedicated EMs to sync and ensure alignmentIt is the organizer’s responsibility to ensure these calls can be recorded regardless of whether the organizer is present by enabling “Alternative Hosts” in Zoom.
Impromptu Zoom meetings for discussing GitLab Dedicated work between individuals are created as needed. It is expected that these meetings are private streamed, or recorded(1*), and then uploaded to GitLab Unfiltered playlist. The outcome of the call is shared in a persistent location (Slack is not persistent). This is especially important as the team grows, because any decisions that are made in the early stage have will be questioned in the later stages when the team is larger.
1* Exceptions to the recording rule are: 1-1 calls, discussions around non-project work, and in cases where parties do not feel comfortable with recording. However, even with the exceptions, outcome of project related discussions need to be logged in a persistent location, such as the main issue tracker.
GitLab confidentiality levels require that we do not publicly identify customers without their express permission.
When it is necessary to refer to a specific customer we use the following guidelines:
| Use Case | Example | Process |
|---|---|---|
| Internal Communication & Collaboration | Slack conversations |
|
| Public Collaboration | Collaboration on issues, including SIRT issues, epics and recorded calls |
|
| Implementation Level | We need the ability to map a codename to a tenant_id within the tech stack |
|
We use GitLab Groups to logically organize team-members working on GitLab Dedicated projects. The groups cover the following use-cases:
@gitlab-dedicated
@gitlab-dedicated/environment-automation, @gitlab-dedicated/switchboard, @gitlab-dedicated/uspubsec, etc.
maintainers and reviewers, for example: @gitlab-dedicated/switchboard/maintainers
reviewers GitLab group access is granted to permanent team-members, external contractors, team-members on borrow, and similar. This GitLab group type is used to distinguish users without merge rights. Initial reviews should be requested from this group, using the quick action, for example /assign_reviewer @gitlab-dedicated/switchboard/reviewersmaintainers GitLab group is granted to permanent team-members only. This group has merge rights, and the group is granted access through CODEOWNERS approval rules. Team members onboard into the maintainer subgroup after meeting the requirements defined in the Dedicated Maintainer Training(internal only)We use epics, issues, and issue/epic boards to organize our work, as they complement each other.
Please see:
We use sub-epics to break larger epics into smaller portions.
The diagram below shows an example of traversing the complete hierarchy:
graph TD A(GitLab Dedicated) A --> D([Epic]) A --> E([Dedicated Runners on AWS - Beta]) A --> G([...]) D --> H([Sub-epic]) E --> I([Dedicated Runners - Operational requirements]) E --> J([Dedicated Runners - Go-live preparations]) E --> K([...]) H --> L([Issue 1]) click A "https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/479" click E "https://gitlab.com/groups/gitlab-com/gl-infra/gitlab-dedicated/-/epics/276" click I "https://gitlab.com/groups/gitlab-com/gl-infra/gitlab-dedicated/-/epics/297" click J "https://gitlab.com/groups/gitlab-com/gl-infra/gitlab-dedicated/-/epics/298"
Note If you are not seeing the diagram, make sure that you have accepted all cookies.
Each epic has a single DRI who is responsible for delivering the project. DRIs for each epic are listed at the top of the description of each epic per Epic Structure. Epic DRI responsibilities are in https://handbook.gitlab.com/handbook/engineering/infrastructure/team/gitlab-dedicated/#epic-owner-responsibilities
The DRI needs to:
Throughout the project, the DRI should continue to adjust the epic description and structure to keep it current with the project status.
Each epic and child sub-epics must include the following:
Description (TBD make epic template)
Epic meta data
Labels are described in the epic label section.
Epic boards are used to track the overall status of epics. DRI’s are encouraged to created any project labels needed to allow a suitable project board to be created.
All significant changes must have a rollout issue using the rollout_coordination template. This issue should be used to review test coverage, rollout timings as well as plan customer communication where needed.
The project is generally following these stages:
Dedicated team projects use an agile-like methodology.
When creating a new issue:
# Ensure the correct parent epic.
/epic &XXX
# Label with a project label.
# /label ~"Hosted Runners for GitLab Dedicated"
/label ~"team::Environment Automation" ~"workflow-infra::Triage"
/confidential
Issue boards track the progress of all ongoing work.
On a single board, the goal is to get issues from workflow-infra::Triage into the workflow-infra::Done state. Each of the workflow labels have a special meaning described in the workflow labels section.
The status for all work relating to GitLab Dedicated is maintained in the description of the top-level GitLab Dedicated epic so that it is visible at a glance.
Both Engineering Cross-Functional DRIs should provide weekly updates for the DRI’s epics according to following process, which allows alignment with Project Management in SaaS Platforms:
By Wednesday at 21:00 UTC the DRI for a project is expected to update the status block in the epic description to:
Top-Level Epic Status Update automation synthesizes updates from status section from description of active epics to provide initiative status in the status section in the description of the top-level initiative Epic.
Weekly engineering/product sync at 16:30 UTC on Mondays Dedicated engineering/product meeting is used to discuss status updates and potential mitigations as necessary.
Status updates will be incorporated into initiative status updates and any initiative reporting in the following week.
Status updates are auto-generated and added to description of GitLab Dedicated top-level epic using a bot running the epic issues summary project.
If no update has been provided in an epic or issue for over a week, the issue will automatically receive workflow-infra::stalled label. Engineering managers are responsible for reviewing the status of the issue and helping it move along.
We provide reports on status of GitLab Dedicated to meet Top Cross-Functional Initiative requirements.
Prior to the start of a new quarter, the team will spend time refining the Epic backlog. This process will be led by the EM + PM, who will go through the Epics targeted for the upcoming quarter (according to the roadmap) and ensure each Epic contains the following information (pulling in different stakeholders to help fill in the details as necessary):
While the above information is being added, the Epic will move from 
to 
. Once the information has been finalized, the Epic will move to 
.
Having this set of refined epics will help us plan for the upcoming quarter and allow engineers to quickly get started on an Epic once it’s ready to be picked up during the quarter.
GitLab Dedicated team respects the Company principle of everything starting with a merge request.
Closes #<issue> or Related to #<issue> so that the merge request change is linked to the appropriate issue within GitLab.The MR approval rule settings for all projects should be:
Prevent approval by author On ✔️Prevent approvals by users who add commits On ✔️Prevent editing approval rules in merge requests Off ❌ for emergencies allowing us to act in good faith when absolutely necessaryRemove all approvals when commits are added to the source branch Off ❌We are prioritising reviewing documentation merge requests above all other ones, until further notice. Every merge request documenting acquired knowledge (or concluded discussion) has an impact beyond only the people working on the project directly. In the early stages of building the product, many stakeholders have a need to find actual information quickly, and that means that every line documented increases efficiency of people working on the project as well as people contributing to the project indirectly as we enable more direct self-service.
Having a passing pipeline (green build) on the default branch is very important. Failed pipeline (red build) causes delays in all MR’s in progress targeting the default branch, and more importantly, it can cause significant regressions if new MR’s are merged into it.
When a red build in the default branch is detected, the first course of action is to revert the MR that introduced failures. Reverting should be done even if it is more work. A couple of answers to the question “why?”:
GitLab Dedicated follows the same pattern for author/reviewer assignment as the standard GitLab practice, documented in the Code Review Guidelines documentation.
The process can be summarized as:
@gitlab-dedicated/environment-automation or @gitlab-dedicated/switchboard in the MR description to help with knowledge sharing.As the merge request author, please don’t mark discussions resolved until the reviewer has had a chance to respond. In general, if the reviewer has not yet approved the MR, and the thread is non-trivial, don’t mark their comments as resolved, let the reviewer review your response and resolve accordingly during the next round of view. If they have approved the MR, but comments remain unresolved, it’s generally fine to resolve comments before merging.
New Dedicated team members work with their manager to decide when to begin Maintainer training. Usually this will be around the third month in the team.
A Maintainer training issue will be created using the maintainer_training template and a training buddy will assigned to support the training.
After training is complete, the new Maintainer will be added to the Environment Automation Maintainers pool.
There are times when we are impacted by an upstream library bug. While waiting for the upstream library fix we need to apply a temporary workaround that fixes the bug or mitigates an incident.
To reduce the team’s cognitive load on having to keep a mental record of all the applied temporary workarounds, we use the following process to track temporary workarounds:
workaround::active to the issuecorrective action and link it to the workaround issuepyyaml bugCommonly used labels are:
team::Environment Automation.workflow-infra labels.component labels.cloud-provider labels.workaround labels.The team::Environment Automation label is used in order to allow for easier filtering of issues applicable to the team that have group level labels applied.
Epics and child epics should contain the following labels:
FY25-Q2workflow-infra labelworkflow-infra::In Progress, then a health status label should be applied. (health::on track, health::needs attention, health:at risk by the epic DRI. This label is regularly updated as part of status updates.)We leverage scoped workflow labels to track different stages of work.
In general, we want to see issues move from workflow-infra::Triage to workflow-infra::Ready stage to indicate that the submitted issue will go through for implementation. Once the issue is marked with workflow-infra::Ready, we are ready to work on the issue until we get the issue marked with workflow-infra::Done.
The standard progression of workflow is from top to bottom in the table below:
| State Label | Description |
|---|---|
|
Default label added to issues created. Issues with this label need to be confirmed as work we would consider. If we don’t want to consider the issue further, we mark it with workflow-infra::Cancelled and close it. If this issue does not need Product validation, and we are ready for implementation, issue is moved to workflow-infra::Ready. Otherwise, we move it to the next stage workflow-infra::Proposal. |
|
In this stage, proposal is being created and put forward for review with the rest of the team. Issues in this stage are also a part of Product validation workflow. If there are no further questions or blockers, the issue is supposed to be sufficiently refined and ready for implementation and can be moved into workflow-infra::Ready. The epics that encapsulate the implementation work for customer facing features must have a Product Manager sign-off before they can be moved to workflow-infra::Ready |
|
The issue is waiting to be picked up for work. |
 |
Issue is assigned to a DRI and work has started. |
 |
Issue is updated with the outcome of the work that was done, and this label is applied and issue closed. |
There are three other workflow labels of importance:
| State Label | Description |
|---|---|
 |
Work in the issue is being abandoned due to external factors or decision to not resolve the issue. After applying this label, issue will be closed. |
 |
If no update has been provided in an issue for over a week, the issue will get this label. The team Engineering Manager is responsible for reviewing the status of the issue and helping it move along. |
 |
Work is blocked due external dependencies or other external factors. Where possible, a blocking issue should also be set. After applying this label, issue will be regularly triaged by the team until the label can be removed. |
Scoped support labels are applied to the issues that are opened when a GitLab Support Engineer escalates a ticket for assistance using the “request for help” process. These requests are reviewed periodically by members of the GitLab Support team. The purpose of this review is to identify whether a request could have been deflected. These reviews primarily lead to updates to the GitLab Dedicated Support workflows and the GitLab docs.
| State Label | Description |
|---|---|
support::reviewed |
The support::reviewed label is applied when these issues have been reviewed and the review did not directly result in an issue or MR. |
support::reviewed-and-improvement-made |
The support::reviewed-and-improvement-made label is applied when an improvement has been made based on a review. Improvements include opened issues or MRs. |
To denote different types of components and services we are working with, we leverage component:: scoped labels. By using these labels, we are able to track the distribution of work across different components which also allows us to change focus where needed. These labels are created on the GitLab Dedicated group level, since they are focused on work required specifically for GitLab Dedicated.
NOTE We do not use Service:: labels given that service labels are used by GitLab SaaS related projects.
Component labels with their description can be found by searching prioritized labels.
These scoped labels are intended to distinguish generic work to everything made for a specific cloud provider.
| Cloud Provider Label | Description |
|---|---|
 |
Amazon Cloud specific implementation |
 |
Google Cloud specific implementation |
Scoped workaround labels are intended to track temporary workarounds applied to GitLab Dedicated tenant instances that are supposed to be removed once a permanent fix is available. These labels need to be added to the follow-up issues that describe the implementation of the permanent fix.
| Workaround label | Description |
|---|---|
 |
This label is applied to issues describing workarounds applied to tenant instances |
We operate a Capacity Planning rotation, which switches on a fortnightly basis amongst all on-call SRES, with the schedule managed in PagerDuty. While Capacity Planning should not require large effort most weeks, in the event of a Capacity Planning shift overlapping with an on-call shift, consider swapping your capacity planning shift with another engineer to ensure both tasks receive the necessary attention. The goal is to give ourselves the best chance of resolving impending saturation events before they become a customer-impacting incident It is based on statistical modeling and human interpretation, and is not expected to be perfect in every situation. Do your best, and understand that the process is inherently imprecise and fuzzy at the edges.
The Dedicated capacity process is built on top of Tamland.
The overall flow of work is to assess any new reported saturation risks, and re-review any which are due to be looked at again. If there is an apparent risk of saturation, initiate further assessment for potential remediation action, and actively manage any such ongoing issues that are assigned to you.
At the start of your shift review the handover issue from the prior shift and close it when you are up to speed.
At the start of each work week while you are on duty, as a high priority task that is second only to active incidents:
violation:hard is more important than violation:soft,
and severity:: provides additional signal.capacity-planning:investigate and investigate when you have dealt with higher priority capacity planning issuescapacity-planning::in-progress:
~capacity-planning:in-progress,capacity-planning::in-progress check on the remediation issue and ensure it is making progress,
and update the due-date to be 1 week in the future.
If remediation has completed,
move the issue to capacity-planning::verification
or, if results are already clearly sufficient, close it.capacity-planning::verification,
check if the remediation results can be considered sufficient,
and if so close the issue.
If not, update the due date to next week for further review.~capacity-planning::monitor
and update the due-date to just before the start of the next shift
for review by the incoming duty engineer.capacity-planning::tune-model,tenant:: labels.When your shift comes to an end, create a handover issue assign it to the incoming duty engineer and populate with any information that the incoming shift should know about. Let comments/discussions on the specific issues speak for mundane and routine matters, to be reviewed on their due date, but consider noting:
Remember to record updates and status comments on each capacity-planning issue when necessary or useful, just like you would on a team issue.
Some tips:
capacity-planning::monitor state is a good course of action in this situationcapacity-planning::monitor and moving its due date out by 2+ weeks.Resources used by the team to conduct work are described on the Development Resources Page.
Switchboard customer portal was suggested by @marin after he spent a day trying to figure out which mixing console (Also known as a switchboard/soundboard) to get for amateur music making. He didn’t buy anything, but the suggestion was accepted.Amp management cluster was suggested by @ccasella, as it is the instance that is “powering” supply of other instances.Environment Automation is a team within the Dedicated Group. Our mission is to develop and operate the automated plumbing of the GitLab Dedicated solution.
We follow the same processes as listed on the Dedicated Group, unless a difference exists which is explicitly noted on this page.
To engage with the Environment Automation team:
The architecture of GitLab Dedicated is maintained in GitLab Dedicated Architecture
Service Level Availability is the percentage of time that GitLab Dedicated (“Dedicated”) is in an available state for use by a customer over a given calendar month. This document defines how Service Level Availability is calculated for Dedicated.
GitLab calculates Service Level Availability based on the available state of certain services and features provided with Dedicated. The services and features that are considered when calculating Service Level Availability include:
Switchboard is a team within the Dedicated Group. Our mission is to empower external GitLab Dedicated customers to manage their tenant environments and reduce the operational overhead on the Environment Automation team so we can scale up the GitLab Dedicated offering. We follow the same processes as listed on the Dedicated Group, unless a difference exists which is explicitly noted on this page.
Product Manager: Loryn Bortins Technical Writer: Lysanne Pinto Product Designer: Divya Alagarsamy
The GitLab US Public Sector Services team mission is to create a fully managed, single-tenant GitLab environment, served through a GitLab Dedicated platform that is purpose-built to help address specific regulatory and compliance requirements of US government agencies at the federal, state, and local level, as well as contractors, educational institutions, and other U.S. customers that run sensitive workloads. It is developed to remove any manual interactions with customer tenant installations, and to ensure that the customer tenants are fully focused on unlocking the power of The One DevOps Platform.
0d0d57dc)
