Distribution

Overview

The Distribution team is composed of two subgroups: Distribution:Build and Distribution:Deploy

Build team is focused on producing artifacts including system packages, container images, and related components like marketplace listings along with tooling required to create and maintain them.

Deploy team is focused on installation and upgrade mechanisms to ensure smooth deployments. This includes system integration, scripting, templating, and related configuration management tooling.

In addition to product deliverables, both groups review a large number of MR’s authored outside the team. These include dependency and security updates along with configuration controls and other bundled components like PostgreSQL, Consul, Patroni.

Target Consumers

The primary user persona for Distribution are system administrators who are responsible for managing GitLab instances. Team goals are to make it as easy as possible to deploy, upgrade and configure GitLab on a range of on-prem and cloud platforms at a variety of scales.

Deployments include everything from single node deployments for evaluating GitLab all the way through the 50K user reference architecture and beyond. The primary goal is to ensure end users have a high-speed, low-friction experience when managing GitLab with limited downtime or sevice disruptions.

Omnibus packages, Helm Charts and Operators are primary deployment methods Distribution currently supports.

Distribution Build

Team | Charter

Responsibilities:

• Research for the support of new clouds, platforms, architecture, and components
• Access controls, permissions, and CVE patches
• Dependency updates
• License management
• Submissions to Partners for validations/certifications
• The install, update, and upgrade pages
• Build and own the infrastructure used for creating the various installation methods
• Maintaining infrastructure used in Distribution

Distribution Deploy

Team Charter

Responsibilities:

• Initial installation and composability for self-managed installations and GitLab.com
• Upgrades / Downgrades
• Scaling deployments
• Migration between platforms or providers
• Data lifecycle management
• Secure configuration & communication
• Research of clouds and platforms for integration to existing tools

Vision

Distribution team vision is shown below, and not limited to:

Technology

• GitLab has an official installation method on all major platforms and architectures.
• GitLab offers official one click installation method on all major cloud platforms.
• GitLab is able to automatically upgrade itself safely and reliably.
• Official repositories for any Kubernetes package repository.
• GitLab runs equally well on both high and low resource systems (such as Raspberry Pi).
• GitLab.com is running using the official installation methods.
• All GitLab features are installed and configured by default.
• Installation interface to set up and configure GitLab.
• Any GitLab installation is able to report installation/upgrade errors automatically.
• Setting up GitLab in HA configuration is automated and simple.
• All installation methods are automatically tested before release.
• Most frequently used configuration options are tested using end-to-end integration tests.

Team

• Each team member is able to work on all team projects along with the ability to focus on specific technologies.
• Each team member is a part of a hiring panel aimed to hire better than the best in the team.
• Team creates documentation to increase knowledge and awareness to support a self-service model.
• Team is able to reach a conclusion independently all the time, consensus most of the time.
• Team has official certifications for frequently used technologies and platforms.
• On-boarding and off-boarding is efficient.
• Career development paths are clear.

Mission

Distribution ensures the experience of installing and maintaining GitLab is easy and safe for everyone. It is Distribution team’s task to think about the widest variety of installation/update use-cases and provide a solution that will satisfy most needs. Distribution is there to provide the best possible experience for a user that is a novice but also a veteran when it comes to installing and maintaining software.

Distribution:Build Charter

Build team focus is ensuring GitLab components are tested, current, license compliant, and available for our users’ platforms and architectures. This group manages the build pipelines, researches support for new services, platforms, and architectures, as well as maintains existing ones. We strive to respond efficiently to build failures, security results, and dependency changes in order to ensure a safe reliable product for our users.

Distribution:Deploy Charter

Deploy team focus is configuration, deployment, and operation of GitLab as a whole product. The goal is to deliver an intuitive, clear, and frictionless installation experience, followed by smooth, seamless upgrade and maintenance processes for deployments of any scale. We strive to deliver ongoing operational behaviors for scaling, little to zero downtime upgrades, and highly reliable experiences for not only instance administrators but their users.

Goals

Increase # of active installations

Reduce average days behind latest version

Team members

Distribution Build Team

The following people are members of the Distribution:Build Team:

Name	Role
Peter Lu	Backend Engineering Manager, Distribution:Deploy

Distribution Deploy Team

The following people are members of the Distribution:Deploy Team:

Name	Role
Peter Lu	Backend Engineering Manager, Distribution:Deploy
Clemens Beck	Distribution Engineer, Distribution:Deploy
Hossein Pursultani	Senior Distribution Engineer, Distribution:Deploy
Jason Plum	Staff Distribution Engineer, Distribution:Deploy
João Alexandre Prado Tavares Cunha	Senior Distribution Engineer, Distribution:Deploy
Jon Doveston	Distribution Engineer, Distribution:Deploy
Lucas Li	Distribution Engineer, Distribution:Deploy

Stable counterparts

The following members of other functional teams are our stable counterparts:

Name	Role
Grant Young	Staff Software Engineer in Test, Core Platform:Distribution

Common links

• Distribution team issue tracker
• Slack chat channel

Team responsibility

In addition to the separate responsibilities for Build and Deploy. The Distribution team as a whole is responsible for:

1. The omnibus-gitlab installation package.
   • Installation using package is simple, quick, secure and protects the data integrity.
2. A cloud native installation method.
   • Installation using the Helm charts is able to scale easily with the increased demand.
3. Triaging issues and Reviewing Merge Requests in all owned projects.
4. Providing regular demonstrations of recent solutions or works in progress

Team objectives

Based on team responsibilities, the following objectives apply:

• Distribution:Build
  • Installation pages need to be complete, correct, easy, helpful, and attractive. Even if team’s primary skill is not design and UX, team can maintain the pages with the help of teams that have this skill as their primary.
  • Nightly builds are installed on dev.gitlab.org using one of the official artifacts of the Distribution Build team. Purpose of this exercise is to have a production level instance using one of the installation methods that the community will be using.
• Distribution:Deploy
  • The omnibus-gitlab installation package and the cloud native installation method should allow a beginner to install GitLab quickly, correctly and completely. Required configuration should be minimal, and it is the Distribution Deploy team’s responsibility to automate as much of this configuration as possible for the user.
  • The omnibus-gitlab installation package and the cloud native installation method should be sufficiently configurable to allow an advanced user a way of setting up a more complex GitLab architecture.
  • The above two goals might sound opposing, however they are not. A compromise can be made between these two goals without increasing the project maintenance cost for the Distribution team. We optimize for the best initial installation experience and then expand to further complexity.
• Distribution Common Objectives
  • Triaging issue tracker is a task that allows us to keep on the pulse of the changes we make. By being in contact with users and customers, we can maintain visibility into most frequently reported bugs or requested features. Not everything reported will be resolved, however all of the reports should be triaged. This also applies to mentions in GitLab CE/EE repositories on issues with Distribution and group::distribution label.
  • Every Distribution team member is responsible for creating a training session for the rest of the team. See the page on team training for details.
  • When the team that manages GitLab.com creates an issue, the item should be raised up directly to the team Engineering Manager and Product Manager. While these issues are important, we don’t necessarily need to provide a complete solution right away, but we need to work with the other team on providing a path forward with their request.

Primary Projects

omnibus-gitlab - This project creates platform-specific, self-contained GitLab packages and images for self-managed consumption in cloud environments and on-premisis hosting.

Cloud Native GitLab provides cloud native containers to deploy GitLab. These containers may be deployed and managed via Helm using GitLab Charts or GitLab Operator on Kubernetes, OpenShift, and Kubernetes compatible container platforms.

Omnibus GitLab project’s product outputs

graph TD
  subgraph Code
  OG --> PKG
  PKG --> DOI
  PKG --> AMI
  end

  subgraph Deploy
  DOI --> DOK
  DOK --> DOC
  AMI --> AWS
  PKG -.-> GET
  GET --> GCP
  GET --> AZURE
  GET -.-> |Future|AWS
  GET -.-> |Future|VMW
  end

  OG[Omnibus GitLab]
  PKG[Linux Package]
  DOI[Container Image]
  DOK(Docker)
  DOC(Docker Compose)
  AMI[Amazon Machine Image]
  GET[GitLab Environment Toolkit]
  AWS(AWS)
  GCP(Google Cloud Platform)
  VMW(VMWare)
  AZURE(Azure)

Cloud Native GitLab project’s product outputs

graph TD
  subgraph Code
  CNG --> HC
  CNG --> GOP
  HC --> GOP
  end

  subgraph Deploy
  GOP --> K8s
  GOP --> OS
  CNG --> DC
  HC --> K8s
  end

  CNG[Cloud Native GitLab containers]
  HC[Helm Chart]
  K8s(Kubernetes)
  GOP[GitLab Operator]
  OS(OpenShift)
  DC(Docker Compose)

All Projects

Working with the community

The install and upgrade process is one of the first features that systems administrators experience when working with GitLab. As a result, the projects managed by the Distribution team have a high level of engagement by the user-base. The GitLab community is made up of more than just code contributors; users logging issues and feature requests are constantly pushing us forward and helping create a better experience.

In Distribution we strive for the following in our public projects:

1. Uphold our Community Code of Conduct.
2. Enable GitLab’s mission that everyone can contribute..
3. Show our work in public.
4. Recognize and thank contributors for their work.
5. Respect contributors donated time by providing a timely review turnaround time.

Working with Open Source communities

The open core of GitLab is built on top of thousands of open source dependencies. These dependencies and their communities are important to the GitLab strategy, and working with these dependencies is an essential part of the projects the Distribution team maintains.

In Distribution we strive to:

1. Consider the impact of our work on the open source communities that we benefit from.
2. Promote the importance of these open source communities within GitLab.
3. Raise issue with any decision that goes against our stewardship promises.
4. Find opportunities to contribute back the changes we make.

Public by default

All work carried out by the Distribution team is public. Some exceptions apply:

• Work has possible security implications - If during the course of work security concerns are no longer valid, it is expected for this work to become public.
• Work is done with a third party - Only when a third party requests that the work is not public.
• Work has financial implications - Unless financial details can be omitted from the work.
• Work has legal implications - Unless legal details can be omitted from the work.

Some of the team work is carried out on our development server at dev.gitlab.org. Infrastructure overview document lists the reasons.

Unless your work is related to the security, all other work is carried out in projects on GitLab.com. If you need to submit a sensitive issue, please use confidential issues.

If you are unsure whether something needs to remain private, check with the team Engineering Manager.

YouTube Playlists

The team regularly publishes demos, discussions and meetings to these playlists:

• Distribution Team Demos (Public) More on team demos.
• Distribution Team Discussions (Mostly public, but some private content)
• Distribution Team Meetings (Private)

Onboarding and offboarding

In addition to general company on-boarding and off-boarding, Distribution team has its own process to get new team members up to speed more quickly.

If you are starting with your onboarding, open an issue in Distribution team issue tracker, select Team-onboarding template and assign the issue to yourself.

Going through the steps noted in the issue should be your top priority, higher than the general company on-boarding issue. This is because items in team on-boarding are specific to your role and it will allow you to get up-to-speed quicker.

Off-boarding should be carried out by the Engineering Manager of the team, using the appropriate issue template in the same issue tracker.

Work Resources

General resources available to developers are listed in the Sandbox cloud page.

In the Distribution team specifically, everyone should have access to the following resources:

• Google projects in Google Cloud Platform
  • testground
  • cloud-native
  • omnibus-build-runners
• AWS build infrastructure
  • Distribution group AWS sandbox account
  • cloud-native EKS cluster for CI (requires a maintainer to grant access)
  • GitLabTop account (To be retired, existing team members only)

If you don’t have access to any of these resources, create an Access Request and assign it to your manager for approval.

Infrastructure and maintenance

As part of the team responsibilities, team owns maintenance of infrastructure used for day to day work. For list of nodes and description of the maintenance tasks, see the infastructure and maintenance page.

Team workflows

General engineering workflow applies to the Distribution team. Since Distribution is working across multiple projects, our team workflow is further explained and summarized on the Distribution workflow page.

Further reading

The following important areas of the GitLab Handbook impact how we work and are worth reading.

• Distribution workflow page
• General engineering workflow page
• How to reinforce our values
• Continue to serve smaller users (internal only)
• Promises to our Open Source community
• How to follow our Product Principles
• Principles of our company strategy
• Effective & Responsible Communication Guidelines
• Test Platform in Distribution group

Work/life harmony

Working all-remote and asynchronous first offer flexibility in how team members approach their workday. Team members must make choices on how best to balance work time with other areas of life.

For new team members, the following resources provide examples on how to focus their time:

• How team members approach their day
• Blog post: A day in life of a remote worker
• The option of a non-linear workday
• GitLab handbook: Work/life Harmony

The following GitLab Handbook areas are key in maintaining a healthy work/life balance.

• Family and Friends First, work second
• Combating burnout, isolation, and anxiety in the remote workplace
• Recognizing Burnout

How to work with Distribution

Everything that is done in GitLab will end up in the supported installation methods that are distributed to users. While that sounds like the last link in the chain, it is one of the most important ones. This means that informing the Distribution team of a change in an early stage is crucial for releasing your feature. While last minute changes are inevitable and can happen, we should strive to avoid them.

We expect every team to reach out to the Distribution team before scheduling a feature in an upcoming release in the following cases:

• Reach out to Distribution Build when:
  • The change requires a new or an update on a gem with native extensions.
  • The change requires a new or updated external software dependency.
    • Also when the external dependency has its own external dependencies.
• Reach out to Distribution Deploy when:
  • The change adds, modifies, or removes files that should be managed by omnibus-gitlab. For example:
    • The change introduces new directories in the package.
    • The change introduces new files in previously not defined locations
  • The change requires a new configuration file.
  • The change requires a change in a previously established configuration.

To sum up the above list:

If you need to do install, update, make, mkdir, mv, cp, chown, chmod, do compilation or configuration change in any part of GitLab stack, you need to reach out to the Distribution team for opinion as early as possible.

This will allow us to schedule appropriately the changes (if any) we have to make to the packages.

If a change is reported late in the release cycle or not reported at all, your feature/change might not be shipped within the release.

If you have any doubt whether your change will have an impact on the Distribution team, don’t hesitate to ping us in your issue and we’ll gladly help.

Requesting a new component or replacing a component

Create a Deliverables Request in the Distribution team [issue tracker].

Requesting feedback on a feature change

For requesting feedback you can ping @gitlab-org/distribution in your issue or merge request.

When looking for a Distribution reviewer for a merge request you should use the process outlined in our merge request workflow for projects owned by the Distribution team. For changes that need Distribution review outside those projects, please ping @gitlab-org/distribution in the merge request.

Engaging Distribution for expertise in support

There are occasions where the experise of the Distribution team may be needed in support of a customer issue. When this does occur, the appropriate method of requesting our engagement is by opening an issue on the Distribution team tracker using the Support Request template. This process allows us to track time involved and ensure that the right parties are involved at the correct time.

Requests should be opened two or more business days before action is needed to ensure the team has time to prepare.

Trivia

How did Distribution get its name? We iterated, as always. “Distribution” was chosen as better than “Install” when renaming the original “Build” team, live on an AMA with our co-founder, Sid. Since then we have iterated further in order to grow the team, and now have subgroups for “Build” and “Deploy”.

Dashboards

View page source - Edit this page - please contribute.