Deployments and Releases

Overview and terminology

This page describes the deployment and release approach used to deliver changes to users. The overall process consists of two significant parts:

1. Monthly self-managed release: GitLab version (XX.YY.0) published every month. From this monthly release, planned patches are scheduled twice a month and unplanned critical patches are created as needed.
2. GitLab.com deployment: A Continous Delivery process to deploy branches created from master branch, on regular intervals.

For more details on the individual processes and how to use them please see the Deployments page for GitLab.com changes and the Releases page for changes for self-managed users.

The main priority of both deployments and releases is GitLab availability & security as an application running on both GitLab.com and for users running GitLab in their own infrastructure.

Deployment and Release Process overview

For testing purposes, all changes are deployed to GitLab.com before being considered for a self-managed release. Deployment and release cadences operate on different timelines with changes deploying to GitLab.com multiple times per day, and packages being released for self-managed users several times a month.

This overview shows how the two processes are connected:

• Diagram source

1. Engineer creates features or bug fixes. Changes reviewed by Maintainers
2. Validated changes merged into the default branch
3. A scheduled pipeline packages all new changes into an “auto-deploy package” for deployment to GitLab.com. Multiple packages are created each day at the listed times
4. If deployments are allowed the auto-deploy pipeline starts. Production Change Locks, unhealthy environments, or other ongoing deployments are examples of events that would prevent a deployment
5. The auto-deploy package is deployed to GitLab.com. For more details see the deployment process
6. Changes that have been successfully deployed to GitLab.com can be considered for packaged release for self-managed users. A new release candidate package is created for these changes
7. The release candidate is deployed to a test environment and automated QA tests execute
8. Release Candidate is officially tagged and published for release

For a more detailed explaination of the processes see the deployments page and the releases page

Release Managers

The overall coordination and operation of the deployment and release process is the responsibility of the release managers.

See the GitLab Release Managers schedule to find out who the current release managers are.

How to contact a Release Manager

You can contact the current Release Managers:

1. On GitLab issues and epics by using @gitlab-org/release/managers handle
2. On Slack by using the @release-managers handle

We use the #releases and #f_upcoming_releases channels to discuss and coordinate deployments and releases. Automated deployment status announcements are made to the #announcements channel.

If you need to escalate a request, please use the release management escalation process

Weekly Delivery Metrics Review

Each week, the current Release Managers walk through the key Delivery Group metrics in the EMEA/AMER Delivery Weekly sync (YouTube Playlist). The goal is to share experiences about recent deployments and releases, and for the Group to identify ways we can improve our tools and processes.

MTTP Monthly - Deployment blockers - Deployment SLO - GitLab: deployment frequency - GitLab: lead time

1. Walkthrough Auto-Deploy packages dashboard
2. Walkthrough the monthly view of GitLab: deployment frequency and GitLab: lead time - note any patterns
3. Walkthrough the Deployment SLO dashboard - Deployment SLO and packager pipeline duration
4. Walkthrough of Deployment Blockers Dashboard and last week’s Deployment Blockers
5. Do we need to take action based on the previous week’s MTTP?

Deployment Targets

This section defines the hierarchy of supported deployment types for GitLab. Note: other deployment types are possible, for instance FreeBSD Ports and Source Installs, but these are out-of-scope as they’re not officially supported. Since this is a hierarchical taxonomy, items can be represented with classifications at multiple levels in the tree.

In some cases, limitations in usage ping data may require broader (non-leaf node) classifications to be used, but in general, deployments should be categorized by the most specific classification available.

Taxonomy

This section describes the taxonomy of deployment types that distribute GitLab across. Where possible, we include the Service Ping attributes that can be used to match these deployment types.

1. SAAS ~deployment-type:saas

   SaaS represents all customer-facing GitLab-managed instances.

   1. GitLab.com ~deployment-type:saas-gitlab

      GitLab.com represents our two multi-tenant deployment types.

      1. Original GitLab.com ~deployment-type:saas-config-mgmt

         GitLab.com is our original monolithic deployment, configured via the config-mgmt project. Primarily uses CNG charts, and uses Omnibus for deploying Gitaly, Patroni and Postgres. No service ping data is sent.

      2. Cells ~deployment-type:saas-gitlab-cells

         Cells is the nascent organizational-sharded deployment, which will be progressively adopted. Cells uses the Dedicated tenant deployment tooling (Instrumentor), so in many ways is provisioned in a similar means to Dedicated.

      Service Ping Attribute	Value
      installation_type	gitlab-cloud-native-image
      gitlab_dedicated	true
      gitlab_environment_toolkit	true
      Autonomous System (ASN)	Google

   2. Dedicated ~deployment-type:saas-dedicated

      Dedicated is our single-tenant SaaS offering. Dedicated primarily uses GitLab Environment Toolkit (GET) Hybrid. In turn, GET relies on Omnibus (for Gitaly) and Cloud Native Charts (CNG) for application services. Where possible Cloud-Managed services are favoured, including Postgres (RDS, CloudSQL), Redis (ElastiCache, MemoryStore) and Kubernetes (EKS, GKE).

      Unlike GitLab.com, Dedicated operators do not have access to the GitLab Admin console, or any other direct access to the GitLab applications. These tenants are intended to be run at scale, so per-tenant configuration options are kept at a minimum, including feature flags and configuration options.

      1. Commercial ~deployment-type:saas-dedicated-comm

         Commercial Dedicated is deployed exclusively on AWS, although Google Cloud is available but unsupported commercially. Currently Commercial Dedicated is the only SaaS platform that is replicated using GitLab Geo.

      Service Ping Attribute	Value
      installation_type	gitlab-cloud-native-image
      gitlab_dedicated	true
      gitlab_environment_toolkit	true
      Autonomous System (ASN)	Amazon

      1. US Public Sector ~deployment-type:saas-dedicated-uspubsec

         Deployed on AWS GovCloud.

         Service Ping Attribute	Value
         installation_type	gitlab-cloud-native-image
         gitlab_dedicated	true
         gitlab_environment_toolkit	true
         Autonomous System (ASN)	AWS GovCloud

2. Self-Managed ~deployment-type:sm

   The Self-Managed classification covers all GitLab instances deployed and operator by our customers.

   1. Cloud-Provider Deployed ~deployment-type:sm-cloud

      By “cloud-provider”, we’re specifically referring to the three cloud providers supported in GET: AWS, Azure and Google Cloud. Other cloud-providers are classified as on-premise. It’s important to distinguish cloud-provider-hosted environments from other self-managed/on-premise environments because these environments provide a relatively homogeneous environment, support managed services for PostgresSQL (RDS, CloudSQL etc), Redis (ElastiCache, MemoryStore, etc), Kubernetes (EKS, GKE, etc) and other resources such as Key Management Services (important for deploying secrets management tooling).

      1. GitLab Environment Toolkit (GET) Provisioned ~deployment-type:sm-get

         GitLab Environment Toolkit targets cloud environments in AWS, Google Cloud and Microsoft Azure. All GET deployment targets are included in the reference architectures, providing guidelines for number of nodes, and appropriate sizing of nodes.

         1. GET Omnibus VM-based ~deployment-type:sm-get-omnibus

            All services are deployed onto VM instances using Omnibus. Customers can choose between Omnibus-managed VMs or use Cloud-Managed alternatives.

            Service Ping Attribute	Value
            installation_type	omnibus-gitlab
            gitlab_dedicated	false
            gitlab_environment_toolkit	true

         2. GET Cloud-Native ~deployment-type:sm-get-cng

            Any GET environment which is deployed partially (Hybrid) or fully using Kubernetes. At present, we’re unable to distinguish Hybrid from Full Cloud Native using Service Ping data.

            Service Ping Attribute	Value
            installation_type	gitlab-cloud-native-image
            gitlab_dedicated	false
            gitlab_environment_toolkit	true

            1. Hybrid ~deployment-type:sm-get-hybrid

               GET will only deploy stateless services (Sidekiq, Webservice, etc) in Kubernetes. Gitaly is deployed to VMs through Omnibus and customers can choose between Omnibus-managed VMs for Postgres and Redis, or use Cloud-Managed alternatives.

               This deployment is dogfooded by the GitLab Tenant Deployment tooling (Instrumentor).

            2. Full Cloud-Native ~deployment-type:sm-get-cng-full

               GET will deploy only stateless services (Sidekiq, Webservice, etc) in Kubernetes, and Gitaly in Kubernetes. Stateful services (Redis, Postgres) must be provisioned with cloud managed services. ops.gitlab.net dogfoods this deployment type.

      2. Self Provisioned ~deployment-type:sm-cloud-sp

         These are cloud-deployed GitLab instances that do not use GET for provisioning cloud resources. Ideally, we should be encouraging these customers to migrate to GET for a more consistent experience, but for a variety of reasons, they have not. The primary difference between these customers and on-premise is that these customers have the option of using managed services for Postgres, Redis, Kubernetes, KMS etc. whereas on-premise customers do not.

         1. Cloud Omnibus ~deployment-type:sm-cloud-omnibus

            Customer is deployed in the Cloud, using Omnibus, but not using GET.

            Service Ping Attribute	Value
            installation_type	omnibus-gitlab
            gitlab_dedicated	false
            gitlab_environment_toolkit	false
            Autonomous System (ASN)	Recognized Cloud

            1. Omnibus Single-Node ~deployment-type:sm-cloud-omnibus-single

               GitLab is deployed to a single node/VM-instance using Omnibus. All application services are deployed via Omnibus. Customer may be using some managed services, such as RDS. dev.gitlab.org is deployed using this approach.

            2. Omnibus Multi-Node ~deployment-type:sm-cloud-omnibus-multi

               GitLab is deployed to multiple VMs using Omnibus. No mechanism is provided for coordination and configuration across each node in the multi-node cluster: this is left up to the customer, and could be done manually, or with Chef, Ansible or SaltStack.

         2. Kubernetes CNG Helm Charts ~deployment-type:sm-cloud-sp-cng

            GitLab is deployed to a customer-provisioned and managed Kubernetes cluster. Like ~deployment-type:sm-get-cng, Postgres and Redis are not provided and must be provisioned separately by the customer. Customers may or may not be using the GitLab Operator.

            Service Ping Attribute	Value
            installation_type	gitlab-cloud-native-image or gitlab-operator-2
            gitlab_dedicated	false
            gitlab_environment_toolkit	false
            Autonomous System (ASN)	Recognized Cloud

   2. On-Premise ~deployment-type:onprem

      On-Premise (“on-prem”) in this document refers to non-cloud-based deployments, or deployments to clouds other than AWS, Google Cloud or Azure. Generally, on-premise deployments are unable to use any managed services for Postgres, Redis, Kubernetes, etc. Historically, Omnibus has provided a means of configuring these services, or users can bring their own. For some proposed new services, such as Secrets Management, on-premise brings unique challenges in that PKCS11-compatible hardware may need to be integrated due to the lack of cloud KMS.

      1. On-Premise Omnibus ~deployment-type:onprem-omnibus

         Customer is not deployed in a recognized ASN Cloud, using Omnibus.

         Service Ping Attribute	Value
         installation_type	omnibus-gitlab
         gitlab_dedicated	false
         gitlab_environment_toolkit	false
         Autonomous System (ASN)	Unrecognized

         1. Omnibus VM-based single-node ~deployment-type:onprem-omnibus-single

            Customer can bring their own Postgres, Redis, but in many cases, these nodes operate with local Omnibus-managed copies of these services.

         2. Omnibus VM-based multi-node ~deployment-type:onprem-omnibus-multi

            Customer must build their own configuration tooling to deploy GitLab (e.g. Ansible, Chef, Saltstack, etc). We encourage customers to follow the reference architectures to choose the number of nodes and sizes of nodes.

      2. Kubernetes CNG Helm Charts ~deployment-type:onprem-cng

         Customer provides their own Kubernetes cluster, own Redis, and own Postgres. Gitaly-on-Kubernetes support is immature, but available.

         Service Ping Attribute	Value
         installation_type	gitlab-cloud-native-image or gitlab-operator-2
         gitlab_dedicated	false
         gitlab_environment_toolkit	false
         Autonomous System (ASN)	Unrecognized

Resources

View page source - Edit this page - please contribute.