Application Deployment with a Cellular Architecture

This page contains information related to upcoming products, features, and functionality. It is important to note that the information presented is for informational purposes only. Please do not rely on this information for purchasing or planning purposes. The development, release, and timing of any products, features, or functionality may be subject to change or delay and remain at the sole discretion of GitLab Inc.
Status Authors Coach DRIs Owning Stage Created
nolith skarbek devops platforms 2024-01-09

Disclaimer: This blueprint requires more cross-functional alignment - Confidence Level: Low

This blueprint describes a deployment strategy that can support the new scaling dimension introduced by the Cell Architecture.

The complexity of this transition will require many teams in the Platforms section to participate by taking ownership of the features necessary to reach the production grade rating on this architecture.

Introduction

Preamble

From a high level perspective, a Cell Cluster is a system made of only 3 items:

  1. Router - An HA routing system deployed independently from the GitLab application.
  2. Legacy Cell - The existing GitLab.com infrastructure.
  3. Zero or more Cells - GitLab installations authoritative for a limited number of Organizations. Those Cells are deployed using GitLab Dedicated tools.
  4. Topology Service - An HA system used for Cluster-wide uniqnesss and classification of resources.
@startuml

actor User

cloud router {
component Router as R
}

component "Legacy Cell" as Legacy
component "Topology Service" as TopologyService
collections "Cells" as Cells

User ==> R
R ==> Legacy
R ==> Cells

Cells --> TopologyService
Legacy --> TopologyService
R --> TopologyService

@enduml

As we can see from the diagram, users interact with the system through the router only. Cells communicate with the Topology Service and have a local copy of all the database rows necessary to operate.

It is important to note that even if Cell supports GitLab Geo out of the box, we will not be able to provide this feature to our users until the Router supports it.

Key Terms

  • Deployment - The GitLab application and its components being installed into infrastructure
  • auto-deploy version - The active version that creates a package viable for deployment
  • ring - A logical partition of the cell cluster. In order to deploy to the next ring a package must be validated inside the current ring.
  • perimeter - the ring marking the “definition of done” for Release Managers, a package validated inside the perimeter is allowed to rollout in the rest of the fleet
  • graduated version - The version deemed safe to deploy to cells outside of the perimeter
  • .com - Our old existing or currently running infrastructure
  • Legacy Cell - The existing GitLab.com infrastructure.
  • Cell(s) - GitLab installation(s) authoritative for a limited number of Organizations. Deployed using GitLab Dedicated tools.

Ring deployment

The scale of the Cell project deployment together with the strong user partitioning maps well with a ring deployment approach.

@startuml

skinparam frame {
  borderColor<<perimeter>> red
}

left to right direction

frame "Ring 3" as r3 {
  component "Cell4" as c4
  component "Cell5" as c5
  component "Cell6" as c6
  component "Cell7" as c7
  component "Cell8" as c8
  component "Cell9" as c9

  frame "Ring 2" as r2 {
    component "Cell1" as c1
    component "Cell2" as c2
    component "Cell3" as c3

    frame "Ring 1" <<perimeter>> as r1 {
      frame "Ring 0" as r0 {
        component "Canary stage" <<legacy>> as cny
        component "QA Cell" as QA

        note as ring0_note
            Ring 0 goes in parallel with canary
            QA tests executed on **canary and QA Cell**
        end note
      }

      component "Main stage\nLegacy Cell" <<legacy>> as Legacy

      note as perimeter_note
      The perimeter marks the definition of done for an auto_deploy package.
      When post-deployment migrations are executed inside the perimeter,
      the package is ready to be pulled by the outer rings
      **outside of the release managers coordinator pipeline**
      end note
    }

    note as baking_areas
     A package cannot rollout to the next ring before it is successfully
     installed inside the current ring.
    end note
  }
}

@enduml

In the image above we are showing a possible ring layout with a cluster made of the Legacy Cell and 10 Cells, the upper bound of the Cell 1.0 milestone.

The general rule is that:

  1. The deployment process progresses from Ring 0 to the outer rings.
  2. Rings are a collection of Cells sharing the same risk factor associated to a deployment.
  3. Deployments can get halted at any stage and the package will not reach the outer rings.
  4. We define the “perimeter” ring that marks the “definition of done” for the Release Managers.
    • Crossing perimeter is the logical point in time of a given package lifecycle after the PDM has successfully run on the Main Stage. Effectively, between Ring 1 and Ring 2 as described throughout this document.
    • A successful run of the Post Deploy Migrations inside the perimeter marks a package as graduated.
    • A graduated package is a valid candidate for the monthly release.
    • A graduated package is rolled out to the rest of the rings automatically.
    • Deployments must be automated: inside the perimeter are responsibility of Release Managers, outside of it are responsibility of Team:Ops.

Application changes lifecycle

In this section, we describe how new packages are deployed in the new infrastructure, explaining the interactions and hooks with the existing tooling and processes.

GitLab.com deployment process is described in great detail in the handbook, we are going to make some simplifications here to reduce the cognitive load to what is necessary to talk about ring deployments.

Today’s process

Today GitLab.com deployments consist of two main processes:

  1. The release-tools coordinator pipeline for auto-deploy packages is responsible for sequencing deployments and QA through staging and production environments. The timeline of this process depends on the available release managers, but it usually happens ten times a day.
  2. The Post Deployment Migrations pipeline executes the post-deployment migrations of the current package running in production, completing the quality assurance of the deployed changes. This process is usually executed only once a day because it marks a point where it is no longer possible to rollback an environment.
flowchart TB
    subgraph release_tools [Release-tools coordinator pipeline]
        direction LR
        gstg_cny[GSTG\n Canary] --> gstg_cny_qa[QA]
        gstg_cny_qa --> gprd_cny[GPRD\n Canary]
        gprd_cny --> gprd_cny_qa[QA]
        gprd_cny_qa --> baking_time[1hr\n baking time]
        baking_time --> promotion[▶️\n manual\npromotion]
        promotion --> gstg[GSTG\nMain] --> gstg_qa[QA]
        promotion --> |delay 30m|gprd[GPRD\nMain]
    end
    pkg((Auto-deploy\npackage\ntagging)) --> release_tools
    subgraph pdm [Post Deployment Migrations]
        direction LR
        pdm_gstg[GSTG] --> pdm_gstg_qa[QA] --> pdm_gprd[GPRD]
    end
    %% invisible link to help content layout
    release_tools ~~~ pdm

A package can be released to self-managed customers only after running its post deployment migrations.

Co-existence of the legacy infrastructure and Cells

For some time the existing (legacy) deployment and Cells will co-exist. Initially our ring deployment implementation will only consist of three rings:

  • Ring 0: hosting a QA Cell and eventually other experimental builds, this ring is kept in sync with the legacy production canary stage.
  • Ring 1: an empty placeholder for the legacy production main stage. As we will be able to begin extracting existing users from the legacy deployment, this ring could host ourselves, our free users, and more in general every organization interested in having a fast track to our new features.
  • Ring 2: hosting our first cell for customers.

Because of the nature of our release process, those first rings will be controlled by the existing release-tools tooling.

flowchart LR
    subgraph release_tools [Release-tools coordinator pipeline]
        direction LR
        gstg_cny[GSTG\n Canary] --> gstg_cny_qa[QA]
        gstg_cny_qa --> gprd_cny[GPRD\n Canary]
        gprd_cny --> gprd_cny_qa[QA]
        gprd_cny_qa --> baking_time[1hr\n baking time]
        baking_time --> promotion[▶️\n manual\npromotion]
        promotion --> gstg[GSTG\nMain] --> gstg_qa[QA]
        promotion --> |delay 30m|gprd[GPRD\nMain]
    end
    subgraph pdm [Post Deployment Migrations]
        direction LR
        pdm_gstg[GSTG] --> pdm_gstg_qa[QA] --> pdm_gprd[GPRD]
    end
    subgraph Ring Deployment
        direction LR
        ring_0[Ring 0] --> ring_0_qa[QA]
        ring_1[Ring 1]
        ring_2[Ring 2]

        %%% invisible link to help content layout
        ring_1 ~~~ ring_2
    end
    pkg((Auto-deploy\npackage\ntagging)) --> release_tools
    %% ring deployment hooks
    gstg_cny_qa --> ring_0
    ring_0_qa --> baking_time
    pdm_gprd ---> |pkg graduation|ring_2
    promotion --> |delay 30m| ring_1
Future - Only Cells

For the sake of completeness we are also describing a potential future scenario where the legacy infrastructure is decommissioned and all of our users are migrated to a new cell.

This is subject to change as the project evolves, as an example, the future of our staging environment will likely be affected by the development of Cells, but here we leave it in place to reduce the scope of this document.

flowchart LR
    subgraph release_tools [Release-tools coordinator pipeline]
        direction LR
        gstg_cny[GSTG\n Canary] --> gstg_cny_qa[QA]
        baking_time[1hr\n baking time] --> promotion[▶️\n manual\npromotion]
        promotion --> gstg[GSTG\nMain] --> gstg_qa[QA]
    end
    subgraph Post Deployment Migrations
        direction LR
        pdm_gstg[GSTG] --> pdm_gstg_qa[QA] --> pdm_gprd[Ring 1]
    end
    subgraph Ring Deployment
        direction LR
        ring_0[Ring 0] --> ring_0_qa[QA]
        ring_1[Ring 1]
        ring_2[Ring 2] --> ring_3[Ring 3] --> ring_N[Ring N]
    end
    pkg((Auto-deploy\npackage\ntagging)) --> release_tools
    %% ring deployment hooks
    gstg_cny_qa --> ring_0
    ring_0_qa --> baking_time
    pdm_gprd ---> |pkg graduation|ring_2
    promotion ---> |delay 30m| ring_1

The changes from the co-existing scenario are the following ones:

  1. New rings exist after Ring 2. The progression of packages is coordinated by the ring-deployment engine enforcing the gates on each ring.
  2. The release-tools coordinator pipeline only manage rings, there is no longer a concept of canary and main stage in production.
  3. The post deployment migration pipeline controls the execution of the migrations in Ring 1 (Subject to further discussion that are not necessary at this stage of the project).

Reference materials

Goals and Non-Goals

Goals

  • Limit the increase in cognitive load for the release manager; in doing so, we defined the perimeter as the clear handover point where a package is no longer a release manager’s responsibility.
  • Limit the blast radius of failures by partitioning the cell cluster into rings, automated validation occurs between each ring.
  • Ensure deployments are reliably automated
  • Ensure automatic handling of failed deployments
  • Provide observability into package rollouts and deployments

Non-Goals

  • Extending release-tools to take ownership of the Cell Application Deployments. A smaller, more specific piece of software will allow us to keep the tooling focused on one job.
  • Introduce major changes related to Release Management
  • Lifecycle management of Cells
  • Management of routing traffic to/from Cells
  • Individual component deploys

Requirements

Before we can integrate Cells to our deployment pipeline, we need a few items immediately:

  1. The router should exist, it must be HA, and have an independent deployment pipeline
    • This is required for appropriate testing. As noted below, we’ll need a QA cell to direct a deployment to for which QA will execute tests against. A router will need to route QA tests to the appropriate Cell.
  2. Assets Deployment
    • This already exists today for .com. Today this is handled via HAProxy, but with Cells, the routing layer will become the responsible party to redirect assets in a similar fashion.
    • If assets are chosen to be managed differently, this changes both how Delivery need to deploy said assets in order to provide as close to Zero-Downtime Upgrades as possible, and configuration to the Cell installation to support routing to assets properly.
  3. Feature Flags
    • We are assuming that the current Feature Flags workflows and tooling will just work on the Legacy Cell and that Cells will not be affected.
    • The use of feature flags to mitigate incidents is limited to only the Legacy Cell.
    • Tooling may need to mature to ensure that Cells do not drift for long periods of time with feature flags. This ensures that customers have a similar experience if their work expands across Cells and that we as operators of .com need not worry about version drift and the implications of code differing behind the feature flag.
    • Further guidance, documentation will need to be developed for this area. Engineers shouldn’t care what cell an organization is a part of. Thus Feature Flag toggles abstract away the need for engineers to care.

Proposed plan of action

From a delivery perspective not much changes between the 3 proposed Cells iterations (1.0, 1.5, and 2.0). The split is an iterative approach based on cutting the scope of the features available for Organizations bound to a given Cell. From a deployment point of view, it should be possible to have multiple Cells from the first iteration so we have to figure out a roadmap to get there that is independent from the Cell architecture version.

Iterations

Cells 1.0

The intent in this iteration is to focus our efforts on building and integrating our own tooling that builds and manages Cells. The following milestones, and their exit criterion, are a collaborative effort of the Platforms section and spans across many teams.

  1. The Dedicated technology stack expansion:
    • Instrumentor and AMP support GCP
    • A cell is defined as a reference architecture in Instrumentor
  2. Control Plane for Cells - Cell Cluster Coordinator
    • Switchboard is currently leveraged by Dedicated but is not an appropriate tool for Cells. We should evaluate the capabilities of other tooling created by Dedicated, amp and instrumentor, to determine how they could be integrated into a deployment workflow.
    • Implement Cell deployment converging the entire infrastructure of the cell (current dedicated capability)
    • Implement the concept of Rings: initially only Rings 0 and 2
  3. First Cell: the QA Cell in Ring 0
    • Build integration with our current tooling to perform deployments to the QA cell via the Coordinator
    • The QA Cell runs it’s own QA smoke tests
    • The QA Cell is updated in parallel with the production canary stage: QA cell failures are considered soft and do not block auto_deploy
  4. Control Plane for Cells - Individual dashboards and alerting
    • observability is at least on par with the legacy infrastructure
    • alerting is at least on par with the legacy infrastructure
  5. First Customer Cell: Ring 2
    • release-tools can graduate a package after the PDM execution
    • the Coordinator can manage Ring 2 deployments
  6. Support for multiple Cells
    • the Coordinator can converge multiple cells in the same Ring to the desired version
  • Limitations:
    • all Cells will be in the same ring, Ring 2
    • Rollbacks are possible but require downtime to achieve on all Cells

Cells 1.5 and 2.0

The following features can be distributed between Cell 1.5 and 2.0, they are all improving the operational aspects and we should prioritize them as we learn more about operating Cells.

  1. Control Plane for Cells - Additional rings
    • Cells can be spread over multiple rings
    • Deployment to the next ring starts automatically after the current ring converged
    • Emergency brake: ability to block package rollout to the next ring
  2. The QA Cell becomes a blocker for auto-deploy
  3. Control Plane for Cells - Cluster dashboards and alerting
    • A dashboard should indicate what package is expected for any given Cell and Ring deployment
    • Any cell not running the desired version should be easily visible and alert if not converged in a reasonable amount of time
    • Deployment health metrics to block package rollout inside a ring (z-score on the four golden signals?)
  4. The Post Deploy Migration (PDM) step of deployments needs to be segregated from the application deployment to ensure we have the ability to perform rollbacks on Cells.
    • Without this capability, a Cell must suffer downtime in order for a rollback to complete successfully. This is disruptive and should not be considered a wise solution.
    • The separation of the PDM on the Legacy Cell already functions as desired. Thus our Legacy Cell will have rollbacks as an option to mitigate incidents.
  5. Modified tooling that enables us to target only Deploying the GitLab application. Currently the destined tooling to be leveraged employs a strategy where the entire installation is converged. This includes the infrastructure and the version of GitLab which creates a lengthy CI pipeline and long running jobs.
  6. Automated Rollbacks - if a deployment fails for any reason, a rollback procedure should be initiated automatically to minimize disruption to the affected Cell. We should be able to use a health metric for this.

The focus here is productionalizing what has been built and cleaning up areas of tech debt incurred during the MVP stage of the first iteration.

Mindmap

%%{init: {'theme':'default'}}%%
mindmap
  root((Cells Deployment))
    Core concepts 📚
      Current .com infra is the Legacy Cell
      It is possible to have multiple Cells in Cell 1.0
      Cell isn't an HA solution
      Cells talks to the Topology Service

    Prerequisites 🏗️
        router
            HA solution
            independent deployment
        Dedicated
            Support GCP
            Cell reference architecture

    Decisions 📔
        Ring style deployment
            Ring 0: Current Canary + a new QA Cell
            Ring 1: Current Main stage
            Ring 2+: New Cells for customers
            The Perimenter
                Marks the handover point between Release Managers and Team:Ops
                Inside: Ring 0 and 1
                Outside: Ring 2+
        Running PDM inside the perimeter graduates a package
            A graduated package is a valid candidate for the monthly release
        We are not porting staging to Cell
            A new QA Cell will validate the package without affecting users

    Procedures needing upgrades 🦾
        Rollbacks
        auto-deploy rollout
        Post Deployment Migration
        Deployment health metrics

    Risk area ☢️
        There is no dogfooding in Cell 1.0  and 1.5

Deployment coordinator and Cell Cluster Coordinator

In the context of auto deploy we have an external coordinator pipeline, inside the release-tools project, that takes care of orchestrating package generation and rollout invoking the specific tool for each job.

In today’s GitLab.com infrastructure, deployments are executed by specific tools (deployer and gitlab-com/k8s-workloads) that can be independently operated by SRE and Release Managers, with the introduction of the Cell cluster we will face new operational challenges like a simple cluster overview, package rollout status, feature flag configuration, provisioning and deprovisioning.

The GitLab Dedicated stack features its own method of controlling installs of GitLab, primarily through a slew of tools, Switchboard, Amp, and Tenctl. The use of Switchboard is not geared towards Cells and thus cannot be leveraged. Other tooling such as Instrumentor and Amp may have a place or modifications to enable them to be more portable for usage between both the Dedicated team and Cells. We’ll need to evaluate these tools, their interactions with Cells, and how we may leverage them. Pending how the work is scheduled, this may be a highly collaborative effort with team members working closely across team boundaries to ensure requirements are met during the initial period or MVP for Cells.

In this paragraph we describe an ideal interaction where a data store is updated with a desired version to be deployed, and a Cell Cluster Coordinator is created to support Cell deployments.

In Cell 1.0, inside the perimeter, we will have a single Cell, the QA Cell. We should expand release-tools to command some-tool to perform a Cell update on demand.

@startuml
participant "release-tools" as rt
participant "Cell Cluster Coordinator" as sb
participant "AMP Cluster" as AMP
collections "Cells" as cells
participant QA

rt -> sb: update QA Cell
note right: In parallel with canary stage rollout
sb -> AMP: schedule deployment job
AMP -> cells: QA Cell: version rollout
cells --> AMP
AMP --> sb
sb --> rt
rt --> QA: test QA Cell
note over rt,QA: It does not replace canary QA
QA -> cells: run tests
cells --> QA
QA --> rt
@enduml

As we mentioned before, when we run post-deployment migrations in Ring 1, release-tools will mark that version as graduated and thus be capable to rollout outside of the perimeter.

Cell Cluster Coordinator will be leveraged to help coordinate automated version upgrades to further rings with automated checks before and after deployments to ensure we are deploying to the correct cells of a desired ring and validate instances are healthy before and after deployments, rolling back in the face of failure, and alerting the appropriate teams as necessary.

@startuml
participant "Cell Cluster Coordinator" as sb
participant "AMP Cluster" as AMP
collections "Cells" as cells

loop
sb -> sb: pull ring-version mapping
opt version mismatch
sb -> AMP: schedule configure job
AMP -> cells: CellX: version rollout
cells --> AMP
AMP --> sb
opt the ring now run the same version
sb -> sb: promote version to the next ring
end
end
end
@enduml

Procedures

Auto-Deploy

Auto-deploy shall continue to work as it does today as our Legacy Cell is equivalent to our legacy .com infrastructure. Thus our existing procedures related to auto-deploy can still be continued to be leveraged. Think hot-patching, rollbacks, auto-deploy picking, the PDM, the existing auto-deploy schedule, etc. A new procedure will be added to ensure that release-tools knows to trigger a deployment after a PDM is executed to the next Ring. Currently release-tools doesn’t understand anything related to Ring Deployments, this is functionality that will need to be added.

  • Auto-deploy is limited to Rings 0 and 1:
    • Ring 0 contains a QA Cell plus the canary stage of the .com infra
    • Ring 1 contains main stage of the .com infra - this is the cut off for release tools
    • All cells will deploy the same way; this eliminates needing to deal with differing deployment technologies
    • release-tools will interact with the Coordinator to pilot the deployments to Ring 0 as part of its coordinator pipeline
  • Release-tools must be able to graduate a package:
    • A graduate version of GitLab is any auto-deploy version which has a successful deploy onto the Main Stage of Production and the Post Deploy Migration (PDM) has completed.
    • This could mean we expect to see a single package deploy each day to our Cells. Currently, the PDM is only run 1 time per day. Note that there are exceptions to this rule.
    • This will enable us to use our existing procedures to remediate high severity incidents where application code may be at fault.
    • We do not want to run official released versions of GitLab as these are produced far slower than auto-deploys thus we risk missing SLA’s on incident response. In the cell architecture, most issues should be found in the Legacy Cell and fixed prior to being deployed to any Cell.
    • We’ll need new procedures, runbooks, and documentation such that when a problem is found through manual testing, we have some ability to halt deployments of what may be labeled a graduated package from actually being deployed.
    • It would be wise to track these failure cases as realistically, QA should be finding issues to enable us to run a automated deployments.

Note that currently, some smaller components deploy themselves to the .com infrastructure. Notably, Zoekt, Container Registry, and Mailroom, have their own cadence of providing newer versions to .com. This aspect will not be carried over into Cells, as currently, the tooling we’ll leverage does not allow a segregation of components to enable this functionality. Instead, we’ll rely on the current defined versions as specified in the default branch which built the auto-deploy package. This mimics how our releases are accomplished and thus should carry over well with Cells.

Rollbacks

Long term, we should aim to modify the deployment tooling such that Cells are provided a grace period to enable each of them to be able to be safely rolled back in the event of a deployment failure, or mitigating a failure that is noticed post deployment. Currently for the legacy .com or the Legacy Cell, we hold the PDM to execute 1 time per day at the discretion of Release Managers. The tooling that performs deployments to Cells currently do not have a way to NOT run the PDM, thus no there does not currently exist a way to rollback without inducing downtime on a particular Cell. Procedures and tooling updates will be required in this area.

Hot patching

Hot patching is one source of our ability to mitigate problems. If we rely on graduate versions, the hot patcher has no place for Cells. It could still be leveraged for our Legacy Cell, however. Though, it would be wise if we can eliminate hot patching in favor of safer deployment methodologies.

For reference, we’ve only hot patched production 1 time for year 2023.

Deployment Health Metrics

Currently we do not automate a deployment to the Main stage of the .com legacy infrastructure, or the Legacy Cell. In order to reduce operational overhead we should be able to rely on existing metrics which form a health indicator for a given installation and automatically trigger a deployment at the appropriate time. This deployment health indicator would also need to be carried into each of our cells. Tooling that triggers a deployment at various rings should be made aware to continue or halt a deploy given the status of earlier rings and the health state of the next target ring.

Feature Flags

Feature Flags are discussed in data-stores#83.

Package Rollout Policy

We have an implicit procedure driven by our current use of auto-deploys. This will become more prominent with Cells. As implied in various formats above, auto-deploy shall operate relatively similarly to how it operates today. Cells becomes an addition to the existing release-tools pipeline with triggers in differing areas. When and what we trigger will need to be keenly defined. It is expected that Cells only receive graduated versions of GitLab. Thus, we’ll leverage the use of our Post Deployment Migration pipeline as the gatekeeper for when a package is considered graduated. In an ideal world, when the PDM is executed successfully on the Legacy Cell, that package is then considered graduated and can be deployed to any outer ring. This same concept is already leveraged when we build releases for self managed customers. This break point is already natural to Release Managers and thus is a good carry over for Cell deployments.

We should aim to deploy to Cells as quickly as possible. For all Cells that exist in a single ring, we should have the ability to deploy in parallel. Doing so minimizes the version drift between Cells and reduces potential issues. If the version drifts too greatly, auto-deploy shall pause itself and an investigation into the reason why we are too far behind begins. Ideally we know about this situation ahead of time. We should aim to be no greater than 1 graduate package behind our PDM. Thus the expectation is that for every PDM, is a deployment to our Cells, every day. There are days which the PDM is skipped. We’ll need to evaluate on a case-by-case basis why the PDM is halted to determine the detriment this will incur on our Cell deployments.

Rings outside of the perimeter are self-managed by the orchestration engine. Once release-tools graduates a package it can forget about it. The orchestration engine will converge the desired GitLab version to all Cell in Ring 2, the first ring outside of the perimeter, and move to next ring only when all Cells converged.

FAQ

Will Developers see indicators on MR’s as they are deployed to various Cells?

No. Our current labeling schema is primarily to showcase that the commit landed in production, the PDM successfully executed, which signals to us that the observed commit is safe for being placed in a release for self-managed customers. Being that after we reach this point, issues with a package should be minimal, there’s no need to update issues/MR’s with the status as we move forward into our many Rings of deployments. Developers should not need to care what version is deployed to what Cell.

A P1/S1 issue exists, how do we mitigate this on Cells?

Cells are still a part of .com, thus our existing bug and vulnerability SLA’s for remediation apply. We can deploy whatever we want to Cells so long as it’s considered graduated. If a high priority issue comes about, we should be able to freely leverage our existing procedures to update our code base and any given auto-deploy branch for mitigation, and maybe after some extra rounds of testing, or perhaps a slower roll out, we can deploy that auto-deploy package into our cells. This provides us with the same mitigation methods that we leverage today. The problem that this causes is that there could exist some code that may not have been fully vetted. We can still rely on rollbacks in this case and revisit any necessary patch for the next round of auto-deployments and evaluate the fix for another attempt to remediate our cells.

What changes are expected from a Developers perspective

Release and Auto-Deploy procedures should largely remain the same. We’re shifting where code lands. Any changes in this realm would increase the most the closer we are to Iteration 2.0 when various environments or stages to GitLab begin to change.

All tiers but one have a failed deploy, what triggers a rollback of that package for all cells?

This depends on various characteristics that we’ll probably want to iterate on and develop processes for. Example, if we fail on the very first cell on the first Tier, we should investigate that cell, but also ensure that this is not systemic to all cells. This can only be handled on a case-by-case basis. If we reach the last tier and last cell and some failure would occur, there should be no reason to rollback any other cell as enough time should have passed by for us to catch application failures.

What happens with self-managed releases?

Theoretically not much changes. Currently we use Production, or .com’s Main Stage as our proving grounds for changes that are destined to be releasable for self-managed. This does not change as in the Cellular architecture, this notion for this exists in the same place. The vocabulary changes, in this case, a graduated package is now considered safe for a release.

What happens to PreProd

This instance specifically tests the hybrid installation of a GitLab package and Helm chart when we create release candidates. It’s our last step prior to a release being tagged. This is not impacted by the Cells work. Though we may change how preprod is managed.

What happens with Staging

Staging is crucial for long term instance testing of a deployment alongside QA. Hypothetically staging could completely go away in favor of a deployment to Tier 0. Reference the above Iteration 3 {+TODO add proper link+}

What happens to Ops

No need to change. But if Cell management becomes easy, it would be prudent to make this installation operate as similar as possible to avoid overloading operations teams with unique knowledge for our many instances.

This same answer could be provided for the Dev instance.

Last modified October 16, 2024: Misspellings - Replace "ther" with "the" (89226592)
View page source - Edit this page - please contribute. Creative Commons License