Cells
This is the handbook page for the Cells project. Cells is one of the top priorities for FY2025, with the goal of providing additional scalability for GitLab.com. This handbook page contains the project information such as the project plan, roadmap, workstreams, DRIs, stakeholders, and communication channels. It also has links to important documentation such as the Cells design blueprints.
Intro
Cells is a new architecture for our software as a service platform. This architecture is horizontally scalable, resilient, and provides a more consistent user experience. It may also provide additional features in the future, such as data residency control (regions) and federated features.
For more information about the goals of Cells, see goals.
Requirements and Architecture
Cells overall architecture blueprint.
Roadmap, Workstreams, and DRIs
Roadmap
|
Cells 1.0
|
Cells 1.5
|
Cells 2.0
|
- For internal customers only
- Organizations are private
- Users cannot interact with other Organizations (including GitLab Org)
- Groups and projects are private in the Organization
- For more details, see Organizations on Cells 1.0
|
- For existing/new customers of GitLab.com
- Organizations are private
- Existing users can interact with private Organizations on Secondary Cells
- Groups and projects are private in the Organization
- For more details, see Organizations on Cells 1.5
|
- Organizations are public or private
- Users can interact with other Organizations
- Groups and projects are private or public in the Organization
- For more details, see Organizations on Cells 2.0
|
DRIs and Stakeholders
|
|
Role |
Responsibility |
|
Sabrina Farmer
|
Executive Sponsor |
|
|
Marin Jankovski
|
Senior Director of Engineering |
|
|
Chun Du
|
Director of Engineering |
- Liaison between project team and cross-functional engineering leaders
- Coordinating temporary staffing arrangements within the Data Stores stage
|
|
Nick Nguyen
|
Senior Engineering Manager |
- Coordinating staffing and unblocking groups in Data Stores
- Drive cross-functional efforts in engineering
- Report on Data Stores progress and mitigate risks
|
|
Sissi Yao
|
Tenant Scale Engineering Manager |
- Status updates of Tenant Scale workstreams
- Mitigate risks
- Collaborate with Tenant Scale Product Manager on Organizations and Cells projects
|
|
Joshua Lambert
|
Director of Product Management |
- Investment and staffing of Core Platform teams
- Liaison between project team and cross functional product managers and product leaders
- Escalation of product priorities competing with Cells
- Decision maker for supported and un-supported features for each iteration of Cells
|
|
Christina Lohr
|
Tenant Scale Product Manager |
- Product definition, requirements, roadmap for Organization workstream within Tenant Scale
- Product definition, requirements, roadmap for Cells workstreams within Tenant Scale
- Point of contact to collaborate with product managers from other teams
- Investment and staffing of Tenant Scale
|
|
Darby Frey
|
Staff Fullstack Engineer, Expansion |
DRI of Expansion Software Development
|
Workstreams
Cells 1.0
All Cells 1.0 work is tracked under the Cells 1.0 Epic.
The Epic is split into multiple phases where each one represents a iteration to achieve Cells 1.0.
Some of these phases have dependencies over one another, and some can be run in parallel.
Phase 1: PreQA Cell
Exit Criteria:
- New GCP organizations created.
- Break glass procedure.
- Ring definition exists.
- Cell provisioned using dedicated stack.
- Able to do configuration changes to Cell.
- Cell available at
xxx.cells.gitlab.com.
- Cell doesn’t handle data uniqueness.
source
Unblocks:
- Phase 3: To provision runway deployment for Topology Service
- Delivery team: Start testing deploys on rings
Dependencies:
Epic:
Exit Criteria:
- 100% of API traffic goes through router using passthrough proxy rule.
- 100% of Web traffic goes through router using passthrough proxy rule.
- 100% of Git HTTPS traffic goes through router using passthrough proxy rule.
- Requests meet latency target
- registry.gitlab.com not proxied.
source
Unblocks:
- Phase 3: Router to be configured with additional rules in phase 3.
Dependencies:
Epic:
Exit Criteria:
- PreQA Cell configured to generate
_gitlab_session with prefix using rails config.
- Route
_gitlab_session with matching prefix to PreQA Cell using TopologyService::Classify (REST only) with static config file.
- Continuous Delivery on Ring 0 with no rollback capabilities and doesn’t block production deployments.
- Topology Service Readiness Review for Experiment
- Topology Service gRPC endpoint not implemented.
Unblocks:
Before/After:
source
Dependencies:
- Phase 2: Passthrough proxy needs to be deployed.
- Phase 1: GCP organizations, Ring definition exists.
Epic:
Exit Criteria:
- Framework to generate routable tokens in Rails.
- Framework to classify routable tokens in HTTP Router.
- Topology Service being able to classify based on more criteria.
- Route Personal Access Tokens to different Cells using TopologyService::Classify.
- Support
PRIVATE-TOKEN: and Authorization: HTTP headers for Personal Access Tokens, create issues for other to be solved in following phases.
- Each routing rule added should be covered with relevant e2e tests.
- Route Job Tokens and Runner Registration to different Cells using TopologyService::Classify.
Before/After:
source
Epic:
Communication
Slack Channels
Meetings
Status updates
Cells Fast Boot 2024
We held a Cells Fast Boot in Dublin, Ireland, between 2024-04-23 and 2024-04-24. Below are the artifacts from the event.
Agenda, Slides, and Videos
Please use the Unfiltered Google account to watch video recordings.
- Main agenda (internal only)
- Introductions, overview, and logistics: Agenda (internal only)
- Cells Services - Global Service: Agenda (internal only), Slides (internal only), Video (internal only)
- Cells Services - Routing: Agenda (internal only), Slides (internal only), Video (internal only)
- Application Readiness - Organizations and Users: Agenda (internal only)
- Application Readiness - Dependencies and OKR alignments: Agenda (internal only)
- Deployment: Agenda (internal only), Slides (internal only), Video (internal only)
- Provisioning: Agenda (internal only)
- Observability and Runners: Agenda (internal only)
- Security: Agenda (internal only), Slides (internal only), Video (internal only)
- Disaster Recovery: Agenda (internal only), Slides (internal only), Video (internal only)
- Cells Mover and Isolation: Agenda (internal only)
- Scalability Headroom and Timeline: Agenda (internal only)
Decisions
- No external customers on Cells 1.0, internal dogfooding only. Cells 1.x is the target to onboard new or existing external customers.
Artifacts
- Day 1 recording: Part 1 (internal only), Part 2 (internal only)
- Day 2 recording (internal only)
- Database breakout recording (internal only)
- Organizations breakout recording (internal only)
Cells is a project that spans the entirety of GitLab. Instead of recreating feature testing done by the other teams, we will reuse and leverage what exists currently and supplement to fill in gaps.
This approach has the following requirements:
It must feed back useful information to the engineering teams in an efficient, non burdensome way It must provide good coverage so we have confidence to release It must be easy to add/enhance/change tests It works with our current process Strategy The testing strategy for Cells follows our practice of testing at the correct level.
