Cells
Status | Authors | Coach | DRIs | Owning Stage | Created |
---|---|---|---|---|---|
ongoing |
ayufan
fzimmer
DylanGriffith
lohrc
tkuah
|
ayufan
|
lohrc
|
devops data stores | 2022-09-07 |
This document is a work-in-progress and represents a very early state of the Cells design. Significant aspects are not documented, though we expect to add them in the future.
Cells is a new architecture for our software as a service platform. This architecture is horizontally scalable, resilient, and provides a more consistent user experience. It may also provide additional features in the future, such as data residency control (regions) and federated features.
For more information about Cells, see also:
Cells Iterations
- The Cells 1.0 target is to deliver a solution for internal customers using the SaaS GitLab.com offering, and foundational work for Cells.
- The Cells 1.5 target is to deliver a migration solution for existing and new enterprise customers using the SaaS GitLab.com offering, built on top of the Cells 1.0 architecture.
- The Cells 2.0 target is to support a public and open source contribution model in a cellular architecture.
Goals
See Goals, Glossary and Requirements.
Technical proposals
The Cells architecture has long lasting implications to data processing, location, scalability and the GitLab architecture. This section links all different technical proposals that are being evaluated.
- Cells Services:
- HTTP Routing Service
- SSH Routing Service
- Topology Service
- Planned: Indexing Service
- Feature Flags
- Cluster wide unique sequences
- Cells: Infrastructure
- Organization migration
Impacted features
The Cells architecture will impact many features requiring some of them to be rewritten, or changed significantly. Below is a list of known affected features with preliminary proposed solutions.
- Cells: Admin Area
- Cells: Backups
- Cells: CI/CD Catalog
- Cells: CI Runners
- Cells: Container Registry
- Cells: Contributions: Forks
- Cells: Data Migration
- Cells: Explore
- Cells: Git Access
- Cells: Global Search
- Cells: GraphQL
- Cells: Organizations
- Cells: Personal Access Tokens
- Cells: Personal Namespaces
- Cells: Secrets
- Cells: Snippets
- Cells: User Profile
- Cells: Your Work
Impacted features: Placeholders
The following list of impacted features only represents placeholders that still require work to estimate the impact of Cells and develop solution proposals.
- Cells: Agent for Kubernetes
- Cells: Data pipeline ingestion
- Cells: GitLab Pages
- Cells: Group Transfer
- Cells: Issues
- Cells: Merge Requests
- Cells: Project Transfer
- Cells: Router Endpoints Classification
- Cells: Schema changes (Postgres and Elasticsearch migrations)
- Cells: Uploads
- …
Frequently Asked Questions
What’s the difference between Cells architecture and GitLab Dedicated?
We’ve captured individual thoughts and differences between Cells and Dedicated over here
The new Cells architecture is meant to scale GitLab.com.
The way to achieve this is by moving Organizations into Cells, but different Organizations can still share server resources, even if the application provides isolation from other Organizations.
But all of them still operate under the existing GitLab SaaS domain name gitlab.com
.
Also, Cells still share some common data, like users
, and routing information of Groups and Projects.
For example, no two users can have the same username even if they belong to different Organizations that exist on different Cells.
With the aforementioned differences, GitLab Dedicated is still offered at higher costs due to the fact that it’s provisioned with dedicated server resources for each customer, while Cells use shared resources. This makes GitLab Dedicated more suited for bigger customers, and GitLab Cells more suitable for small to mid-size companies that are starting on GitLab.com.
On the other hand, GitLab Dedicated is meant to provide a completely isolated GitLab instance for any Organization. This instance is running on its own custom domain name, and is totally isolated from any other GitLab instance, including GitLab SaaS. For example, users on GitLab Dedicated don’t have to have a different and unique username that was already taken on GitLab.com.
Can different Cells communicate with each other?
Not directly, our goal is to keep them isolated and only communicate using global services.
How are Cells provisioned?
The GitLab.com cluster of Cells will use GitLab Dedicated tooling to create instances. Once this instance gets provisioned it could join the GitLab.com cluster and become a Cell. One requirement will be that the instance does not contain any prior data.
To reach shared resources, Cells will use Private Service Connect.
See also the design discussion.
What is a Cells topology?
See the design discussion.
How are users of an Organization routed to the correct Cell?
TBD
How do users authenticate with Cells and Organizations?
See the design discussion.
How are Cells rebalanced?
TBD
How can Cells implement disaster recovery capabilities?
TBD
How do I decide whether to move my feature to the cluster, Cell or Organization level?
By default, features are required to be scoped to the Organization level. Any deviation from that rule should be validated and approved by Tenant Scale.
The design goals of the Cells architecture describe that all Cells are under a single domain and as such, Cells are invisible to the user:
- Cell-local features should be limited to those related to managing the Cell, but never be a feature where the Cell semantic is exposed to the customer.
- The Cells architecture wants to freely control the distribution of Organization and customer data across Cells without impacting users when data is migrated.
Cluster-wide features are strongly discouraged because:
- They might require storing a substantial amount of data cluster-wide which decreases scalability headroom.
- They might require implementation of non-trivial data aggregation that reduces resilience to single node failure.
- They are harder to build due to the need of being able to run mixed deployments. Cluster-wide features need to take this into account.
- They might affect our ability to provide an on-premise like experience on GitLab.com.
- Some features that are expected to be cluster-wide might in fact be better implemented using aggregation techniques that use trusted intra-cluster communication using the same user identity. For example, user Profile is shared across the cluster.
- The Cells architecture limits what services can be considered cluster-wide. Services that might initially be cluster-wide are still expected to be split in the future to achieve full service isolation. No feature should be built to depend on such a service (like Elasticsearch).
Will Cells use the reference architecture for up to 1000 RPS or 50,000 users?
See reference architecture for up to 1000 RPS or 50,000 users.
The infrastructure team will properly size Cells depending on the load. The Tenant Scale team sees an opportunity to use GitLab Dedicated as a base for Cells deployment.
Decision log
- ADR-001: Routing Technology using Cloudflare Workers
- ADR-002: One GCP Project per Cell
- ADR-003: One GKE Cluster per Cell
- ADR-004: One VPC per Cell, with Private Service Connect for internal communication between Cells
- ADR-005: Cells use Flexible Reference Architectures
- ADR-006: Use Geo for Disaster Recovery
- ADR-007: Cells 1.0 for internal customers only
- ADR-008: Cluster wide unique database sequences
- ADR-009: Initial Cell Sizes
- ADR-010: HTTP Router uses static rules and HTTP-based caching mechanism
- ADR-011: Cell Specific Configuration
Links
- Internal Pods presentation
- Cells Epic
- Database group investigation
- Shopify Pods architecture
- Opstrace architecture
- Adding Diagrams to this blueprint
f1b69711
)