Database Platform

Current GitLab.com Architecture

GitLab.com uses 4 PostgreSQL clusters running on virtual machines in GCP managed by Terraform, Chef and Ansible for complex tasks. With Cells, we want to create significant more than 4 PostgreSQL clusters, if we want 50 Cells, we’ll have to have 50+ PostgreSQL clusters.

@startuml
component "Rails"
component "Container Registry"

database "main"
database "ci"
database "registry"
database "embedding"

component "pgbouncer" as pgbouncermain
component "pgbouncer" as pgbouncerci
component "pgbouncer" as pgbouncerregistry
component "pgbouncer" as pgbouncerembedding

"Rails" --> pgbouncermain
pgbouncermain --> "main"

"Rails" --> pgbouncerci
pgbouncerci --> "ci"

"Rails" --> pgbouncerembedding
pgbouncerembedding --> "embedding"


"Container Registry" --> pgbouncerregistry
pgbouncerregistry --> "registry"


:User: --> "Rails"
:User: --> "Container Registry"
@enduml

We currently have 4 different database clusters, each hosting a single production database

The data will be sharded by the organization_id, and an organization_id can live in 1 Cell.

During the Cells rollout most of the data will remain in the current database clusters, so it is important to accommodate for the requirements of this larger monolithic systems as well.

Future Architecture

For the upcoming Cells project the requirements are different. The current estimate is that each Cell will host a relatively small number of users compared to our current monolith, posing the new challenge to host many small database clusters and to create them on short notice

The exact requirements will develop over time, but the following is requested:

• Creation and removal of database clusters via API, which will be part of the provisioning of a cell.
• Database clusters up to the size of our reference architecture, 50,000 users API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS.
• Up to 2,000 Cells and database clusters.

Our current automation involves many manual steps, including the creation of dedicated Chef roles for each cluster. The change process to create new cluster is the largest overhead, requiring multiple Merge Requests and Peer Review as well execution by Change Request. This is incompatible with the newly given requirements.

However, Cells will start iterative with one single Cell, needing one database cluster. The current platform can therefore be used at least for Cells 1.0, but could be kept until all data is stored in the new Cell local database clusters.

It is also possible to migrate the current data to a new database platform, before the full sharing across multiple smaller Cells. If this is desired, the requirements for the current monolith need to be meet by the target platform as well.

Requirements

Requirements will be split into what we currently need for the monolithic data stores for GitLab.com and what we estimate for the individual cells.

Requirements can be put into the following categories. They can be high, soft, low.

• high, is needed to operate GitLab.com.
• medium, is currently needed for operation, but could be replaced
• low, nice to have

Universal Requirements

Current Platform Requirements

Performance

Performance is a key factor. The first task is to define performance outlines and a reliable method to test if they are met.

{- TODO: Define performance requirements and test procedure -}

We observed the following load peaks and can use them as an indication:

• ~100k read-write TPS on the primaries
  • ~80k TPS on the Main primary
  • ~32k TPS on the CI primary
• ~1M read-only TPS distributed among the standbys
  • ~714k TPS on the Main standbys
  • ~150k TPS on the CI standbys

Cells Requirements

TODO: Define performance requirements and check with different steak holders. Discussed with @rnienaber, we will start with the largest reference architecture.

• Up to around 50.000 users, our largest reference architecture, Up to 50,000 users API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS
• This might increase in the future, or multiple hundred Cells are required to host all current users

Decomposition

The application data for GitLab.com is currently decomposed into two separate database clusters, Main and CI. We are evaluating if we can further decompose the Main database with decomposing `Secure- and Software Supply Chain Security-related tables to a separate Postgres DB to gain more headroom and scalability for the current platform.

For Cells it is a design choice to scale horizontally by adding more Cells and to rebalance by moving organizations to less saturated cells. Cells should not be scaled vertically to a point where decomposition is reasonable. Therefore decomposition within Cells will not be needed and is also not supported by the current Dedicated / GET / Cells tooling. Adding support for Decomposition in GET and Dedicated in the future would be possible as a medium-complexity task.

In line with this, only a single database cluster is provisioned per Cell, capable of hosting all needed databases.

Overview - Possible Solutions

Non-PostgreSQL Offerings

We briefly looked into non-PostgreSQL, claiming certain levels of PostgreSQL compatibility, like AlloyDB. We came to the conclusion that cost and risk are not in any meaningful proportion to the expected benefit. Currently, there is no inherent shortcoming of PostgreSQL itself that justifies further investigation.

Further details below:

• Alternatives To Postgres DB (Google Spanner vs AlloyDB)
• Switching to a proprietary database, instead of PostgreSQL

Cloud SQL

Cloud SQL is Google’s standard PostgreSQL offering. It is a custom fork, claimed to be 100% compatible to the upstream Releases and is simply referenced as PostgreSQL in the Cloud SQL documentation. GitLab currently recognizes Cloud SQL as a supported PostgreSQL implementation.

Most of the information above can be found in the official Cloud SQL documentation. Some estimates were given to us verbally by the GCP team, as seen in the meeting notes Discuss Cloud SQL and AlloyDB with GitLab (internal).

Things to validate

• Could we use the offered logical replication feature (pglogical) for our migration needs? - estimate 2-4 weeks
• Can we access WAL and base_backups, as it appears in the pitr documentation, in contrast to our meeting, where GCP denied it. - estimate < 1 week
• How long does a major upgrade take for our 50k reference architecture? - estimate 4-5 weeks
• Is Query Insights a sufficient replacement for the current observability tooling.
• How long does it take to create a read-replica? How long does it take to create a new cluster from backup? 10GB, 100GB, 1TB - estimate 1 week

k8s Operator

There are currently multiple available operators that seam feasible, but detailed evaluation and benchmarking to ensure our requirements are met is critical. One feature that might be lacking is zero downtime upgrades. Currently, we maintain our own automation for this as well and could adapt it until an operator provides this feature.

Things to validate

• Compare Candidates (4-5 weeks)
  • CYBERTEC PostgreSQL Operator (zalando with commercial support)
  • CloudNativePG

Amazon RDS PostgreSQL

Amazon Relational Database Services PostgreSQL is AWS’s managed database service offering fully compatible with PostgreSQL community version. In fact, Amazon only packs and deploys the PostgreSQL community binaries into the RDS instance underlying infrastructure. GitLab currently recognizes Amazon RDS PostgreSQL as a supported PostgreSQL implementation.

Things to validate

• We need to validate how to export RDS Performance Insights metrics into our Monitoring tools
• How to export PostgreSQL logs into Elastic
• Validate a connection pooling solution for both Write and Read-Only workloads
  • pgbouncer over VMs
  • RDS Proxy
• Compare Migration options
  • Native logical replication
  • Amazon Database Migration Services
• Evaluate time and impact of Major Version Upgrades 50k reference architecture?
  • Also evaluate Blue/Green deployments
• How long does it take to create a read-replica, or a new cluster from a backup? 10GB, 100GB, 1TB
• How long is the database service downtime with Blue/Green major version upgrade method?
  • We should also test if pgbouncer or RDS Proxy can hold requests to alleviate the impact during a Blue/Green deployment.

Evaluate changes over current Dedicated-RDS deployment

• Cost of Enable Enhanced Monitoring with small granularity ( < 10 seconds)
• Evaluate offload of read operations into Standby Replicas
• Evaluate “Enable auto minor version upgrade”
• Evaluate performance gain with “Dedicated Log Volume”
• Increase logging level: slow queries, temp usage, autovacuum, lock waits, connections/disconnections, DDL statements
• Configure pg_stat_statements settings
• Load and Configure auto_explain
• Implement “logical backup” solution
• Review Alarms: RDS Event Subscriptions, AWS Cloud Watch

Refactor Current Automation

The current platform to host all PostgreSQL clusters for GitLab.com is based mainly on Terrafrom and Chef, managing GCP virtual machines. Some complex operations are orchestrated via Ansible.

The main problems currently with this architecture is the historical growth and technical debt. It was designed to host a single, or small fixed number of database clusters. Hosting any arbitrary number of clusters is theoretically possible, but requires manual steps and overhead.

These inefficiencies are incompatible with the estimated requirements of hosting many database clusters, e.g. >100.

Some examples:

• For each database cluster we have to create multiple Chef roles
• For each database cluster we have to manually define a network subnet and write it in a configuration file
• By policy for each database cluster in production, we have an equivalent in staging

These shortcomings and bottlenecks could be removed by improving the automation with Chef or replacing it by a different tool set.

Cells 1.0

Cells 1.0 is the first iteration for Cells, which should become GA during 2024. This first iteration will only host a limited number of test users and projects and therefore does not have the full requirements. It poses an opportunity to learn and iterate on the database platform.

During the development the number of Cells can grow from 1 to up to 10. The estimate is that it will take multiple month until a second Cell will be created, and a second cluster is needed.

Cloud SQL currently does not fulfill the given high priority requirements, and there is no roadmap which fully includes them as well. We will use Cloud SQL temporary for Cells 1.0 despite that to unblock the Cells 1.0 time-line. Therefor we acknowledge and agree to the associated risks and bottlenecks.

• There will be downtime to migrate to the final platform!
• The quality of service will not be equal to GitLab.com.
  • There is no clear SLO for deep level support.
  • No major version upgrades with reasonable downtime.
• We can not grow reliable grow this solution to the target of 50k users, we see problems for 3k users regarding connection pooling already.

We prioritize finding a solution meeting the requirements, to make it available after Cells 1.0. The most promising candidates are the existing k8s PostgreSQL operators, offering PostgreSQL as a Service like experience.

Cells 1.x

As highlighted in Cells 1.0 we start with Cloud SQL as a temporary solution to free resource to build a sufficient platform for the long-term requirements.

The first phase will evaluate which k8s operators are available and which are the most fitting candidates for a full evaluation.

Current Database Platform

The current database platform is limited in future growth, but currently not at the edge of saturation. It is not clear when the database will become a bottleneck, but based on the prior experience from before decomposition a database with 200% size poses operational challenges.

There are currently no predictions how long it takes for our database to double in size, but it is fair to assume this will not happen in the next 24 month, until 2026.

This gives enough time to migrate significant data into cells, or to improve / change the database platform here as well.

View page source - Edit this page - please contribute.