Sec DB Decomposition Working Group

Attributes

Exit Criteria

The charter of this working group is to:

• Successfully decompose the Sec datasets to a separate gitlab_sec database in order to reduce pressure on the primary GitLab.com DB and assist in future scalability and stability concerns.
• Consider the timing, scope, and impact of the decomposition related to prioritization and implementation of additional efforts to support GitLab.com db performance and optimization for related tables - OKR (GitLab internal)
• Evaluate the impact of the decomposition on Self-Managed instances regarding feature parity, performance/hardware requirement, improvements for different size of DBs, and admin’s effort to support.
• Provide an effective migration guide and/or tooling to assist Self-Managed instances in the decomposition of their local CI and Sec databases in alignment with GitLab.com

Objectives

Key results we’d like to achieve within the scope of the working group to ensure the outcome has the most desireable outcome.

Glossary

Overview

There is high impetus within GitLab to reduce pressure on the primary GitLab database server. The Database and Scalability teams have been taking a variety of steps to mitigate the ongoing pressure on the database server to maintain the growth and stability of GitLab in the long term. One such endeavour is Cells, however, there is desire to provide further mitigation in the short to medium term. Decomposition of the Sec dataset from the primary database was identified as a strong possible solution, similar to how the CI decomposition aided in this regard in the past.

Decomposition of the Sec dataset is a significant engineering effort due to the magnitude of the data interactions related to these features. The domain accounts for 25% of all database write traffic, and is only set to grow as we expand our feature set and grow our customer base. Further statistics and technical details can be found on the associated epic.

As this has become a scalability and stability concern for all of GitLab.com, as well as significantly constraining the ability of the Stages in the Sec section to implement new features due to continuously growing performance concerns, it is necessary to form an organised effort to effectively achieve this project.

We have the benefit of being able to lean heavily on the prior art and experience of the database-scalability working group who decomposed the CI database to achieve this goal. However, some key challenges we may face is the scale of the existing Sec codebase, and the need to maintain ongoing operations with (no/minimal) disruption to our customer base. A full GitLab.com downtime is heavily disfavoured due to our uptime SLA agreements with customers, but the scale of our operations may mean that some processes for this kind of decomposition may not be feasible.

Benefits

1. Reduce write pressure on the GitLab.com primary Write database in advance of Cells 1.5
2. Improve stability of GitLab operations, by isolating the primary database from Sec feature pressure
3. General performance improvement for both the Core and Sec feature sets due to seperation of concerns.
4. Improve iteration speed of Sec feature development without significant concern for compromising stability of the platform.

Risks

1. Significant developer commitment for a currently unknown duration.
2. Increased database maintenance requirement for the new decomposed database and it’s associated replicas.
3. Possibly unable to be delivered prior to the delivery of Cells 2.0
4. May require a full downtime of GitLab.com, which may be difficult to arrange with our customers.

Interdependencies

Sec Data has a high degree of integration with CI and standard GitLab data, such as Users, Projects and Namespaces. The past CI decomposition has succesfully delinked query interdependency of the associated CI dataset, however, significant effort will be necessary to do the same between the core GitLab dataset and Sec functionality.

Timeline

The group has determined that a gradual rollout is not advisable since it doesn’t relieve pressure on the main database cluster, and increases the amount of work to be done. As such, all slices will be rolled-out simultaneously to gitlab.com.

Progress

gantt
dateFormat YYYY-MM-DD
title 50% confidence timeline

section Work
Gitlab Decomposition Ready :active , decompose, 2024-07-01, 2025-02-14
Non-Slice Work :active, nonslicework, 2024-07-15, 2025-02-14
Slice 1 :active, slice1, 2024-07-23, 2025-01-13
Slice 2 :active, slice2, 2024-08-06, 2024-12-30
Slice 3 :active, slice3, 2024-07-15, 2025-02-14
Gitlab Application Ready for Decomposition :milestone, allslices, after slice1 slice2 slice3 nonslicework, 0d
Phase 1 & 2 : phase12, 2024-09-11, 16w
Phase 4 : phase4, after allslices phase3 decompose, 3w
Phase 5 : phase5, after phase4, 3w
Phase 6 : phase6, after phase4, 3w
Phase 7 : phase7, after phase6, 4w
Rollout complete :milestone, rollout, after phase7, 0d
axisFormat  %Y-%m

Source.

Decomposition

Last update: 2025-01-13.

Plan

1. Introduce separate gitlab_sec schema
2. Introduce gitlab_sec database connection (defaulting to fallback to using gitlab_main database)
3. In parallel, begin decomposition of foreign keys and cross-database transactions following the loose order of SBOM, Security, and Vulnerability code boundaries. For each slice perform the following breakdown:
   1. Migrate tables with low referentiality (few foreign keys)
   2. Migrate tables with higher referentiality (many foreign keys)
   3. Identify and allowlist cross-joins to be addressed
   4. Identify and allowlist cross-database transactions to be addressed
   5. Remove previously identified cross-joins and cross-database transactions allowances
4. Formulate a logical replication path for the safe migration of the Sec dataset to a new physical database.
5. Open Change Request to migrate tables using a single replication event for all tables in scope of decomposition
6. Update documentation around migrating self-managed instances to multiple databases

Data Migration Proposal

See rollout for full details

1. With physical-to-logical replication we replicate the full DB before converting to logical replication for the relevant sec tables
   1. Deploy the decomposed database instance as a streaming replica of main
   2. Begin replicating the full Sec data to the new database instance
   3. Establish a separate sec DB connection pointed at the same main DB
   4. Write the necessary code to enable GitLab.com to begin utilising the new DB connection generically and for all Sec features.
   5. As this is a potentially risky operation, ensure production snapshots are ready and that customers are sufficiently informed of potential problems or dataloss in the event of failure.
   6. Begin testing transition of the Sec featureset to using the new database instance as it’s new primary (gstg -> canary -> grpd)
   7. If successful, globally rollout usage of the decomposed database for the full featureset.
2. Truncate legacy sec tables from the GitLab main database.

Roles and Responsibilities

Related Performance Projects

1. Tuple Reduction
   • Brian Williams (DRI)
   • Fabien Catteau
   • Michael Becker
2. Vulnerability Management Application Limits and Vulnerability Management Retention Policy
   • Mehmet Emin Inaç (DRI)
   • Joey Khabie
3. Cells 1.0
   • Subashis Chakraborty (DRI)

Useful References

Thanks

Much information and inspiration and experience is being enjoyed from the database-scalability working group who accomplished the succesful decomposition of the CI database.

View page source - Edit this page - please contribute.