Secure Govern Decomposition Working Group

Attributes

Exit Criteria

The charter of this working group is to:

• Successfully decompose the Secure/Govern datasets to a separate gitlab_sec database in order to reduce pressure on the primary GitLab.com DB and assist in future scalability and stability concerns.
• Consider the timing, scope, and impact of the decomposition related to prioritization and implementation of additional efforts to support GitLab.com db performance and optimization for related tables - OKR (GitLab internal)
• Evaluate the impact of the decomposition on Self-Managed instances regarding feature parity, performance/hardware requirement, improvements for different size of DBs, and admin’s effort to support.
• Provide an effective migration guide and/or tooling to assist Self-Managed instances in the decomposition of their local CI and Secure/Govern databases in alignment with GitLab.com

Objectives

Key results we’d like to achieve within the scope of the working group to ensure the outcome has the most desireable outcome.

Glossary

Overview

There is high impetus within GitLab to reduce pressure on the primary GitLab database server. The Database and Scalability teams have been taking a variety of steps to mitigate the ongoing pressure on the database server to maintain the growth and stability of GitLab in the long term. One such endeavour is Cells, however, there is desire to provide further mitigation in the short to medium term. Decomposition of the Secure/Govern dataset from the primary database was identified as a strong possible solution, similar to how the CI decomposition aided in this regard in the past.

Decomposition of the Secure/Govern dataset is a significant engineering effort due to the magnitude of the data interactions related to these features. The domain accounts for 25% of all database write traffic, and is only set to grow as we expand our feature set and grow our customer base. Further statistics and technical details can be found on the associated epic.

As this has become a scalability and stability concern for all of GitLab.com, as well as significantly constraining the ability of the Secure/Govern stages to implement new features due to continuously growing performance concerns, it is necessary to form an organised effort to effectively achieve this project.

We have the benefit of being able to lean heavily on the prior art and experience of the database-scalability working group who decomposed the CI database to achieve this goal. However, some key challenges we may face is the scale of the existing Secure/Govern codebase, and the need to maintain ongoing operations with (no/minimal) disruption to our customer base. A full GitLab.com downtime is heavily disfavoured due to our uptime SLA agreements with customers, but the scale of our operations may mean that some processes for this kind of decomposition may not be feasible.

Benefits

1. Reduce write pressure on the GitLab.com primary Write database in advance of Cells 1.5
2. Improve stability of GitLab operations, by isolating the primary database from Secure/Govern feature pressure
3. General performance improvement for both the Core and Secure/Govern feature sets due to seperation of concerns.
4. Improve iteration speed of Secure/Govern feature development without significant concern for compromising stability of the platform.

Risks

1. Significant developer commitment for a currently unknown duration.
2. Increased database maintenance requirement for the new decomposed database and it’s associated replicas.
3. Possibly unable to be delivered prior to the delivery of Cells 2.0
4. May require a full downtime of GitLab.com, which may be difficult to arrange with our customers.

Interdependencies

Secure/Govern Data has a high degree of integration with CI and standard GitLab data, such as Users, Projects and Namespaces. The past CI decomposition has succesfully delinked query interdependency of the associated CI dataset, however, significant effort will be necessary to do the same between the core GitLab dataset and Govern/Secure functionality.

Timeline

The group has determined that a gradual rollout is not advisable since it doesn’t relieve pressure on the main database cluster, and increases the amount of work to be done. As such, all slices will be rolled-out simultaneously to gitlab.com.

Progress

gantt
dateFormat YYYY-MM-DD
title 50% confidence timeline

section Work
Decompose tables :active , decompose, 2024-07-01, 2024-12-09
Slice 1 :active, slice1, 2024-07-23, 2024-12-09
Slice 2 :active, slice2, 2024-08-06, 2024-10-04
Slice 3 :active, slice3, 2024-07-15, 2024-12-09
Table decomposition complete :milestone, allslices, after slice1 slice2 slice3, 0d
Phase 1 & 2 : phase12, 2024-09-11, 7w
Phase 3 : phase3, after phase12, 3w
Phase 4 : phase4, after allslices phase3 decompose, 3w
Phase 5 : phase5, after phase4, 3w
Phase 6 : phase6, after phase4, 3w
Phase 7 : phase7, after phase6, 4w
Rollout complete :milestone, rollout, after phase7, 0d
axisFormat  %Y-%m

Source.

Decomposition

Last update: 2024-09-18.

Plan

1. Introduce separate gitlab_sec schema
2. Introduce gitlab_sec database connection (defaulting to fallback to using gitlab_main database)
3. In parallel, begin decomposition of foreign keys and cross-database transactions following the loose order of SBOM, Security, and Vulnerability code boundaries. For each slice perform the following breakdown:
   1. Migrate tables with low referentiality (few foreign keys)
   2. Migrate tables with higher referentiality (many foreign keys)
   3. Identify and allowlist cross-joins to be addressed
   4. Identify and allowlist cross-database transactions to be addressed
   5. Remove previously identified cross-joins and cross-database transactions allowances
4. Await results of Logical Replication Production test to determine the viability of this as a migration path.
5. Depending on the results of the production test, formulate a path for the safe migration of the Secure/Govern dataset to a new physical database. These may take the form of the headings below.
6. Open Change Request to migrate tables using either (A) a phased approach mirroring code boundary slices above or (B) a single replication event for all tables in scope of decomposition
7. Update documentation around migrating self-managed instances to multiple databases

Migration Proposal A: Logical Replication

1. Research and test the possiblity of a staged logical replication in which we migrate small subsets of the Secure/Govern featureset at a time, such as SBOM.
   1. If a staged rollout is possible
      1. Identify the highest value feature subset to decompose
      2. Plan a decomposition strategy to separate only that feature to achieve a production benefit sooner.
      3. Establish the decomposed database instance
      4. Begin replicating the Secure/Govern data to the new database instance
      5. Write the necessary code to enable GitLab.com to begin utilising the new instance generically, and for the chosen feature subset.
      6. As this is a potentially risky operation, ensure production snapshots are ready and that customers are sufficiently informed of potential problems or dataloss in the event of failure.
      7. Begin testing transition of the feature to using the new database instance as it’s new write primary.
      8. If successful, globally rollout usage of the decomposed database for the feature subset.
      9. Repeat for each sufficiently sectionable feature subset until decomposition is completed.
   2. If a staged rollout is not possible
      1. Establish the decomposed database instance
      2. Begin replicating the full Secure/Govern data to the new database instance
      3. Write the necessary code to enable GitLab.com to begin utilising the new instance generically and for all Secure/Govern features.
      4. As this is a potentially risky operation, ensure production snapshots are ready and that customers are sufficiently informed of potential problems or dataloss in the event of failure.
      5. Begin testing transition of the Secure/Govern featureset to using the new database instance as it’s new write primary.
      6. If successful, globally rollout usage of the decomposed database for the full featureset.
2. Cleanup legacy data from the GitLab core database.

Migration Proposal B: Physical Replication

1. Determine acceptability of a full downtime for GitLab, or a temporary suspension of use for the entire Secure/Govern featureset to prevent dataloss. (Alternatively, notify users that there will be dataloss related to this featureset after a certain Date and Time)
   1. Begin communicating with customers ahead of time to minimise disatisfaction as a result of this disruption.
   2. Establish the decomposed database instance
   3. Write the necessary code to enable GitLab.com to begin utilising the new instance generically and for all Secure/Govern features.
   4. Begin testing transition of the Secure/Govern featureset to using the new database instance as it’s new write primary.
   5. Take GitLab down so that write traffic stops.
   6. Wait for replication to catch up on the node before promoting it to be the new leader of a new Secure DB cluster. Configure GitLab to write to this new Secure DB cluster.
   7. Globally rollout usage of the decomposed database for the full featureset.
   8. Cleanup legacy Secure/Govern data from the GitLab Core database.
   9. Cleanup legacy Core data from the new Secure/Govern database.

Migration Proposal C: Application Replication

1. As a staged rollout is possible, identify the highest value feature subset to decompose.
2. Plan a decomposition strategy to separate only that feature to achieve a production benefit sooner.
3. Establish the decomposed database instance
4. Write the necessary code to sync all possible data changes relating the chosen feature subset to the new database instance from whereever they may occur in the application.
5. Begin replicating the Secure/Govern data to the new database instance for the chosen feature subset.
6. Write the necessary code to enable GitLab.com to begin utilising the new instance generically, and for the chosen feature subset.
7. As this is a potentially risky operation, ensure production snapshots are ready and that customers are sufficiently informed of potential problems or dataloss in the event of transition failure, as some data may not be able to be synced back to the Core database.
8. Begin testing transition of the feature to using the new database instance as it’s new write primary.
9. If successful, globally rollout usage of the decomposed database for the feature.
10. Repeat for each sufficiently sectionable feature subset until decomposition is completed.

Roles and Responsibilities

Related Performance Projects

1. Tuple Reduction
   • Brian Williams (DRI)
   • Fabien Catteau
   • Michael Becker
2. Vulnerability Management Application Limits and Vulnerability Management Retention Policy
   • Mehmet Emin Inaç (DRI)
   • Joey Khabie
3. Cells 1.0
   • Subashis Chakraborty (DRI)

Useful References

Thanks

Much information and inspiration and experience is being enjoyed from the database-scalability working group who accomplished the succesful decomposition of the CI database.

View page source - Edit this page - please contribute.