Database Framework Group

Vision

Developing solutions for scalability, application performance, data growth and developer enablement especially where it concerns interactions with the database.

Mission

Focusing on the database, our mission is to provide solutions that allow us to scale to our customer’s demands. To provide tooling to proactively identify performance bottlenecks to inform developers early in the development lifecycle. To increase the number of database maintainers and provide database best practices to the community contributors and development teams within GitLab.

Team Members

The following people are permanent members of the Database Team:

Name	Role
Alex Ives	Backend Engineering Manager, Database
Backend Engineer	Backend Engineer, Database
Jon Jenkins	Senior Backend Engineer, Database
Krasimir Angelov	Staff Backend Engineer, Database
Leonardo da Rosa	Backend Engineer, Database
Matt Kasa	Staff Backend Engineer, Database
Maxime Orefice	Senior Backend Engineer, Database
Prabakaran Murugesan	Senior Backend Engineer, Database
Simon Tomlinson	Staff Backend Engineer, Database

Stable Counterparts

The following members of other functional teams are our stable counterparts:

Stable Counterparts to other teams

The Database Group is often called upon to provide consulting to other groups. To more efficiently support these requests we have created this stable counterparts table.

Meetings

Whenever possible, we prefer to communicate asynchronously using issues, merge requests, and Slack. However, face-to-face meetings are useful to establish personal connection and to address items that would be more efficiently discussed synchronously such as blockers.

• Database Group Sync every Tuesday and Thursday at 1:00 PM UTC
  • Tuesdays - we start with any ~infradev issues requiring reviews, then we focus on weekly priorities.
  • Thursdays - are optional and open agenda. Anyone can bring topics to the team to discuss. Typically we reserve the first Thursday after the milestone closes to hold a synchronous retrospective.
• Database Office Hours (internal link); YouTube recordings
  • Wednesdays, 3:30pm UTC (bi-weekly)
  • (APAC) Thursdays, 3:30am UTC (bi-weekly, alternating)

Work

We follow the GitLab engineering workflow guidelines. To bring an issue to our attention please create an issue in the relevant project. Add the ~"group::database" label along with any other relevant labels. If it is an urgent issue, please reach out to the Product Manager or Engineering Manager listed in the Stable Counterparts section above.

What we do

The team is responsible for the PostgreSQL application interactions to enable high performance queries while offering features to support scalability and strengthen availability. PostgreSQL is the heart of Rails application, and there is no shortage of work to make GitLab more performant, scalable, and highly available from database perspective. Some of the current priorities include implementing partitioning to improve query performance and creating tooling to enable development teams to implement their own partitioning strategies more easily. We are working on tools that will help developers “shift left” in their migration testing prior to deployment. We are always looking for ways to continuously care for the performance of our databsae and improve our developer documentation. For more in-depth details of what we are working on please review our Roadmap section below.

In order to follow what the database group is currently working on, we recommend watching our group’s kickoff presentations for new milestones and the respective milestone planning issues.

Activity Log

Since end of 2021, we maintain an activity log to keep track of past projects and outcomes.

Planning

We use a planning issue to discuss priorities and commitments for the milestone. This happens largely asynchronously, but when we do need to discuss synchronously we discuss during the Tuesday team meeting timeslot.

Issue Weights

The database group is experimenting with using expected merge request count as an issue weight. Before each milestone starts, we’ll ping each assigned issue without a weight and ask folks to add weights to them.

We decided to use merge request count as an issue weight for a few reasons:

• The process encourages folks to consider ahead how an issue could be broken down more and enumerate it in advance
• It’s easy to describe and learn, making it easier for the team to come to a shared understanding
• Merge request rate is one of the main ways our team is measured

Process for weighting Issues

1. With an emphasis towards smaller more iterative changes rather than large changes that may take longer to review and merge, consider how many merge requests could this be broken into.

2. Add a comment enumerating the expected merge requests. For example:

   Just one merge request to documentation

   One to gitlab for database changes, one for new functionality, one for documentation changes, and one to omnibus

3. Add the count as a weight. For example, if you think there could be one to gitlab for database changes, one for new functionality, one for documentation changes, and one to omnibus - you would assign /weight 4

Timeline for implementation

15.4 - 15.7: We’ll ping each issue in the milestone without a weight and ask folks to add one to collect data 15.8 +: TBD

Triage rotation

We have a fairly simple triage rotation. Each week one team member is dedicated to triaging incoming issues for the database group. This allows for the rest of the team to focus on their current priorities with fewer interruptions. Each week, a bot will file an issue that gets automatically assigned to next team member in the rotation. We order the triage rotation by alpha-order based on first name to keep it very simple. If a team member is on PTO the week they are assigned, the issue will be re-assigned to the next person.

Issues needing triage can come in through many different paths. Some common areas to monitor while on triage:

• Newer issues (< 7 days old) with the ~database label that are not assigned to a group. Example search
• Newer issues that were assigned ~group::database but do not have a throughput label or ~database::triage labels. Example search
• Newer issues that were assigned ~database::triage and have not previously been reviewed
• When we get pinged on the #g_database slack channel for assistance

When the triage team member discovers an issue requiring team attention some of the possible outcomes are:

• Directly address the issue if it is a simple fix
• Direct to our customer support counterparts as appropriate
• Add the ~database::triage label and review during team sync meeting
• Add a milestone and ping the manager, or label the issue ~workflow::scheduling
• Close as duplicate and link to the duplicate issue

The goal is to keep the number of issues for triage low and manageable.

Tip: In order to remove closed issues from the triage board, use this search and edit multiple issues at once to remove the ~database::triage label.

Boards

Database by Milestone The Milestone board gives us a “big picture” view of issues planned in each milestone.

Database: Build · Boards · GitLab.org · GitLab The build board gives you an overview of the current state of work for group::database. These issues have already gone through validation and are on the Product Development Build Track. Issues are added to this board by adding the current active milestone and group::database labels. Issues in the workflow::ready for development column are ordered in priority order (top down). Team members use this column to select the next item to work on.

Database: Validation The validation board is a queue for incoming issues for the Product Manager to review. A common scenario for the Database Team validation board is when an issue is created that requires further definition before it can be prioritized. The issue typically states a big picture idea but is not yet detailed enough to take action. The Database Team will then go through a refinement process to break down the issue into actionable steps, create exit criteria and prioritize against ongoing efforts. If an issue becomes too large, it will be promoted to an epic and small sub-issues will be created.

Database: Triage The triage board is for incoming issues that require further investigation for team assignment, prioritization, previously existing issues, etc. Within the Database Group we have implemented a weekly triage rotation where one team member is responsible for monitoring this board for timely responses.

Say/Do Ratio

We use the ~Deliverable label to track our Say/Do ratio. At the beginning of each milestone, during a Database Group Weekly meeting, we review the issues and determine those issues we are confident we can deliver within the milestone. The issue will be marked with the ~Deliverable label. At the end of the milestone the successfully completed issues with the ~Deliverable label are tracked in two places. We have a dashboard in Tableau that will calculate how many were delivered within the milestone and account for issues that were moved. Additionally, our milestone retro issue lists all of the ~Deliverable issues shipped along with those that missed the milestone.

Roadmap

The Database Group Roadmap gives a view of what is currently in flight as well as projects that have been prioritized for the next 3+ months.

Weekly Team Updates

The enablement section is using status issues in order to provide regular status updates. Each week, the team’s engineering manager posts general announcments, and members of the team post updates on their in progress projects.

These issues can be found here (internal).

Documentation

We document our insights, road maps and other relevant material in this section.

1. Database Lexicon - terms and definitions relating to our Database
2. Database Strategy: Guidance for proposed database changes
3. On table partitioning (February 2020)
4. Postgres: Sharding with foreign data wrappers and partitioning
5. Sharding GitLab by top-level namespace
6. Sharding with CitusDB (April 2020)
7. Table partitioning: Issue group search as an example (March 2020)
8. Working with the GitLab.com database for developers
9. Database schema proposals for Container Registry (September 2020)
10. Workload analysis for GitLab.com (October 2020)
11. Multi-database Background migrations (October 2021)

Performance Indicators (Internal)

1. Enablement::Database - Performance Indicators Dashboard
2. Average Query Apdex for GitLab.com
   • Master
     • Target: 100ms - Tolerable 250ms
     • Target: 50ms - Tolerable 100ms
   • Replicas
     • Target: 100ms - Tolerable 250ms
     • Target: 50ms - Tolerable 100ms

Common Links

• Slack Channel #g_database - Official Business
• Slack Channel #db-lounge - Team Chat
• Database Epics
• Database Subgroup - Issues and templates related to team processes.
• Product Development Timeline
• YouTube: Database Team Playlist
• YouTube: Database Office Hours Playlist

Dashboards

View page source - Edit this page - please contribute.