Data Retention: Policy Framework

Motivation

As part of our effort to Limit Table Sizes, the database team is developing a data retention policy framework that other teams will follow in order to actively limit their table sizes.

Today, retention and data lifecycle management on GitLab.com is handled inconsistently. Individual teams solve it per table, with no shared vocabulary for describing the retention window, no agreed way of enforcing it, and no single place to record why a given decision was made. The result is that table growth is discovered reactively, and remediation work is designed from scratch each time.

This document proposes a unified, declarative framework for configuring the data lifecycle of a table, so that a retention decision is made once, recorded next to the table, and enforced by tooling.

Recommended framework

A unified, declarative framework for configuring the data lifecycle of a table. The fields below are added as top-level keys in db/docs/data_retention/<table_name>.yml, and validated in CI. Because the configuration lives in its own dedicated file, the fields do not need to be nested under a wrapping key.

Enforcement tooling will be set in place to verify that retention is actively being applied to all the tables in question.

Field specification

Exclude

Excludes the table from data retention. Unset by default, meaning the table is subject to retention. A table is excluded by setting exclude.reason to one of a small, fixed list of values. No database team approval is required — exclusion is self-served by picking a valid reason and recording the justification in the work_item. CI validates that reason is one of the allowed values.

Allowed reasons:

1. indefinite_retention — data must be retained indefinitely, for example for compliance or audit purposes, or a core entity table (for example organizations) that does not accumulate rows that can be aged out. This is a stopgap: the goal is to eventually move indefinitely retained data out of the hot OLTP database into cold storage, so tables using this reason are expected to be revisited.
2. technical_complexity — a table larger than the 50 GB soft limit that cannot be partitioned, so drop_partition is not achievable. The specific technical constraints that make partitioning impossible MUST be justified in the work_item.

Being technically difficult to delete from is not, on its own, a valid exclusion reason. If a table can remove data — even if doing so is awkward or requires engineering effort — it is not excluded; it uses delete_rows (BBO) and records the reasoning in the work_item. The exception is technical_complexity, which is reserved specifically for large tables where partitioning is not possible, with the constraints justified in the work_item.

When exclude.reason is set, the two coupled fields are fixed:

1. retention_window MUST be -1.
2. enforcement_strategy MUST be none.

The coupling is bidirectional: retention_window: -1 and enforcement_strategy: none are valid only when exclude.reason is set. In every other case both fields MUST be set to concrete values following the decision tree. CI validation on these fields enforces this coupling.

Retention window

How long the data remains in the database since its initial creation.

Value type: numeric (days). -1 is valid only when exclude.reason is set (see Exclude); in all other cases this field MUST be set to a positive number of days.

Enforcement strategy

How the retention window is enforced in our systems.

Potential values:

1. drop_partition — table is partitioned and partitions are dropped once they age past retention_window.
2. delete_rows (BBO) — data cannot be retrieved after deletion.
3. transient_data — data tied to a user or feature lifecycle that is already deleted as part of that lifecycle.
4. none — the table is excluded from retention. Valid only when exclude.reason is set (see Exclude).

Outside the excluded case, this field MUST be set to drop_partition, delete_rows, or transient_data following the decision tree.

flowchart TD
    A{"Is the data naturally deleted<br>by the feature lifecycle?"}
    A -->|Yes| B(["transient_data<br>(no-op; handled by the feature lifecycle)"])
    A -->|No| G{"Can the data be deleted?"}
    G -->|No| I(["none<br>(set exclude.reason: indefinite_retention)"])
    G -->|Yes| C{"Is the table larger than 50 GB?"}
    C -->|Yes| J{"Is partitioning possible?"}
    J -->|Yes| D(["drop_partition"])
    J -->|No| K(["none<br>(set exclude.reason: technical_complexity)"])
    C -->|No| E{"Is partitioning possible?"}
    E -->|Yes| F(["drop_partition"])
    E -->|No| H(["delete_rows<br>(BBO)"])

Caveat: data recovery

The point at which data becomes unrecoverable differs by strategy:

1. drop_partition — after the retention period the partition is detached and kept for a grace window before being dropped (7 days by default in the current PartitionManager, configurable to a lower value). Data remains recoverable from the detached partition until it is dropped.
2. delete_rows — a hard cutoff: rows are lost immediately on deletion, as no soft-delete system is in place today.
3. transient_data — data loss is governed by the owning feature’s lifecycle, not by this framework.
4. none — no data loss; the table is retained indefinitely.

Why deleting rows is a last resort

A DELETE is a write, and enforcing a retention window through deletes turns cleanup into a continuous write workload that scales with volume. To hold a retention window on a growing table, the deletion rate has to keep pace with the insertion rate — so every row inserted is eventually a row deleted, and the table carries roughly double the write load it would otherwise. This cost is paid forever, in proportion to how much data flows through the table:

1. Throughput. To keep the retention window, the deletion rate must match the insertion rate. Every deleted row is itself a write: it dirties the heap page, leaves a dead entry in every index, and generates WAL, all of which autovacuum later reads and rewrites to clean up. Enforcing retention through deletes therefore roughly doubles the table’s write load, and if deletion cannot keep up with insertion the retention window is never enforced.
2. Connections. Every batch holds a pooled connection for the length of its transaction, and keeping up with a high insertion rate means running many of them in parallel.
3. I/O load. Each deleted row writes its heap page and leaves a dead entry in every index, which autovacuum then reads and rewrites.
4. Replication load. The resulting WAL ships to every replica and to the WAL archive, adding lag.

drop_partition is not free either, but its cost is one-time: converting an existing large table to a partitioned one is an expensive migration (see Large tables). After that, DETACH PARTITION CONCURRENTLY is cheap regardless of how many rows the partition held, and tables partitioned from the start avoid the migration cost entirely. So delete_rows pays continuously in proportion to volume, while drop_partition pays once. See PlanetScale’s The only scalable delete for more.

delete_rows is still reasonable for narrow, lightly indexed, low-traffic tables with no viable partition key — record the reasoning in the work_item.

Large tables

The hard limit for a table on GitLab.com is 100 GB. We use 50 GB as a soft limit — a buffer that leaves room to partition existing tables or perform actions to consistently reduce the size footprint before the hard limit is reached. Tables larger than this 50 GB soft limit must use partitioning. Any non-partitioning proposal for a table over this limit must record its justification in the work_item. See Large tables limitations for the constraints that apply to these tables.

Enforcing

Defines whether the retention policy enforcement is active.

A retention policy can be selected at table creation time with enforcing set to false, and switched to true once the retention policy enforcement is actually in place. This separation exists because the policy is declared when the table is created, while the enforcement implementation often lands later.

Work item

An issue or epic that records both the justification for the retention decision and the implementation plan for enforcing it on the table. Enabling the enforcing flag requires this work item to be provided. The database team will use this work item to track enforcement progress.

Pause mechanism

How retention enforcement can be disabled/paused for a table.

Potential values: none, application_setting, feature_flag

Pause mechanism::Name

The name of the application setting or feature flag responsible for enabling and disabling retention for the specified table. A single setting or flag can be shared across multiple tables — for example, one setting covering the CI domain.

Defaults for GitLab.com

Based on the standard requirements for GitLab.com, the defaults are the following. These values are for reference only — the decision tree above should be followed for optimal decision making.

Options for GitLab Self-Managed

Data retention is enabled by default on all new instances. Owning teams are responsible for providing ways to disable data retention for their domains. They can create application settings or feature flags shared across multiple tables to cover entire domains.

Enforcement tooling

Data retention enforcement checks

A tool that checks whether tables with enforcing: false — or with enforcing: true while paused via their pause_mechanism (application setting or feature flag) — have had incoming traffic for a given period. Where they have, the owning team is notified and asked to check in on the retention enforcement progress.

The rationale is that the retention policy is populated when the table is created, while the actual retention enforcement may be implemented at a later stage. This check closes the gap between declaring a policy and enforcing it.

Table size checks

A tool that monitors table sizes and flags tables whose average size is higher than the 50 GB soft limit. When a table is over the limit, an issue is created for the owning team to address it — either by enforcing retention or by planning the transitionary actions (such as partitioning) needed to bring the table back under the limit before it reaches the 100 GB hard limit.

View page source - Edit this page - please contribute.