Repository Backups

Summary

This proposal seeks to provide an out-of-a-box repository backup solution to GitLab that gives more opportunities to apply Gitaly specific optimisations. It will do this by moving repository backups out of backup.rake into a coordination worker that enumerates repositories and makes per-repository decisions to trigger repository backups that are streamed directly from Gitaly to object-storage.

The advantages of this approach are:

• The backups are only transferred once, from the Gitaly hosting the physical repository to object-storage.
• Smarter decisions can be made by leveraging specific repository access patterns.
• Distributes backup and restore load.
• Since the entire process is run within Gitaly existing monitoring can be used.
• Provides architecture for future WAL archiving and other optimisations.

This should relieve the major pain points of the existing two strategies:

• backup.rake - Repository backups are streamed from outside of Gitaly using RPCs and stored in a single large tar file. Due to the amount of data transferred these backups are limited to small installations.
• Snapshots - Cloud providers allow taking physical storage snapshots. These are not an out-of-a-box solution as they are specific to the cloud provider.

Motivation

Goals

• Improve time to create and restore repository backups.
• Improve monitoring of repository backups.

Non-Goals

• Improving filesystem based snapshots.

Filesystem based Snapshots

Snapshots rely on cloud platforms to be able to take physical snapshots of the disks that Gitaly and Praefect use to store data. While never officially recommended, this strategy tends to be used once creating or restoring backups using backup.rake takes too long.

Gitaly and Git use lock files and fsync in order to prevent repository corruption from concurrent processes and partial writes from a crash. This generally means that if a file is written, then it will be valid. However, because Git repositories are composed of many files and many write operations may be taking place, it would be impossible to schedule a snapshot while no file operations are ongoing. This means the consistency of a snapshot cannot be guaranteed and restoring from a snapshot backup may require manual intervention.

WAL may improve crash resistance and so improve automatic recovery from snapshots, but each repository will likely still require a majority of voting replicas in sync.

Since each node in a Gitaly Cluster is not homogeneous, depending on replication factor, in order to create a complete snapshot backup all nodes would need to have snapshots taken. This means that snapshot backups have a lot of repository data duplication.

Snapshots are heavily dependent on the cloud provider and so they would not provide an out-of-a-box experience.

Downtime

An ideal repository backup solution would allow both backup and restore operations to be done online. Specifically we would not want to shutdown or pause writes to ensure that each node/repository is consistent.

Consistency

Consistency in repository backups means:

• That the Git repositories are valid after restore. There are no partially applied operations.
• That all repositories in a cluster are healthy after restore, or are made healthy automatically.

Backups without consistency may result in data-loss or require manual intervention on restore.

Both types of consistency are difficult to achieve using snapshots as this requires that snapshots of the filesystems on multiple hosts are taken synchronously and without repositories on any of those hosts currently being mutated.

Distribute Work

We want to distribute the backup/restore work such that it isn’t bottlenecked on the machine running backup.rake, a single Gitaly node, or a single network connection.

On backup, backup.rake aggregates all repository backups onto its local filesystem. This means that all repository data needs to be streamed from Gitaly (possibly via Praefect) to where the Rake task is being run. If this is CNG then it also requires a large volume on Kubernetes. The resulting backup tar file then gets transferred to object storage. A similar process happens on restore, the entire tar file needs to be downloaded and extracted on the local filesystem, even for a partial restore when restoring a subset of repositories. Effectively all repository data gets transferred, in full, multiple times between multiple hosts.

If each Gitaly could directly upload backups it would mean only transferring repository data a single time, reducing the number of hosts and so the amount of data transferred over all.

Gitaly Controlled

Gitaly is looking to become self-contained and so should own its backups.

backup.rake currently determines which repositories to backup and where those backups are stored. This restricts the kind of optimisations that Gitaly could apply and adds development/testing complexity.

Monitoring

backup.rake is run in a variety of different environments. Historically backups from Gitaly’s perspective are a series of disconnected RPC calls. This has resulted in backups having almost zero monitoring. Ideally the process would run within Gitaly such that the process could be monitored using existing metrics and log scraping.

Automatic Backups

When backup.rake is set up on cron it can be difficult to tell if it has been running successfully, if it is still running, how long it took, and how much space it has taken. It is difficult to ensure that cron always has access to the previous backup to allow for incremental backups or to determine if updating the backup is required at all.

Having a coordination process running continuously will allow moving from a single-shot backup strategy to one where each repository determines its own backup schedule based on usage patterns and priority. This way each repository should be able to have a reasonably up-to-date backup without adding excess load to any Gitaly node.

Updated Repositories Only

backup.rake packages all repository backups into a tar file and generally has no access to the previous backup. This makes it difficult to determine if the repository has changed since last backup.

Having access to previous backups on object-storage would mean that Gitaly could more easily determine if a backup needs to be taken at all. This allows us to waste less time backing up repositories that are no longer being modified.

Point-in-time Restores

There should be a mechanism by which a set of repositories can be restored to a specific point in time. The identifier (backup ID) used should be able to be determined by an admin and apply to all repositories.

WAL (write ahead log)

We want to be able to provide infrastructure to allow continuous archiving of the WAL. This means providing a central place to stream the archives to and being able to match any full backup to a place in the log such that repositories can be restored from the full backup, and the WAL applied up to a specific point in time.

WORM

Any Gitaly accessible storage should be WORM (write once, read many) in order to prevent existing backups being modified in the case an attacker gains access to a nodes object-storage credentials.

The pointer layout currently used by repository backups relies on being able to overwrite the pointer files, and as such would not be suitable for use on a WORM file store.

WORM is likely object-storage provider specific:

• AWS object lock
• Google Cloud WORM retention policy.
• MinIO object lock

bundle-uri

Having direct access backup data may open the door for clone/fetch transfer optimisations using bundle-uri. This allows us to point Git clients directly to a bundle file instead of transferring packs from the repository itself. The bulk repository transfer can then be faster and is offloaded to a plain http server, rather than the Gitaly servers.

Proposal

The proposal is broken down into an initial MVP and per-repository coordinator.

MVP

The goal of the MVP is to validate that moving backup processing server-side will improve the worst case, total-loss, scenario. That is, reduce the total time to create and restore a full backup.

The MVP will introduce backup and restore repository RPCs. There will be no coordination worker. The RPCs will stream a backup directly from the called Gitaly node to object storage. These RPCs will be called from backup.rake via the gitaly-backup tool. backup.rake will no longer package repository backups into the backup archive.

This work is already underway, tracked by the Server-side Backups MVP epic.

Per-Repository Coordinator

Instead of taking a backup of all repositories at once via backup.rake, a backup coordination worker will be created. This worker will periodically enumerate all repositories to decide if a backup needs to be taken. These decisions could be determined by usage patterns or priority of the repository.

When restoring, since each repository will have a different backup state, a timestamp will be provided by the user. This timestamp will be used to determine which backup to restore for each repository. Once WAL archiving is implemented, the WAL could then be replayed up to the given timestamp.

This wider effort is tracked in the Server-side Backups epic.

Design and implementation details

MVP

There will be a pair of RPCs BackupRepository and RestoreRepository. These RPCs will synchronously create/restore backups directly onto object storage. backup.rake will continue to use gitaly-backup with a new --server-side flag. Each Gitaly will need a backup configuration to specify the object-storage service to use.

Initially the structure of the backups in object-storage will be the same as the existing pointer layout.

For MVP the backup ID must match an exact backup ID on object-storage.

The configuration of object-storage will be controlled by a new config config.backup.go_cloud_url. The Go Cloud Development Kit tries to use a provider specific way to configure authentication. This can be inferred from the VM or from environment variables. See Supported Storage Services.

Alternative Solutions

View page source - Edit this page - please contribute.