GitLab System Admin - Hands-On Lab: Kubernetes Backup and Restore

This Hands-On Guide demonstrates how to backup and restore your GitLab Kubernetes instance

Estimated time to complete: 30 minutes

Objectives

The objective of this lab is to demonstrate how to back up a GitLab instance on a virtual machine, and restore said instance to a previous state. For more information about backing up/restoring a GitLab instance, click here.

Task A. Configure backup settings

Open an SSH session on your GitLab instance server.
Open your helm values using your text editor of choice.

Inside this file, you will need to update a few configurations to enable backups. To start, add the following global configuration:

global:
appConfig:
    artifacts:
    bucket: gitlab-scosentino-3k-backups
    backups:
    bucket: gitlab-scosentino-3k-backups
    lfs:
    bucket: gitlab-scosentino-3k-backups
    packages:
    bucket: gitlab-scosentino-3k-backups
    terraformState:
    bucket: gitlab-scosentino-3k-backups
    tmpBucket:
    bucket: gitlab-scosentino-3k-backups
    uploads:
    bucket: gitlab-scosentino-3k-backups

These configurations set the name of the remote storage bucket for each backup type.

Kubernetes backups are made through the toolbox pod. To enable this pod to run backups, it needs to be able to connect to the backup provider. The following configuration can be added to the gitlab block to achieve this:
```
toolbox:
    backups:
    objectStorage:
        backend: s3
        config:
        key: config
        secret: my-s3cfg
```
To be able to access these buckets, you need to provide credentials for an AWS service account. To do this, run the command kubectl create secret generic my-s3cfg --from-file=config=my-s3cfg.
After adding this configuration, run the command helm upgrade --install gitlab gitlab/gitlab --timeout 100s -f values.yml.

Task B. Backup the GitLab instance

To take a full backup, start by locating your toolbox pod.
```
kubectl get pods -lapp=toolbox
```

Next, run the backup utility from the toolbox pod.

kubectl exec <toolbox-name>  -it -- backup-utility

Finally, retrieve your Kubernetes secrets and save them to a file.

#Get your rails secrets
kubectl get secrets | grep rails-secret

#Save the secrets to a local location
kubectl get secrets <rails-secret-name> -o jsonpath="{.data['secrets\.yml']}" | base64 --decode > gitlab-secrets.yaml

At this point, the backup is now stored in your object storage. Take note of the file name, for example, s3://bucket/1729261040_2024_10_18_17.4.1-ee_gitlab_backup.tar

Important: Take note of the timestamp (in this example: 1729261040_2024_10_18_17.4.1-ee). You will need this for the restore process.

Task C. Make some changes to GitLab settings

Sign into your GitLab instance with a web browser and open your sidebar. In the bottom left corner, click Admin area.
In the left sidebar, select Settings > General.
Expand Account and limit and change the maximum attachment size to 500 MiB, and the default project limits to 10000.
Click Save changes to save the changes.
Refresh the page and verify your changes were applied.

Task D. Restore from backup

Return to the SSH session on your GitLab instance server.

Delete your current Kubernetes secrets:

kubectl delete secret <rails-secret-name>

Create a new set of secrets based on your backed up Kubernetes secrets:

kubectl create secret generic <rails-secret-name> --from-file=secrets.yml=gitlab-secrets.yaml

Restart your Kubernetes pods to apply the secret

kubectl delete pods -lapp=sidekiq,release=gitlab
kubectl delete pods -lapp=webservice,release=gitlab
kubectl delete pods -lapp=toolbox,release=gitlab

To start the restore process, locate the name of your toolbox pod.
```
kubectl get pods lapp=toolbox
```

Scale your Kubernetes pod replicas down to 0

kubectl scale deploy -lapp=sidekiq,release=gitlab --replicas=0
kubectl scale deploy -lapp=webservice,release=gitlab --replicas=0
kubectl scale deploy -lapp=prometheus,release=gitlab --replicas=0

Run a restore using the timestamp ID you received from the backup command
```
kubectl exec <Toolbox pod name> -it -- backup-utility --restore -t your-timestamp-id
```
If you are prompted to start the restore process, type yes.

After the restore process completes, scale your pods back up:

kubectl scale deploy -lapp=sidekiq,release=gitlab --replicas=1
kubectl scale deploy -lapp=webservice,release=gitlab --replicas=1
kubectl scale deploy -lapp=prometheus,release=gitlab --replicas=1

Try to access your Kubernetes instance and verify that the restore completed successfully.

Lab Guide Complete

You have completed this lab exercise. You can view the other lab guides for this course.

Suggestions?

If you’d like to suggest changes to the GitLab System Admin Basics Hands-on Guide, please submit them via merge request.

Last modified November 14, 2024: Updating names (23df6bff)

View page source - Edit this page - please contribute.