GitLab System Admin - Hands-On Lab: Logging and Monitoring GitLab Kubernetes

Estimated time to complete: 30 minutes

Objectives

The objective of this lab is to demonstrate how to use Prometheus and Grafana for log analysis and monitoring.

Task A. Access the Prometheus Service

1. GitLab’s Prometheus server can be reached by TCP port 9090. To access this port, you need to create a proxy into your Kubernetes cluster with kubectl proxy. The proxy runs on your GitLab instance and only listens on localhost:8001 of that instance, so the web browser on your local computer cannot reach it directly. To make it reachable, you will connect to your instance with an SSH tunnel that forwards port 8001 from your local computer to the instance.

2. If you currently have an SSH session open on your GitLab instance, type exit and press Enter to close it. An SSH tunnel is set up when the connection is opened, so you need to reconnect to add one.

3. From a terminal on your local computer, SSH into your GitLab instance again, this time with a tunnel on port 8001:

   ssh -L 8001:localhost:8001 -i <keyfile_name> student-user@<vm_ip_address>
   

   If port 8001 is already in use on your local computer, forward a different local port instead, such as -L 8081:localhost:8001, and use 8081 instead of 8001 in the URLs throughout this lab.

4. To access Prometheus, you need to know its pod name. To find this, run the command kubectl get pods in the SSH session. Search for Prometheus in the output.

5. In the same SSH session, start the proxy with the following command:

   kubectl proxy
   

   kubectl proxy keeps running in the foreground. Leave this terminal open for the rest of the lab, because closing it stops both the proxy and the SSH tunnel. If you need to run other commands on your instance, open a second terminal and SSH into your instance again.

6. Navigate to http://localhost:8001/api/v1/namespaces/default/pods/PROMETHEUS_POD_NAME:9090/proxy/ in a web browser on your local computer to view the built-in Prometheus server.

7. To view the available metrics in Prometheus, navigate to http://localhost:8001/api/v1/namespaces/default/pods/PROMETHEUS_POD_NAME:9090/proxy/metrics.

   To see a more detailed list of each metric, check out the documentation.

Task B. Viewing Prometheus Metrics

For this example, suppose you wanted to monitor HTTP requests sent into your GitLab instance. To achieve this, we can use some built in GitLab metrics.

1. Navigate to http://localhost:8001/api/v1/namespaces/default/pods/PROMETHEUS_POD_NAME:9090/proxy/.

2. In the input beside the magnifying glass button, type http_requests_total.

3. Click the Execute button.

4. Click Table to view a table of all the results generated by the query.

   You will likely see a variety of different HTTP requests in the results. Note that you can see the job, instance, HTTP method and status of each request.

5. Click Graph to view a graph of your data.

   On the left hand side of the graph, you will see a - and + button. You can click these buttons to change the scale of the graph. Try adjusting the scale to see how it impacts your data.

Task C. Alert Configurations

Prometheus can be used to alert administrators based on specific metrics. GitLab comes with a set of predefined alerts for Prometheus.

1. Navigate to http://localhost:8001/api/v1/namespaces/default/pods/PROMETHEUS_POD_NAME:9090/proxy/.

2. Click Alerts in the top menu.

   Note that there are two sets of alert rules, one named GitLab and one named Node. The GitLab alerts correspond to platform issues such as a service being down, Postgres being down, or high queueing within services. A Node alert corresponds to the server GitLab is deployed on.

3. Note that in Kubernetes deployments, alert rules are configured through ConfigMaps and the Prometheus Helm chart configuration rather than file-based rules.

   As an administrator, you can define custom rules through Kubernetes ConfigMaps or by updating the Prometheus Helm values to add alerting rules.

Task D. Common Performance Metrics

There are a variety of different metrics administrators can use to monitor the performance of their GitLab instances. Let’s examine a few common queries you can use to monitor an instance.

1. Navigate to http://localhost:8001/api/v1/namespaces/default/pods/PROMETHEUS_POD_NAME:9090/proxy/.

2. In the input beside the magnifying glass button, type the following query:

   sum(rate(gitlab_cache_misses_total[5m])) by (cache_key)
   

   This query will show you the rate of cache misses in GitLab, which can indicate performance issues. Note that the standard node metrics (memory, CPU) require additional exporters like node-exporter or kube-state-metrics to be installed in the Kubernetes cluster.

3. Click Graph to view the cache miss rate over time.

4. In the query input, delete your previous query and replace it with the following query:

   sum(rate(http_requests_total[5m])) by (job)
   

   This query will show you the rate of HTTP requests per job in your GitLab instance.

5. Click Graph to view the HTTP request rate over time.

   For these metrics and many others, you may want to consider alerting for specific conditions. For example, you could alert for cases where the CPU usage exceeds 50% consistently, as it might indicate an issue.

Lab Guide Complete

You have completed this lab exercise. You can view the other lab guides for this course.

Suggestions?

If you wish to make a change to this lab, please submit your changes through a Merge Request.

View page source - Edit this page - please contribute.