GitLab System Administration - Hands-on Lab: Configure Instance Monitoring

Estimated time to complete: 30 minutes

Objectives

The purpose of this lab is to walk you through using Prometheus Service, and how to view Prometheus metrics. To learn more about Prometheus, click here.

Task A. Access the Prometheus Service

1. GitLab’s Prometheus server can be reached via TCP port 9090. Unfortunately, the training environment currently blocks inbound traffic to that port. As a workaround, you can open an SSH tunnel as follows.

   ssh -L 9090:localhost:9090 -i <SSH_KEY_NAME> root@<GITLAB_INSTANCE_HOSTNAME>
   

2. Navigate to http://localhost:9090 in a web browser to view the built-in Prometheus server.

3. To view the available metrics in Prometheus, navigate to http://localhost:9090/metrics.

   To see a more detailed list of each metric, check out the documentation.

Task B. Viewing Prometheus Metrics

For this example, suppose you wanted to monitor HTTP requests sent into your GitLab instance. To achieve this, we can use some built in GitLab metrics.

1. Navigate to http://localhost:9090.

2. In the input beside the magnifying glass button, type http_requests_total.

3. Click the Execute button.

4. Click Table to view a table of all the results generated by the query.

   You will likely see a variety of different HTTP requests in the results. Note that you can see the job, instance, HTTP method and status of each request.

5. Click Graph to view a graph of your data.

   On the left hand side of the graph, you will see a - and + button. You can click these buttons to change the scale of the graph. Try adjusting the scale to see how it impacts your data.

Task C. Alert Configurations

Prometheus can be used to alert administrators based on specific metrics. GitLab comes with a set of predefined alerts for Prometheus.

1. Navigate to http://localhost:9090.

2. Click Alerts in the top menu.

   Note that there are two sets of alert rules, one named GitLab and one named Node. The GitLab alerts correspond to platform issues such as a service being down, Postgres being down, or high queueing within services. A Node alert corresponds to the server GitLab is deployed on.

3. Note the file that contains the alerting rules. By default, the file is /var/opt/gitlab/prometheus/rules/gitlab.rules.

   As an administrator, you can define custom rules inside this file to use for alerting. This will allow you to monitor your system for any potential issues.

Task D. Common Performance Metrics

There are a variety of different metrics administrators can use to monitor the performance of their GitLab instances. Let’s examine a few common queries you can use to monitor an instance.

1. Navigate to http://localhost:9090.

2. In the input beside the magnifying glass button, type the following query:

   ((node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) or ((node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes) / node_memory_MemTotal_bytes)) * 100
   

   This query will show you the percentage of memory available on your instance.

3. Click Graph to view the percentage of memory over time.

4. In the query input, delete your previous query and replace it with the following query:

   1 - avg without (mode,cpu) (rate(node_cpu_seconds_total{mode="idle"}[5m]))
   

   This query will show you the percentage of CPU used on your instance.

5. Click Graph to view the percent of CPU used over time.

   For these metrics and many others, you may want to consider alerting for specific conditions. For example, you could alert for cases where the CPU usage exceeds 50% consistently, as it might indicate an issue.

Lab Guide Complete

You have completed this lab exercise. You can view the other lab guides for this course.

Suggestions?

If you’d like to suggest changes to the GitLab System Admin Hands-on Guide, please submit them via merge request.

View page source - Edit this page - please contribute.