GitLab System Admin - Hands-On Lab: Logging and Monitoring GitLab Omnibus
Estimated time to complete: 30 minutes
Objectives
The objective of this lab is to demonstrate how to use Prometheus and Grafana for log analysis and monitoring.
Task A. Access the Prometheus Service
-
GitLab’s Prometheus server can be reached via TCP port 9090. Unfortunately, the training environment currently blocks inbound traffic to that port. As a workaround, you can open an SSH tunnel as follows.
ssh -L 9090:localhost:9090 -i <SSH_KEY_NAME> root@<GITLAB_INSTANCE_HOSTNAME>
-
Navigate to
http://localhost:9090
in a web browser to view the built-in Prometheus server. -
To view the available metrics in Prometheus, navigate to
http://localhost:9090/metrics
.To see a more detailed list of each metric, check out the documentation.
Task B. Viewing Prometheus Metrics
For this example, suppose you wanted to monitor HTTP requests sent into your GitLab instance. To achieve this, we can use some built in GitLab metrics.
-
Navigate to
http://localhost:9090
. -
In the input beside the magnifying glass button, type
http_requests_total
. -
Click the Execute button.
-
Click Table to view a table of all the results generated by the query.
You will likely see a variety of different HTTP requests in the results. Note that you can see the
job
,instance
,HTTP method
andstatus
of each request. -
Click Graph to view a graph of your data.
On the left hand side of the graph, you will see a - and + button. You can click these buttons to change the scale of the graph. Try adjusting the scale to see how it impacts your data.
Task C. Alert Configurations
Prometheus can be used to alert administrators based on specific metrics. GitLab comes with a set of predefined alerts for Prometheus.
-
Navigate to
http://localhost:9090
. -
Click Alerts in the top menu.
Note that there are two sets of alert rules, one named GitLab and one named Node. The GitLab alerts correspond to platform issues such as a service being down, Postgres being down, or high queueing within services. A Node alert corresponds to the server GitLab is deployed on.
-
Note the file that contains the alerting rules. By default, the file is
/var/opt/gitlab/prometheus/rules/gitlab.rules
.As an administrator, you can define custom rules inside this file to use for alerting. This will allow you to monitor your system for any potential issues.
Task D. Common Performance Metrics
There are a variety of different metrics administrators can use to monitor the performance of their GitLab instances. Let’s examine a few common queries you can use to monitor an instance.
-
Navigate to
http://localhost:9090
. -
In the input beside the magnifying glass button, type the following query:
((node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) or ((node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes) / node_memory_MemTotal_bytes)) * 100
This query will show you the percentage of memory available on your instance.
-
Click Graph to view the percentage of memory over time.
-
In the query input, delete your previous query and replace it with the following query:
1 - avg without (mode,cpu) (rate(node_cpu_seconds_total{mode="idle"}[5m]))
This query will show you the percentage of CPU used on your instance.
-
Click Graph to view the percent of CPU used over time.
For these metrics and many others, you may want to consider alerting for specific conditions. For example, you could alert for cases where the CPU usage exceeds 50% consistently, as it might indicate an issue.
Task B. Install Grafana
Grafana provides you with a method of visualizing logs and metrics for your GitLab instance. Let’s start by installing Grafana
-
SSH into your GitLab instance.
-
Pull the Grafana key to your instance using
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor > /etc/apt/keyrings/grafana.gpg
. -
Add the repository to your package manager using
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | tee /etc/apt/sources.list.d/grafana.list
. -
Update the apt repository:
sudo apt-get update
. -
Install Grafana and its related components using
apt-get install loki promtail grafana
. -
Start the Grafana server using
sudo systemctl start grafana-server
. -
To verify that Grafana is running, navigate to
http://your-ip:3000/login
. Your default username and password areadmin
.
Task C. Configure Grafana to collect logs
Grafana provides you with useful tools for log collection and analysis. Let’s see how Grafana can be configured to capture and display some of our GitLab log files.
-
SSH onto your GitLab instance.
-
Open the file
/etc/promtail/config.yml
in the text editor of your choice. -
In this file, verify your server, positions, and clients:
server: http_listen_port: 9080 grpc_listen_port: 0 positions: filename: /tmp/positions.yaml clients: - url: http://localhost:3100/loki/api/v1/push
-
Take note specifically of the
loki
URL and port. You will use this later. -
Log collection configurations are set in the
scrape_configs
section of the configuration file. These configurations require the following values to be set:targets
: The location of the target log file, usuallylocalhost
.job
: A unique name for the log scraping job.__path__
: The location of the log files.
-
As an example, copy the following
scrape_configs
into your Grafana configuration file, replacing teh existingscrape_configs
file.scrape_configs: - job_name: system static_configs: - targets: - localhost labels: job: nginx __path__: /var/log/gitlab/nginx/* - targets: - localhost labels: job: workhorse __path__: /var/log/gitlab/gitlab-workhorse/* - targets: - localhost labels: job: rails __path__: /var/log/gitlab/gitlab-rails/production_json.log
This configuration adds three log files to Grafana: Nginx, Workhorse, and rails.
-
After adding this data, save the file. Restart
promtail
usingsudo systemctl restart promtail
. -
With this complete, navigate to Grafana at
http://your-gitlab-ip:3000/login
. -
Authenticate as your admin user.
-
In the left sidebar, select Connections > Data Sources.
-
Select Loki.
-
In the URL, input
http://localhost:3100
. -
Select Save & Test.
-
You should see the message Data source successfully connected.
Task D. Analyze data with Grafana
-
Navigate to your Grafana instance.
-
In the left sidebar, select Explore.
-
Make sure Loki is selected in the dropdown beside Outline.
-
In the Label filters enter
job
. -
In the value, enter nginx.
You can replace this with any job name defined in your configuration file if you wish.
-
Select Run query to see your results. Take some time here to review the logs and options for filtering logs in Grafana.
Lab Guide Complete
You have completed this lab exercise. You can view the other lab guides for this course.
Suggestions?
If you’d like to suggest changes to the GitLab System Admin Basics Hands-on Guide, please submit them via merge request.
23df6bff
)