GitLab System Administration - Hands-on Lab: Troubleshoot GitLab

TThis Hands-On Guide walks you through troubleshooting GitLab services NGINX, Puma, and Gitaly.

Estimated time to complete: 30 minutes

Objectives

The purpose of this lab is to show how to troubleshoot the GitLab server by using the gitlab-ctl command. For this lab exercise, refer to GitLab’s application architecture to review GitLab’s major services and interactions.

Task A. Troubleshoot NGINX

From a shell session on your GitLab instance, view one of the NGINX active logs.
```
sudo gitlab-ctl tail nginx/gitlab_access.log
```
Note the log adds new entries every few seconds. Most of these entries are gitlab-runner checking in with the GitLab instance via HTTP.
Stop the NGINX services.
```
sudo gitlab-ctl stop nginx
```
Attempt to navigate to http://<your_gitlab_instance> using a web browser. Your web browser should display “This site can’t be reached” or a similar message.

Check nginx/access_log again.

sudo gitlab-ctl tail nginx/gitlab_access.log

The log should no longer be updating since no clients can make HTTP/HTTPS requests to GitLab after stopping NGINX.

Verify web services aren’t running or listening anywhere.

curl -i http://localhost/nginx_status
curl -i http://localhost:80

Restart NGINX services.
```
sudo gitlab-ctl restart nginx
```
Verify the clients (e.g. the GitLab Runner) can communicate with GitLab again.
```
sudo gitlab-ctl tail nginx/gitlab_access.log
```
Verify the webserver is running and listening on port 80.
```
curl -i http://localhost/nginx_status
```

Task B. Troubleshoot Puma

Connect to your GitLab instance with a web browser. Verify you can click around to projects, the Admin Area, etc.
From a shell session on the GitLab instance, stop Puma.
```
sudo gitlab-ctl stop puma
```
Refresh GitLab in your web browser. You should immediately see an error that reads “502: GitLab is taking too much time to respond”. NGINX is running, so it can accept HTTP requests. However, when workhorse tries to pass an HTTP request to the Rails application, there is no running service to accept it.
View the GitLab Workhorse logs.
```
sudo gitlab-ctl tail gitlab-workhorse/current
```
You will see a variety of 502 and badgateway errors in the output.
View Puma logs.
```
sudo gitlab-ctl tail puma
```
You should see a message in puma/puma_stdout.log about the Puma service shutting down. You may also see errors in puma/puma_stderr.log.
Restart Puma.
```
sudo gitlab-ctl restart puma
```
View Puma’s runit log.
```
sudo gitlab-ctl tail puma/current
```
You may see output indicating Puma has restarted.
View puma/puma_stdout.log.
```
sudo gitlab-ctl tail puma/puma_stdout.log
```
You should see that Puma is running and consuming resources again.
Wait about 2 minutes, then refresh GitLab in your web browser. The application should now be reachable.
View the GitLab Workhorse log.
```
sudo gitlab-ctl tail gitlab-workhorse/current
```
Recent entries should indicate successful requests to Puma (i.e. when you reloaded GitLab in your web browser).

Task C. Troubleshoot Gitaly

Connect to your GitLab instance with a web browser.
Navigate to Menu > Projects > Your Projects.
Select New Project.
Select Create blank project.
Name the project Test project. Set project visibility to Public, and ensure Initialize repository with a README is selected. Leave all other settings as they are.
Select Create project. You will be redirected to the project’s landing page.

SSH into your GitLab Runner server.

ssh -i <SSH_HOST_KEY> root@<GITLAB_runner_host>

Download Git if it is not already installed.
```
sudo apt-get install -y git
```
Back on GitLab’s Test project, select Clone on the right side of the page.
Next to Clone with HTTP, select Copy URL.
From your GitLab Runner server, clone the repository.
```
git clone <URL_copied_from_previous_step>
```
Verify the project is correctly cloned.
```
cd ~/test-project
ls -a
git status
```
Enter exit to exit the SSH session on your GitLab Runner server.
Open an SSH session on your GitLab Omnibus instance.
```
ssh -i <SSH_HOST_KEY> root@<GITLAB_OMNIBUS_HOST>
```
Verify Gitaly is running.
```
sudo gitlab-ctl status gitaly
```
View Gitaly logs.
```
sudo gitlab-ctl tail gitaly
```
You should see many recent gRPC requests relating to Test Project (you can see the references more clearly if you grep the output, e.g. sudo gitlab-ctl tail gitaly | grep test-project).
Stop Gitaly services.
```
sudo gitlab-ctl stop gitaly
```
Verify Gitaly (and only Gitaly) is stopped.
```
sudo gitlab-ctl status
```
Navigate back to Test Project in your web browser. On the project page select the dropdown that says main under the project title. Ordinarily you would be able to select a Git branch to switch to. Now you see an error and the branch list will not load.
In the left sidebar, select Repository > Files. Note the 404 error as GitLab is unable to fetch any repository files.
Select the Back button to go back to the project landing page. Then refresh the page.
Note the additional errors. One error may read “An error occurred while fetching folder content”. GitLab cannot checkout the HEAD of the default branch because Gitaly is not running to handle the request.
Return to your GitLab instance SSH session. Check Gitaly’s recent log entries.
```
sudo gitlab-ctl tail gitaly/current
```
Note the many errors in the log output.
Enter exit to exit the SSH session on your GitLab Instance server.

SSH back into your GitLab Runner server.

ssh -i <SSH_HOST_KEY> root@<GITLAB_RUNNER_HOST>

Navigate into your cloned Test Project.
```
cd ~/test-project
```
Try to fetch any new changes from the remote repo on the GitLab instance.
```
git fetch
```
You may see error 503 in the output, indicating Gitaly is not reachable and can not handle the request.
Enter exit to exit the SSH session on your GitLab Runner server.
Re-initiate an SSH session on your GitLab Omnibus instance.
```
ssh -i <SSH_HOST_KEY> root@<GITLAB_OMNIBUS_HOST>
```
Restart Gitaly services.
```
sudo gitlab-ctl start gitaly
```
Check Gitaly logs.
```
sudo gitlab-ctl tail gitaly/current
```
The output should now show successful gRPC requests.
Return to Test Project in your web browser. Refresh the page. You should now be able to navigate around the repository, view files, and check out branches.
SSH back into your runner server and test git fetch. The command should now run without errors (there probably will not be any output since files have not changed in GitLab).

Task D. Run the gitlabsos utility

Navigate to the gitlabsos project page. Read through the project summary and README to learn the utility’s purpose and usage.
Connect to your GitLab Omnibus instance via SSH.

Clone the gitlabsos utility.

/opt/gitlab/embedded/bin/git clone --recursive https://gitlab.com/gitlab-com/support/toolbox/gitlabsos.git

Run gitlabsos.
```
cd gitlabsos
sudo ./gitlabsos.rb
```
The script may take a few minutes to run.
Once the script is finished, examine the resulting report file and its contents.
```
ls
tar -tvf gitlabsos.<GITLAB_FQDN>.<timestamp>.tar.gz
```
GitLab Support may ask for this report to assist with troubleshooting.

Lab Guide Complete

You have completed this lab exercise. You can view the other lab guides for this course.

Suggestions?

If you’d like to suggest changes to the GitLab System Admin Hands-on Guide, please submit them via merge request.

Last modified July 3, 2024: Updates to system administration labs (8f6ffe56)

View page source - Edit this page - please contribute.