GitLab CI/CD - Hands-On Lab: Investigating Broken Pipelines

This Hands-On Guide demonstrates how to troubleshoot and fix CI/CD pipelines

Objectives

  • Syntax error catching
  • Cases where no jobs run due to rules/workflows

In the next section, we are going to take a look at deploying an application using an SSH key. To start this process, we will need to add an SSH key as a variable to our pipeline. This process can sometimes cause errors due to the formatting of the SSH key. To help with implementing this process correctly, as well as learning how to troubleshoot, let’s take a look at a common error.

Task A. Setup the SSH Connection

As a part of this course, you were provided with an SSH key to use for deployments. You will need to add this SSH key to GitLab to use it during your CI/CD process. To do this:

  1. Navigate to your project in GitLab.

  2. In the left sidebar, go to Settings > CI/CD.

  3. Select Expand next to Variables.

  4. In Group variables (inherited), you will see a variable named SSH_PRIVATE_KEY

To demonstrate a common SSH related error, we will copy the SSH key and create a new variable based on its value:

  1. Select the group next to the SSH_PRIVATE_KEY variable.

  2. Select Expand next to Variables.

  3. Select the Copy icon next to the value of the SSH_PRIVATE_KEY variable .

  4. Select Add variable.

  5. Set the Type to file.

  6. In the key field, enter SSH_INVALID_KEY.

  7. In the value, paste your SSH key.

  8. Delete the new line at the end of the key value. Doing this will create an error when we try to use the key.

  9. Select Add variable.

Your new SSH key variable will now be accessible during any CI/CD jobs you run in your group. Now, let’s create a job to test the SSH connection.

  1. Navigate to your CI/CD project.

  2. Select your .gitlab-ci.yml file.

  3. Select Edit > Edit in pipeline editor.

  4. Add a new stage named deploy:

    stages:
  - test
  - build
  - run
  - release
  - deploy

  5. For this stage, we will have a single job named deploy app.The first thing this job will do is check if there is an SSH agent available, and install one if not.

    deploy app:
  stage: deploy
  script: 
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client git -y )'
    - eval $(ssh-agent -s)

  6. Next, we will setup a pem file from our SSH key variable, setting the required permissions on the file so it can be used for SSH purposes. Note that we are using the SSH_INVALID_KEY variable.

    deploy app:
  stage: deploy
  script: 
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client git -y )'
    - eval $(ssh-agent -s)
    - chmod 400 "$SSH_INVALID_KEY"
    - ssh-add "$SSH_INVALID_KEY"
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh

  7. Finally, we can add a simple SSH command to test if the connection is working.

deploy app:
  stage: deploy
  script: 
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client git -y )'
    - eval $(ssh-agent -s)
    - chmod 400 "$SSH_INVALID_KEY"
    - ssh-add "$SSH_INVALID_KEY"
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - ssh-keyscan -t rsa,ed25519 $ip >> ~/.ssh/known_hosts
    - ssh root@$ip 'ls /'

When you commit these changes, you will see an error in your deploy job. To view this error, navigate to the Build > Pipelines page and view the failed pipeline and job.

$ eval $(ssh-agent -s)
Agent pid 3211
$ chmod 400 "$SSH_PRIVATE_KEY"
$ ssh-add "$SSH_PRIVATE_KEY"
Error loading key "/builds/scottcosentinogitlab/cicd_lab_rewrite.tmp/SSH_PRIVATE_KEY": error in libcrypto

Let’s try to figure out what happened!

Task C.1. Isolate the command that causes the error

The first logical step is to isolate the command that is causing the error. We can see in the logs that the ssh-add "$SSH_INVALID_KEY" command looks to cause the error.

With the command isolated, we can consider some ways to verify the command. One option would be to try running the command locally if possible. This can help rule out potential issues with the runner. For this example, we can assume the runners are working correctly, meaning the issue lies in the actual command itself.

In these cases, often the variable/input of the command is the main source of the error. From the error message, it looks that the key is not formatted correctly. Let’s consult the documentation to see why.

Task C.2. Search the Documentation

Often, common errors will be present in our documentation with solutions to the problems. To find this error:

  1. Try searching a part of it in the documentation: error in libcrypto. The first result you get is an article titled: Using SSH keys with GitLab CI/CD.

  2. If you scroll to the Troubleshooting section of this page, you will see a section on the exact error we are facing.

The documentation tells us that the issue is not having a new line at the end of the key. Let’s try adding one and see if it fixes the problem.

  1. Return to your variables by selecting Settings > CI/CD > Variables.

  2. Select the group next to the SSH_INVALID_KEY variable.

  3. Expand the group variable section and select the Edit icon next to the SSH_INVALID_KEY variable.

  4. Add a new line to the end of the variable value, then select Save Changes.

To test if this fixes the error:

  1. Navigate back to your CI/CD project.

  2. Select Build > Pipelines from the left sidebar.

  3. Select Run pipeline.

  4. Leave all values as default and select Run pipeline again. You will now see the job complete successfully!

Lab Guide Complete

You have completed this lab exercise. You can view the other lab guides for this course.

Suggestions?

If you wish to make a change to the Hands-On Guide for GitLab CI/CD, please submit your changes via Merge Request!

Last modified November 14, 2024: Minor formatting fixes (b559de89)
View page source - Edit this page - please contribute. Creative Commons License