Quality Engineering Tips and Tricks

Overview

This page lists a number of tips and tricks we have found useful in day to day Quality Engineering related tasks.

Running GitLab-QA pipeline against a specific GitLab release

While working on the GitLab-QA codebase, it is sometimes helpful to run the GitLab-QA pipeline against a specific release of the GitLab project. This could be due reasons such as that particular GitLab release containing specific code needed for validating the changes made in GitLab-QA. To run a GitLab-QA pipeline against a specific GitLab release, we need to know the GitLab release version created and tagged by the omnibus pipeline. This can be found by either observing the RELEASE variable in any of the package-and-test test jobs or in the last output line of the Trigger:gitlab-docker job triggered by the package-and-test job. Here is an example of what the RELEASE string looks like:

1

registry.gitlab.com/gitlab-org/omnibus-gitlab/gitlab-ee:41b42271ff37bf79066ef3089432ee28cfd81d8c

Copy this string and create a new GitLab-QA pipeline with a RELEASE variable and use the copied string as its value. Create another variable called QA_IMAGE and set it to the value that can be found in the package-and-test upstream job. Here is an example of what the QA_IMAGE string looks like:

1

 registry.gitlab.com/gitlab-org/gitlab/gitlab-ee-qa:qa-shl-use-unique-group-for-access-termination-specs

Note that the string is the same as RELEASE except for the -qa suffix on the image name and the tag which is the branch name on the GitLab project.

Now run the pipeline against the branch that has your changes.

It’s also possible to trigger a manual GitLab-QA pipeline against a specific GitLab environment using the RELEASE and QA_IMAGE variable from the package-and-test job of GitLab’s Merge Request. For example, here is the link to run a manual GitLab QA pipeline against Staging.

• Note: If registry.gitlab.com is used, you will also need to include the GITLAB_QA_CONTAINER_REGISTRY_ACCESS_TOKEN variable with the value set to the production gitlab-qa user’s access token to avoid authentication errors.

Running end-to-end test pipelines using code from a specific GitLab-QA branch

Running from a specific GitLab-QA branch against a live environment

It is often needed to test the impact of changes in the GitLab-QA codebase on gitlab-org/gitlab nightly schedule pipeline, Staging, Pre-Prod, Canary or Production pipelines. This can be achieved by manually triggering a pipeline in any of these projects and setting the QA_BRANCH variable to the branch name you are working on in the GitLab-QA project. As a result, the pipeline will checkout the specified branch and build the gitlab-qa gem instead of using the latest published gem.

Running from a specific GitLab-QA branch against a GitLab branch MR

You can checkout a test branch and edit the Gemfile to change the gitlab-qa line to install via the GitLab-QA branch.

For example in the qa/gemfile:

gem 'gitlab-qa', git: 'https://gitlab.com/gitlab-org/gitlab-qa.git', branch: '<GitLab-QA-branch>'

Make sure to also bundle install and commit the Gemfile.lock as well. Doing so successfully will allow the gitlab-qa gem to be built from a custom branch.

Determine the version, revision, branch and package deployed in GitLab environments

To find out the version, revision, branch and package deployed in gitlab.com, staging and canary environments, run this in the #chat-ops-test Slack channel:

1

/chatops run auto_deploy status

You will need access to the https://ops.gitlab.net/gitlab-com/chatops project to run /chatops commands. Ask to be added to this project in the #development Slack channel.

Determine if a change has been deployed to an environment using revision SHA

If you have a revision SHA that is deployed on an environment, you can find out if a change has made it to that environment. For example, if the revision SHA deployed on an environment is c46489109e4 and you want to find out if a change in restrict_by_ip_address_spec.rb has made it there, you can use:

1

git show c46489109e4:qa/qa/specs/features/ee/browser_ui/1_manage/group/restrict_by_ip_address_spec.rb

You can determine the revision SHA deployed on a GitLab instance by either navigating to www.example.com/help, by calling the http://www.example.com/api/v4/version API or by running /chatops run auto_deploy status in a Slack channel such as #chat-ops-test.

You can also determine if your commit has been deployed on a GitLab environment using ChatOps. For example, if your commit ref is 347e530c5b3dec60c0ce2870bc79ca4c8273604d you can run this command in a Slack channel such as #chat-ops-test:

1

/chatops run auto_deploy status 347e530c5b3dec60c0ce2870bc79ca4c8273604d

Determine the commit SHA of a nightly image

The commit SHA for a nightly pipeline can be determined in the following ways:

By visiting the /help page or calling the /api/v4/version API

Run the nightly docker image

1
2
3
4

docker run \
    --hostname localhost \
    --publish 443:443 --publish 80:80 --publish 22:22 \
gitlab/gitlab-ee:nightly

The commit SHA can be determined by visiting the http://localhost/help page after sign-in or by calling the /api/v4/version API where it is displayed as a value of the revision attribute.

By inspecting the pipeline that created the nightly image

Nightly images are created by scheduled pipelines here: https://dev.gitlab.org/gitlab/omnibus-gitlab/pipeline_schedules

You can look at the last pipeline by clicking the pipeline number for CE nightly or EE nightly under the “Last pipeline” column.

In the pipeline view click a job under the “Gitlab_com:package” column. The SHAs for GitLab Components are listed towards the end of the logs. The GitLab commit SHA is displayed as a value of gitlab-rails.

Configure VS Code for gitlab-qa debugging

The Ruby VS Code extension adds a few Ruby-related capabilities to VS Code, including the ability to debug Ruby code.

After you install the extension you can use VS Code to debug end-to-end specs running against your local GDK. You will need to add a Run configuration to launch.json. For example, the following launch.json will add a configuration named Debug Test::Instance::All current file to the list in the Run view of your Sidebar. Then, with a spec file open in the editor you can start debugging (F5) and VS Code will run the tests in the spec file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Debug Test::Instance::All current file",
      "type": "Ruby",
      "request": "launch",
      "useBundler": true,
      "pathToBundler": "<path_to_bundler>",
      "cwd": "${workspaceRoot}/qa",
      "program": "${workspaceRoot}/qa/bin/qa",
      "env": {
        "CHROME_HEADLESS": "false",
        "QA_DEBUG": "true"
      },
      "args": [
          "Test::Instance::All",
          "http://localhost:3000",
          "--",
          "${file}"
      ]
    }
  ]
}

You can include multiple configurations, and any environment variables or command line options can be included. For example, here’s one that will debug smoke tests while running them on Staging:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

{
  "name": "Debug Staging Smoke tests",
  "type": "Ruby",
  "request": "launch",
  "useBundler": true,
  "pathToBundler": "<path_to_bundler>",
  "cwd": "${workspaceRoot}/qa",
  "program": "${workspaceRoot}/qa/bin/qa",
  "env": {
    "CHROME_HEADLESS": "false",
    "QA_DEBUG": "true",
    "GITLAB_USERNAME": "gitlab-qa",
    "GITLAB_PASSWORD": "from 1Password",
    "GITLAB_QA_ACCESS_TOKEN": "from 1Password"
  },
  "args": [
      "Test::Instance::All",
      "https://staging.gitlab.com",
      "--",
      "--tag", "smoke"
  ]
}

Set up experimental auto-scaled runners

Sometimes, it’s useful to deploy auto-scaled runners to try out and compare different machine types.

To do so, follow these steps:

1. Create a service account to access shared cache in the gitlab-qa-resources GCP project.

   1. Download the service account credentials JSON file.

2. Create a VM instance to host the auto-scaled runners manager in the gitlab-qa-resources GCP project.

   1. Add the service account to the VM instance.
   2. Add you SSH key to the VM instance.
   3. Upload the service account credentials JSON file to the VM instance (e.g. scp ~/Downloads/gitlab-qa-resources-abc123.json <your-username>@VM-IP:/home/<your-username>).

3. Create a storage bucket for the shared cache in the gitlab-qa-resources GCP project.

4. SSH into the VM instance (using GCP’s Web interface).

5. Follow the installation steps for auto-scaled runners manager:

   1. Install gitlab-runner.
   2. Install Docker Machine.
   3. Register the runner
      1. Make sure to set a specific tag for the runner.
      2. Set docker+machine as the runner executor.
   4. Move the service account credentials JSON file to its final destination (using sudo): sudo mv home/<your-username>/gitlab-qa-resources-abc123.json /etc/gitlab-runner/service-account.json
   5. Edit the runner manager with something like the following configuration (make sure to check the [runners.machine] documentation):

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75

   concurrent = 500 check_interval = 0 [session_server] session_timeout = 1800 [[runners]] name = "n2d-highcpu-4 Relevant description of the runner manager" url = "https://gitlab.com/" token = "[REDACTED]" executor = "docker+machine" limit = 500 pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\"" request_concurrency = 500 environment = [ "DOCKER_TLS_CERTDIR=", "DOCKER_DRIVER=overlay2", "FF_USE_DIRECT_DOWNLOAD=true", "FF_GITLAB_REGISTRY_HELPER_IMAGE=true" ] [runners.custom_build_dir] enabled = true [runners.cache] Type = "gcs" Shared = true [runners.cache.gcs] BucketName = "BUCKET-NAME-FOR-THE-SHARED-CACHE" CredentialsFile = "/etc/gitlab-runner/service-account.json" [runners.docker] tls_verify = false image = "ruby:2.7" privileged = true disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = false shm_size = 0 volumes = [ "/cache", "/certs/client" ] [runners.machine] IdleCount = 0 IdleTime = 600 MachineDriver = "google" MachineName = "rymai-n2d-hc-4-%s" MachineOptions = [ # Additional machine options can be added using the Google Compute Engine driver. # If you experience problems with an unreachable host (ex. "Waiting for SSH"), # you should remove optional parameters to help with debugging. # https://docs.docker.com/machine/drivers/gce/ "google-project=gitlab-qa-resources", "google-zone=us-central1-a", # e.g. 'us-central1-a', full list in https://cloud.google.com/compute/docs/regions-zones/ "google-machine-type=n2d-highcpu-4", # e.g. 'n1-standard-8' "google-disk-size=50", "google-disk-type=pd-ssd", "google-label=gl_resource_type:ci_ephemeral", "google-username=cos", "google-operation-backoff-initial-interval=2", "google-use-internal-ip", "engine-registry-mirror=https://mirror.gcr.io", ] OffPeakTimezone = "" OffPeakIdleCount = 0 OffPeakIdleTime = 0 MaxBuilds = 20 [[runners.machine.autoscaling]] Periods = ["* * 8-18 * * mon-fri *"] # During the weekends IdleCount = 0 IdleTime = 600 Timezone = "UTC"

Scripts and tools for automating tasks

Toolbox

The Quality Toolbox contains several scripts that can be useful when working with GitLab end-to-end tests, such as one to generate a report of flaky tests, or one to report job success rates.

Rake tasks

The qa/tools directory contains rake tasks that perform automated tasks on a schedule (such as deleting subgroups after a test run), or that can be run as needed (such as revoking personal access tokens).

Delete Test SSH Keys

This script deletes SSH keys for a specific user. It can be executed via the delete_test_ssh_keys rake task in the qa directory.

The rake task accepts three arguments that can be used to limit the keys that are deleted, and to perform a dry run.

• The first argument, title_portion, limits keys to be deleted to those that include the string provided.
• The second argument, delete_before, limits keys to be deleted to those that were created before the given date.
• The third optional argument, dry_run, determines if the command will be executed as a dry run, summarizing the keys to be deleted. Set to true to execute as a dry run.

Two environment variables are also required:

• GITLAB_ADDRESS is the address of the target GitLab instance.
• GITLAB_QA_ACCESS_TOKEN should be a personal access token with API access and should belong to the user whose keys will be deleted.

For example, the following command will delete all SSH keys with a title that includes E2E test key: and that were created before 2020-08-02 on https://staging.gitlab.com for the user with the provided personal access token:

1

GITLAB_QA_ACCESS_TOKEN=secret GITLAB_ADDRESS=https://staging.gitlab.com bundle exec rake "delete_test_ssh_keys[E2E test key:, 2020-08-02]"

Set default password and create a personal access token

There is a rake task to set the default password (from Runtime::User.default_password) if it’s not set already, and it creates a personal access token.

This is useful when testing on a fresh GitLab instance (e.g., an omnibus-gitlab docker image) and you don’t want to have to log in and create an access token manually.

Usage example (run from the gitlab/qa directory):

1
2
3
4

$ bundle exec rake  'initialize_gitlab_auth[http://gitlab.test]'
Signing in and creating the default password for the root user if it's not set already...
Creating an API scoped access token for the root user...
Token: s8xbMN3qMjyUyQATDWgp

View page source - Edit this page - please contribute.