Test Coverage

Offline environments / Airgapped GitLab QA scenario

The Test Platform Department has a GitLab QA scenario that supports offline environment / air-gapped testing. The scenario Test::Instance::Airgapped is part of GitLab QA test scenarios. The suite runs against a test environment including Gitaly Cluster which have been configured using iptables to drop traffic other than specific ports which allow our test access to the test instances.

Test run schedule

This is triggered on the gitlab-org/gitlab nightly schedule pipeline, where a near full suite of tests for CE and EE are executed (exclusions are made for tests of product features which depend on network connectivity such as importing data from external sources). Results of the gitlab-org/gitlab nightly schedule pipeline can be found in the Allure report, where test states such as failures can be filtered on. Nightly pipelines are visible at the gitlab-org/gitlab nightly schedule pipelines page (internal only). The offline environment / airgapped test job names are ce:airgapped and ee:airgapped. This is one of the pipelines monitored by the Test Platform team as part of the Test Platform Department pipeline triage on-call rotation.

Other reference guides

Secure stage has additional testing to test that analyzers can execute in an offline fashion. More information on secure tests(internal only).

Otherwise for setting up an offline environment for testing, the Getting started with an offline GitLab Installation guide can be followed. Instructions for working with secure scanners can be found in the Offline environments guide.

GitLab Upgrades

The goal of GitLab Upgrades test coverage is to ensure that a customer following the upgrade path will be successful.

To achieve the best coverage, Test Platform follows the Test Pyramid approach by shifting left to unit tests without build environments in merge requests and going up to system level testing with actual environments being built:

1. Lower level testing - Multi-version migration upgrade jobs
2. System level testing - Single-node/Docker upgrades
3. System level testing - Multi-node/Self-Managed upgrades

Multi-version migration upgrade jobs

db:migrate:multi-version-upgrade validates that the migrations pass for multi-version upgrade from the latest required upgrade stop to the author’s working branch. It allows catching migration error(s) at unit-level without building an environment. Test job runs Database migrations against PostgreSQL dump created from the latest known GitLab version stop with test data.

GitLab QA update scenario

The Test Platform Department has a Test::Omnibus::UpdateFromPrevious GitLab QA scenario that verifies update from the previous (major or minor) version to the current GitLab version (scenario code).

Test run schedule

1. Test::Omnibus::UpdateFromPrevious scenario is run with:

   • e2e:test-on-omnibus-ee / e2e:test-on-omnibus-ce jobs which executes from a scheduled pipeline every 2 hours against GitLab master.
   • e2e:test-on-omnibus-nightly job which executes from a nightly scheduled pipeline against GitLab master. Results of these jobs can be found in the Allure report, where test states such as failures can be filtered on. The update test job names are update-major, update-minor, and update-ee-to-ce.

These pipelines are monitored by the Quality Engineering team as part of the Quality Department pipeline triage on-call rotation.

Performance environments nightly upgrades

Quality team supports test performance environments listed on Reference Architecture page. These environments are built with GitLab Environment Toolkit and are upgraded daily or weekly depending on environment to the latest nightly image.

Detailed process is described on Performance and Scalability page.

Upgrade Tester

Focused on building and testing different upgrade paths using the Reference Architectures, the Upgrade Tester pipelines build and upgrade environments starting at a specified version and ending at either the latest nightly package or a specific version. For each upgrade the path used to upgrade differs depending on the start and end versions used. For example, when starting with version 16.0.0 the upgrade path would be 16.0.0, 16.1.6, 16.3.7, 16.7.7, nightly.

More information can be found within the Upgrade Tester project about the schedule and Reference Architecture types that are used for testing. Test results are reported to #qa-upgrade-results channel in Slack and monitored by Self-Managed Platform team.

Work in progress

Test Platform team is working on improving GitLab upgrades coverage and this effort is tracked in epic#12458.

View page source - Edit this page - please contribute.