Pipeline Monitoring

End-to-end test pipelines

The test pipelines run on a scheduled basis, and their results are posted to Slack. The following are the end-to-end test pipelines that are monitored every day.

NOTE: For information on how to investigate failing tests and pipelines, check out Debugging Failing Tests and Test Pipelines

Test metrics

For visibility on the test health, we have test execution results exported to:

• an InfluxDb instance and use Grafana dashboards to visualize the results
• a Google Cloud Storage (GCS) instance which is then accessible through Snowflake

Test reports

Allure report

Another tool we have to present the test results is through the Allure test reports. Tests that run on pipelines generate Allure reports. The QA framework uses the Allure RSpec gem to generate source files for the Allure test report. An additional job in the pipeline:

• Fetches these source files from all test jobs.
• Generates and uploads the report to the S3 bucket gitlab-qa-allure-report located in AWS group project eng-quality-ops-ci-cd-shared-infra.

Each type of scheduled pipeline generates a static link for the latest test report according to its stage:

These reports are also included in the pipeline status alerts on Slack.

Test session issue

For each end-to-end pipeline that runs in the various environments we automatically test, we create a test session issue that contains the test session information. Test session issues group test results by DevOps stages, and link to test cases, and test failure issues.

Example of a test session issue: https://gitlab.com/gitlab-org/quality/testcase-sessions/-/issues/72516

Test session issues are a workaround for a missing GitLab feature. Once GitLab stores test data, we can improve failure reporting and management.

Test result issue

Each test is associated to a GitLab testcase.

  RSpec.describe 'Stage' do
    describe 'General description of the feature under test' do
      it 'test name', testcase: 'https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/:test_case_id' do
        ...
      end

      it 'another test', testcase: 'https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/:another_test_case_id' do
        ...
      end
    end
  end

The test failure stack trace and the issue stack trace are compared, and the existing issue for which the stack trace is the most similar (under a 15% difference threshold) to the test failure is used. The test failure job is then added to the failure report list in the issue. Group label is automatically inferred based on the product_group metadata of the test.

View page source - Edit this page - please contribute.