Test Platform Roadmap

The Test Platform sub-department roadmap is divided into multiple Tracks.

• Each track consists of smaller epics or issues.
• Each track captures the type of work grouped by a theme. Each theme is broken down into categories and phases so they can be worked on in iterations and put on a delivery timeline.
• Each Track maps directly to a track level epic. They are labeled with “~Quality” and ending with “track”

Whole department roadmap view of all tracks.

• An umbrella epic Test Platform sub-department Roadmap shows the timeline of everything.
• This epic is static and should not be edited. It is designed to only contain track level epics.

Roadmap Management

Linking epics

• The track level epics and the Test Platform sub-department Roadmap epic are the only epics allowed to have child epics.
• Each numbered list item in the roadmap should be a link either to an epic or an issue. They should be added to one of the track level epics.
• Child epics that form a track should only have issues.
• Epic structure:
  1. Department roadmap.
  2. Tracks.
  3. Everything else.

Creating new epics

• In the spirit of keeping things easily discoverable and reducing unnecessary epics, please refrain from creating new epics unless there are 5 or more issues created/scoped for that new epic.
• Please refrain from creating new tracks level epics unless it is really nessessary.

Tracks

Coverage track (roadmap view)

Coverage track (roadmap view)

Work to improve the overall test coverage.

Testcase Management

A testcase management system to document all E2E tests and their types, and link to test reports.

1. Basic Google spreadsheet to plan type of meta data. => Done
2. Use GitLab issues as testcase management using gitlab-org/quality/testcases project.
3. Integration testcases in gitlab-org/quality/testcases with automated tests. Rspec reporter that references testcase from code so they can be updated/synced dynamically.
4. Testcase Management as GitLab Native feature. Epic

Cross-browser tests

Issues: Crossbrowser & Mobile Browser testing coverage and infrastructure

1. Basic capability to run on other browsers besides chrome (IE11, Firefox).
2. Internet Explorer, and Edge coverage.
3. Other desktop browsers.
4. Mobile browser coverage.
5. Run smoke test on soon to be new stable versions of important browsers to detect issues early.

Ecosystem tests

Epic: QA: GitLab 3rd party ecosystem testing

1. Basic tests for LDAP and SAML.
2. Github and Google OAuth tests.
3. Jenkins and Jira integration tests.

Visual-diff tests

Browser screenshot visual testing to catch visual bugs. Helps with validating layout, UX, and accessibility.

Issue: End-to-end visual regression validation.

1. Lean on 3rd party tools to get something running fast and learn from the product.
2. Basic pixel to pixel comparison implemented as part of GitLab.
3. Visual coverage smoke test for all stage groups.
4. Use visual diff as a GitLab Native feature.

Performance tests

1. Basic functional tests that creates a big issue and merge request. => Done
2. First stress test environment for on-prem customers with test runs and monitoring. => Done
3. Real 10,000 user reference architecture with customer reference traffic load testing.

Security tests

1. XSS functional tests.

Mutation tests

Issue: Mutation tests

1. Implement a POC for the ruby and JavaScript code to gather data
2. Data analysis and action items
3. Depends on the above

Test planning

Test planning process.

1. Roll out testplan format and planning process.
2. Bake in test planning as part of feature planning process.
3. Iterate and come up with a shorter version of the 10 min ACC framework.

Efficiency track (roadmap view)

Efficiency track (roadmap view)

Work that increases our efficiency and productivity.

Fault Tolerance

1. Test retry.
2. Dynamic Page Object locators validation.
3. API client automatically retries on 5xx errors.

Faster Execution

Running test faster.

1. Basic parallelization via runners.

2. Parallelization at the process level for all E2E tests, exponential cost saving of CI runners.

3. Run all tests at the same time, the whole suite takes only as long as the longest test.

4. Evaluation of a subset of tests instead of running all the E2E tests depending on what changed.

   • Investigate the use of running jobs when there are changes for a given path for GitLab QA jobs.

API Usage

Use API in all E2E tests. Achieve optimal test layering with API suite more than %60 of total tests.

1. Use API calls to build the most used resources (Group, Project, User) in tests that are not focused on testing these resources behaviors. => Done
2. Use API calls in login and logout for smoke tests.
3. Implement API fabrication for all the resources that support it.
4. Use API calls for setting up all resources in every E2E tests. Roll out as a standard for every new test.

Lean pyramid

Optimize test coverage across the layers of the test pyramid, to remove redundant tests and achieve higher coverage with greater efficiency.

• Phase 1: TBD

Test data

Come up with standardized test data that can be seeded in all environments for productivity.

1. Test data curation, define a test datamodel which is static. Define better project structure for ease of debugging, more readability in automated test data output, better group, project and issue naming (not just using timestamps).

2. Script to setup testdata and clean them up.

   • Idempotent script based on API calls (E.g. adds project if missing, uses existing if exists).

3. Setup 50% of planned test data from Phase 2 in GDK, Review Apps, Staging and Canary/Production.

4. Setup 100% of planned test data from Phase 2 in GDK, Review Apps, Staging and Canary/Production.

Test results

Work that will allow us to debug tests more easily. Includes better reporting and more informative artifacts.

1. Better readability in test output.
2. Basic HTML reporting.
3. Automated reporting into the Testcase management system.
4. Automated test name update via linking of issue ID in test code.

Test readability

1. Standard Page Object method names for click navigation.
2. TBD

Triage track (roadmap view)

Triage track (roadmap view)

Work that will help us triage issues and merge requests more efficiently.

Triage reports

1. Team level triage reports. => Done
2. Group level triage reports. => Done
3. Summary report in group reports showing the amount of bugs for that group. Divide up into quadrants of severity and priority (S/P) labels Generate 5x5 grid heat map report on existing open bugs for all group triage reports. => Done
4. Bug SLA summary report in group reports. New issue first triage SLA

Refinement

1. Basic reminder for issues and merge requests. => Done

   • Merge requests that are open for a long time
   • Merge requests that do not have appropriate stage, group, and type labels.
   • Issues that are open for a long time (3 months / 6 months).
   • Merge requests that do not have any labels or milestones.

2. Enforce one team label per merge request.

3. Automatically infer stage and group label from category labels

4. Automatically infer team label from author.

5. Automatic labelling via natural language processing of issue description.

   • Investigate integrating https://gitlab.com/tromika/gitlab-issues-label-classification with GitLab Insights.

Measure track (roadmap view)

Measure track (roadmap view)

Work involving metrics that will allow us to make good data-driven decisions and report them to stakeholders early.

Insights

1. GitLab Insights prototype with initial metrics. => Done.
2. Migrate GitLab Insights into GitLab. => Done.

Bug metrics

1. Overall creation rate of bugs. => Done.
2. Creation rate of bugs displayed in each group stage’s dashboard.
3. On-prem customer incidents per month

SLA metrics

1. Mean time to resolve priority::1 and priority::2 defects.
2. New issue first triage

Releases track (roadmap view)

Releases track (roadmap view)

Work that helps in validating the release process.

Scheduling

1. Milestone refinement introduction When a milestone ends, close expired milestone and bulk reschedule unfinished work (Issues& MRs) to the next milestone => Done
2. Next iteration of closing milestones and moving issues and MRs to the next milestone

Review apps

1. Improve review apps reliability
2. Make review app a mandatory testing gate with smoke tests.
3. Shift QA tests to completely run against review apps, only orchestrated test run in the test-on-omnibus job.
4. Improve review apps usefulness, add testdata into review apps to ease testability.