Improve Ops Quality

Attributes

Business Goal

Mitigate future incidents by empowering developers to efficiently work on test coverage; identifying and addressing missing test infrastructure and testing categories; and filling test gaps in high-risk product categories.

Entry Criteria

1. Identify high-risk product groups.

Exit Criteria (100% completed)

1. Analyze all high-severity incidents in the past 3 months and list improvement themes needed (e.g. mixed deployments, new staging, load testing, better tests outside unit tests). => 100%

• Review package incidents review
• Pipeline Execution/Runner incidents review
• Runner executable E2E environment

1. Add runner executable E2E test environment => 100%
2. Add test coverage for high risk runner functionality => 100%
3. Set up load testing environment for PE => 100%
4. Address critical test gaps with new smoke tests for Package group => 100%
5. Update our process, documentation, on-boarding, and culture going forward. => 100%

• Create risk map for PE, Runner, and Package => 100%
• Create generalized risk mapping framework => 100%
• Create 3 end-to-end overview videos for training => 100%
• Train 1 maintainer each for Package, Runner, PE on end-to-end testing => 100%

Post working group planned improvements

Items identified to continue work on past the scope of the Working Group but necessary for ongoing improvement of Ops quality.

1. Runner staging environment
2. Mixed deployment test environment
3. Package and Container Registry performance test environments
4. Add a performance test for Pipeline Execution
5. Add e2e test contributor role to team.yml for non-SET/EPE Engineers who have been trained to or have worked with the e2e test framework

Roles and Responsibilities