Improve Ops Quality
Work on critical test gaps to mitigate future incidents and empower developers to efficiently work on test coverage
Attributes
Property | Value |
---|---|
Date Created | March 3rd, 2021 |
Target End Date | July 7th, 2021 |
Slack | #wg_improve-ops-quality (only accessible from within the company) |
Google Doc | Working Group Agenda (only accessible from within the company) |
Task Board | Issue board |
Business Goal
Mitigate future incidents by empowering developers to efficiently work on test coverage; identifying and addressing missing test infrastructure and testing categories; and filling test gaps in high-risk product categories.
Entry Criteria
- Identify high-risk product groups.
Exit Criteria (100% completed)
- Analyze all high-severity incidents in the past 3 months and list improvement themes needed (e.g. mixed deployments, new staging, load testing, better tests outside unit tests).
=> 100%
- Review package incidents review
- Pipeline Execution/Runner incidents review
- Runner executable E2E environment
- Add runner executable E2E test environment
=> 100%
- Add test coverage for high risk runner functionality
=> 100%
- Set up load testing environment for PE
=> 100%
- Address critical test gaps with new smoke tests for Package group
=> 100%
- Update our process, documentation, on-boarding, and culture going forward.
=> 100%
- Create risk map for PE, Runner, and Package
=> 100%
- Create generalized risk mapping framework
=> 100%
- Create 3 end-to-end overview videos for training
=> 100%
- Train 1 maintainer each for Package, Runner, PE on end-to-end testing
=> 100%
Post working group planned improvements
Items identified to continue work on past the scope of the Working Group but necessary for ongoing improvement of Ops quality.
- Runner staging environment
- Mixed deployment test environment
- Package and Container Registry performance test environments
- Add a performance test for Pipeline Execution
- Add
e2e test contributor
role toteam.yml
for non-SET/EPE Engineers who have been trained to or have worked with the e2e test framework
Roles and Responsibilities
Working Group Role | Person | Title |
---|---|---|
Executive Sponsor | Mek Stittri | VP of Quality |
Facilitator | Joanna Shih | Quality Engineering Manager, Ops |
Functional Lead | Sofia Vistas | Sr. Software Engineer in Test, Package:Package |
Functional Lead | Tiffany Rea | Software Engineer in Test, Verify:Pipeline Authoring |
Functional Lead | Zeff Morgan | Sr. Software Engineer in Test, Verify:Runner |
Stakeholder | Christopher Lefelhocz | VP of Development |
Stakeholder | Brent Newton | Director of Infrastructure, Reliability |
Member | Kenny Johnston | Sr. Director of Product Management, Ops |
Member | Sam Goldstein | Director of Engineering, Ops |
Member | Jackie Porter | Group Manager, Product, Verify |
Member | Dan Croft | Sr. Manager, Engineering, Continuous Delivery |
Member | Darby Frey | Sr. Manager, Engineering, Verify |
Member | Cheryl Li | Backend Engineering Manager, Verify:Pipeline Execution |
Member | Elliot Rushton | Backend Engineering Manager, Verify:Runner |
Member | Tanya Pazitny | Interim Director of Quality Engineering |
Member | Nailia Iskhakova | Sr. Software Engineer in Test, Enablement:Distribution |
Last modified September 1, 2023: Fix markdown and display errors for working-group (
59b1bdb6
)