Improve Ops Quality

Work on critical test gaps to mitigate future incidents and empower developers to efficiently work on test coverage


Property Value
Date Created March 3rd, 2021
Target End Date July 7th, 2021
Slack #wg_improve-ops-quality (only accessible from within the company)
Google Doc Working Group Agenda (only accessible from within the company)
Task Board Issue board

Business Goal

Mitigate future incidents by empowering developers to efficiently work on test coverage; identifying and addressing missing test infrastructure and testing categories; and filling test gaps in high-risk product categories.

Entry Criteria

  1. Identify high-risk product groups.

Exit Criteria (100% completed)

  1. Analyze all high-severity incidents in the past 3 months and list improvement themes needed (e.g. mixed deployments, new staging, load testing, better tests outside unit tests). => 100%
  1. Add runner executable E2E test environment => 100%
  2. Add test coverage for high risk runner functionality => 100%
  3. Set up load testing environment for PE => 100%
  4. Address critical test gaps with new smoke tests for Package group => 100%
  5. Update our process, documentation, on-boarding, and culture going forward. => 100%

Post working group planned improvements

Items identified to continue work on past the scope of the Working Group but necessary for ongoing improvement of Ops quality.

  1. Runner staging environment
  2. Mixed deployment test environment
  3. Package and Container Registry performance test environments
  4. Add a performance test for Pipeline Execution
  5. Add e2e test contributor role to team.yml for non-SET/EPE Engineers who have been trained to or have worked with the e2e test framework

Roles and Responsibilities

Working Group Role Person Title
Executive Sponsor Mek Stittri VP of Quality
Facilitator Joanna Shih Quality Engineering Manager, Ops
Functional Lead Sofia Vistas Sr. Software Engineer in Test, Package:Package
Functional Lead Tiffany Rea Software Engineer in Test, Verify:Pipeline Authoring
Functional Lead Zeff Morgan Sr. Software Engineer in Test, Verify:Runner
Stakeholder Christopher Lefelhocz VP of Development
Stakeholder Brent Newton Director of Infrastructure, Reliability
Member Kenny Johnston Sr. Director of Product Management, Ops
Member Sam Goldstein Director of Engineering, Ops
Member Jackie Porter Group Manager, Product, Verify
Member Dan Croft Sr. Manager, Engineering, Continuous Delivery
Member Darby Frey Sr. Manager, Engineering, Verify
Member Cheryl Li Backend Engineering Manager, Verify:Pipeline Execution
Member Elliot Rushton Backend Engineering Manager, Verify:Runner
Member Tanya Pazitny Interim Director of Quality Engineering
Member Nailia Iskhakova Sr. Software Engineer in Test, Enablement:Distribution