Bug Prioritization

Overview

This page describes the bug prioritization process performed by the quality engineering sub-department as part of the cross-functional prioritization process.

An overview of the full process can be found on the cross-functional prioritization handbook page.

Timeline

As defined in our Product Development Timeline, there are specific deadlines for prioritization. Given a milestone m that will be shipped in month M (on the release date):

• By the Monday, 19 days before the milestone begins:
  • Bug Prioritization triage report is created with a list of prioritized type::bug issues to the product managers of each groups.
• By the Monday, 12 days before the milestone begins:
  • the quad for each counterpart group (PMs, EMs, UX, and Quality) discusses which issues to include in the upcoming milestone.
  • the product manager, taking into consideration prioritization input from the quad, creates a plan of issues for the upcoming milestone.

Bug Prioritization triage report

An automated Bug Prioritization triage report is created for each group on a monthly basis containing the Top 10 open issues of ~"type::bug". The report is generated with issues that are prioritized based on the below conditions:

1. Severity based on label severity::*
2. Security vulnerabilities based on label bug::vulnerability
3. Customer issues based on label customer
4. Age of the issues(oldest)

Bug Prioritization triage report are automatically assigned to specific PMs, EMs of each group, listed in the group definition file, or directly in the triage report policy files policy YAML files.

To change who an issue gets assigned to, open a merge request for the above files. If the group definition file is changed, we’ll need to run some scripts to update the generated files as well.

Bug Prioritization Dashboard

The bug prioritization dashboard can also be used to put together a suggested list of bugs for each group. This dashboard should be revisited for each milestone, as the data will change over time.

As a general guideline, the total amount of bugs proposed should match the burndown plan from the dashboard for your specific group. This total can be found under Total Bugs in the chart titled Milestone planning (bug counts to schedule).

It is also helpful to include the Total Bug % as well, which can be found in the Milestone planning (bug % of scheduled issues) chart. This metric can help groups aim toward scheduling the appropriate percentage of bugs if they do not have the capacity to work on every bug listed.

This process is not intended to be one-size-fits-all, and groups are encouraged to find what best works for the team.

Dashboard Walkthrough

This table includes a list of the charts that can be found on the bug prioritization dashboard and a description of what each one represents.

The bug prioritization dashboard walkthrough video also offers an overview of the dashboard with examples.

Prioritization Guidelines

The following considerations can be helpful to keep in mind when determining which bugs to prioritize.

• Age (older vs. newer)
• Severity (higher vs. lower)
• Past due bugs (bugs that slipped from previous milestones, missed SLOs, etc.)
• Customer reported vs. internal (issues tagged with customer)
• “Support Priority” or “Support Efficiency” which indicate a significant number of customer tickets
• Popularity/number of users impacted
  • This will require some investigation and discussion among the quad, but there a few indicators that can help us get started:
    • Number of upvotes on an issue
    • Context from comments or customer support mentioning the number of users impacted
    • Kibana and Sentry prod logs for tracking how many users are experiencing specific errors
    • Some product groups have dashboards and other tools to track xMAU (monthly active usage) for their features. This can help us gauge how impactful a bug may be if it is affecting a feature with high usage.
• Not actively being worked on
• Not already assigned to a future milestone (unless it requires escalation)
• Not currently blocked
• Aligns with the group’s current feature work and roadmap (ex: it may be more efficient to fix bugs relating to feature A together, while feature A is being updated in a future milestone)

We should also aim to align our decisions with the below product prioritization framework

*indicates forced prioritization items with SLAs/SLOs

Forced Prioritization

Any of the items with a “*” are considered issues driven by the attached SLO or SLA and are expected to be delivered within our stated policy. There are two items that fall into Forced Prioritization:

1. Security Issues labeled with bug::vulnerability must be delivered according to the stated SLO
2. Issues supporting our product’s scale which include bug::availability with specific SLOs as well as infradev, Corrective Action, ci-decomposition::phase* that follow the stated type::bug SLO

Any issues outside of these labels are to be prioritized using cross-functional prioritization. Auto-scheduling issues based on automation or triage policies are not forced prioritization. These issues can be renegotiated for milestone delivery and reassigned by the DRI.

View page source - Edit this page - please contribute.