CI Queue Time Stabilization Working Group

Attributes

Business Goal

Increase the stability and predictability of the CI job queue times on GitLab.com.

Intent is to:

1. Analyze and remediate situations where our CI job queue times for shared runners exceed reasonable expectations
2. Define metrics and tune alerting that more precisely correspond to the expectations of the CI job queues
3. Develop troubleshooting and investigation guides to use in cases of excessive CI job queue times
4. Perform predictive analysis on system health and growth and create issues to remediate anticipated future bottlenecks

Exit Criteria

1. Creation and tuning of metrics and alerts that trigger when system behaviour no longer matches expectations -> Done
2. 1-week of running with above mentioned tuned alerts without them going off -> Done
3. Published or updated documentation of runbook information on how to diagnose, respond, and restore abnormal behavior into being normal -> Done

Artifacts

1. Updated runbook for ci-runners service has a apdex score (latency) below SLO alert that could be triggered

Roles and Responsibilities

View page source - Edit this page - please contribute.