CI/CD Build Speed (time-to-result)

Attributes

Overview

We aim to create a working group to establish a repeatable process and framework to measure CI build speed and performance (time-to-result). The objective is to communicate to the market and customers a GitLab point of view, position GitLab CI build performance as the market leader, and provide guidance to customers on optimization considerations so that they can maximize developer efficiency while balancing compute costs for CI builds.

This group is primarily focused on measuring CI build speed, comparing GitLab-hosted runners to 3rd party solutions and other platform-hosted runners such as GitHub-hosted runners, Circle CI.

Context:

CI build speed and performance (time-to-result) and CI build cost efficiency are essential competitive vectors especially given the improved maturity of the CI/CD solutions in the market. The brand Q4 FY23 qualitative research study data indicates that “GitLab leads the pack in associations with speed.” However, our internal benchmark testing, (slides, report, of CI build performance on GitLab SaaS had mixed results. Therefore GitLab SaaS customers’ perception of CI build performance could be different than self-managed customers or even GitLab SaaS customers that choose to manage their own CI build environment.

Competitors like Github and Harness.io have generally focused on build time duration when discussing CI build speed and performance. While build time duration is the default performance measure, the working group must evaluate if there are other discrete measurements to include in the framework.

Additionally, new players are entering the market advocating for up to 5x GitLab SaaS runner build speed such as puzl.cloud and actuated.dev, shifting the perception of GitLabs positioning as a leader in CI build speed.

Out-of-scope

The visibility & observability of build speed for customer jobs will not be scope of this working group, as it’s part ot CI Insights from the Fleet Visibility team.

Business Outcomes

1. Establish a repeatable process and framework to measure CI build speed and performance. The process must include the frequency of measurements (quarterly, semi-annual, annual).
2. Conduct an in-depth analysis of competitors product features that are focused on improving CI build speed and efficiency.
3. Conduct a technical analysis on CI build speed and performance improvements.
4. Recommendation to product leadership regarding new investment or prioritization of features to improve GitLab CI competitiveness specific to build speed and performance.
5. Establish comprehensive material for customers on how to improve CI build speed and cost efficiency on GitLab.
6. Review and agreement by legal on how we can communicate CI build speed data externally in blog posts or other marketing materials.

Exit Criteria

1. CI build speed benchmark process codified in the handbook
2. Internal charts for continuous monitoring of CI build speed
3. Initial blog post published on how to improve GitLab CI build speed

Roles and Responsibilities

View page source - Edit this page - please contribute.