CI/CD Build Speed

This page contains information related to upcoming products, features, and functionality. It is important to note that the information presented is for informational purposes only. Please do not rely on this information for purchasing or planning purposes. The development, release, and timing of any products, features, or functionality may be subject to change or delay and remain at the sole discretion of GitLab Inc.
Status Authors Coach DRIs Owning Stage Created
ongoing grzesiek grzesiek gabrielengel_gl 2024-01-12

Summary

GitLab CI is a Continuous Integration platform which is widely used to run a variety of jobs, builds, pipelines. It was integrated into GitLab in September 2015 and has become one of the most beloved CI/CD solutions.

With years we’ve added a lot of new features and code to the GitLab CI platform. In order to retain the “one of the most beloved solutions” status, we also need keep attention to making it fast, reliable and secure. This design doc is describing the path towards the former: making GitLab CI fast by improving CI build speed.

Goals

  1. Establish a CI Speed Benchmark, used to compare GitLab CI to other platforms.
  2. Build CI Benchmark Framework to measure the GitLab CI speed over the long term.
  3. Describe next steps for improving GitLab CI Build Speed.

Proposal

CI Speed Benchmark

First, we plan to build a CI Speed Benchmark solution, that will allow us to run specific scenarios on various CI/CD platform and ingest results into our data warehouse.

This will make it possible to define a baseline of the CI Build Speed for many different scenarios and track the progress we, and other providers, are making over time.

The core part of this goal is to define a set of scenarios that will allow us to build a proxy metrics for build speed. For example, we could run following scenarios:

  1. Time to first byte of build log for echo "Hello World" build.
  2. Time to result to perform a CPU-intensive cryptographic operation.
  3. Time to result to perform a memory-intensive for a given amount of bytes.
  4. Time to result to build a Linux kernel.

The scenarios should be idempotent and deterministic.

In the first iteration, we will only focus on the total job execution time, and not go into detail e.g. comparing specific startup times.

CI Benchmark Framework

Once we define scenarios that we want to implement, we should build a CI Benchmark Framework. The framework will be used to run scenarios in a Continuous Integration environment, and to send the results back to our data warehouse, for analysis and comparison.

The main principles behind design choices for the framework, are:

  1. Make it CI-platform agnostic. Can run on any Continuous Integration platform.
  2. Do not depend on any specific technology that might not be available on some platforms.
  3. Easy installation setup, not requiring many dependencies. Zero-dependency would be ideal.
  4. Send results back to GitLab through an HTTP request, unless there is a better way.

Improve CI Build Speed

Once we can measure CI Build Speed, improving it can be possible. We will define the next steps for improving the speed once we have initial results.

CI Build Speed Benchmarking Framework

In order to understand how GitLab CI performs in terms of CI build speed, we plan to build CI Build Speed Benchmarking Framework.

Benchmark

In order to run the benchmark, we will:

  1. Install the benchmarking tool.
  2. Start the tool.
  3. Runs scenarios.
  4. Report results back to GitLab data warehouse.

In the first iteration, we will focus on measuring the speed of GitLab CI, GitHub Actions, and CircleCI.

Principles

There are a few design principles we should abide by:

Last modified August 23, 2024: Ensure frontmatter is consistent (e47101dc)
View page source - Edit this page - please contribute. Creative Commons License