Distribution Team Infrastructure and Maintenance - Build Infrastructure

Details and maintenance tasks for the Distribution Team’s build nodes

Build Infrastructure

Distribution uses GitLab CI to build GNU/Linux packages, Docker images, AWS AMIs, and other artifacts that can be used to install GitLab in user’s machines. Many of these build processes require greater computing power and resources, and hence Distribution maintains their own infrastructure to provide these resources.

Runner manager machines

The Distribution team maintains 2 runner manager machines - one to be used in GitLab.com and the other to be used in dev.gitlab.org. The configurations of runner managers are maintained in the chef-repo project, and the secrets to be used together with these configurations are maintained in Vault.

The runners are named using the syntax runners-manager-distribution-(com|dev)-(blue|green)-<count>.

NOTE: There is no blue/green deployments configured for these runner managers yet, but issue 1138 discusses using a blue/green deployment for easier upgrades of GitLab Runner in these runner manager machines. Currently, only a single “blue” runner manager is available for each of gitlab.com and dev.gitlab.org.

Build machines

Both com and dev runners spin up identical machines using the docker-machine executor with autoscale configuration. The different type of machines spun up by them are as follows:

  1. distribution-(com|dev)-amd64-runner-gcp - These runners spin up n1-highcpu-32 machines with 100GB SSD in GCP that are used for building omnibus-gitlab GNU/Linux packages for amd64 architecture.

  2. distribution-(com|dev)-arm64-runner-gcp - These runners spin up t2a-standard-8 machines with 100GB SSD in GCP that are used for building omnibus-gitlab GNU/Linux packages for arm64 architecture.

  3. distribution-(com|dev)-arm64-runner-aws - These runners spin up m6g.2xlarge machines with 100 GB SSD in AWS that are usded for building omnibus-gitlab GNU/Linux packages for armv7 architecture (Raspberry Pi). They are used because GCP’s T2A machines don’t support 32-bit instruction set set that is used to build armv7 packages while AWS’s Graviton does.

Maintenance tasks

Requirements:

  • Access to the node.
  • Access to specific folders in Chef Vault.
  • Access to merge into master on the ops chef repo. This is not usually available to team members. So after opening an MR, get it reviewed and merged by someone with access - mostly someone from Infrastructure team.

Changing version of GitLab CI runner

To be performed by any Distribution team member:

  1. Create a new merge request on the chef repo that updates the runner version

  2. Ensure that CI passes, and the MR is reviewed by another team member

  3. Merge the change into the chef repo

  4. Login to the node and run,

    sudo /root/runner_upgrade.sh

    to perform the upgrade. This will stop the chef-client service, stop the runner and cleanup the machines, run the chef-client to fetch the new version and finally, start GitLab Runner again.

Last modified August 16, 2024: Replace aliases with redirects (af33af46)
View page source - Edit this page - please contribute. Creative Commons License