Engineering Fellow, Infrastructure - Andrew Newdigate
Image by Randall Munroe, xkcd.com
About me
I’m Andrew Newdigate. I’m from Cape Town in South Africa. I have a degree in computer science from the University of Cape Town, and I’ve worked as a software engineer in the healthcare, finance, telecoms, and tech sectors.
I’ve founded several companies, but the only one you may have heard of is Gitter, a developer-messaging startup I co-founded and was CTO of. GitLab acquired Gitter in 2017, and this is how I joined the company.
I lived in London in the UK for 17 years, but moved back to Cape Town in 2019. When I’m not working I love building stuff, travel, photography, music and camping and hiking in the great outdoors.
My time at GitLab
- Git Infrastructure Lead (2017): founding lead of the team that built Gitaly, the Git RPC service extracted from the Rails monolith, through to its rollout across GitLab.com and Self-Managed installations.
- Interim Director of Infrastructure (2018): managed the Gitaly and Database teams and led the migration of GitLab.com from Azure to Google Cloud, moving around 200TB of data with zero downtime using GitLab Geo.
- Staff Engineer, Infrastructure (2018): used observability data to find and remove saturation bottlenecks, single points of failure and sources of instability across the GitLab.com fleet.
- Distinguished Engineer, Infrastructure (2019): helped establish the Scalability team, drove the hardening of GitLab.com ahead of the 2021 IPO, and defined the architecture of, and led hands-on development of, GitLab Dedicated.
- Sr. Distinguished Engineer, Infrastructure (2023): architect of the GitLab Dedicated product suite, including Dedicated for Google Cloud and GitLab Dedicated for Government, which received FedRAMP Moderate authorization in 2025.
- Engineering Fellow, Infrastructure (2026): leading the effort to deliver modular components consistently across all of GitLab’s deployment targets.
Responsibilities
The Engineering Fellow, Infrastructure is a member of Platforms Engineering, and is responsible for org-wide engineering efforts. The role works with engineering leadership and individual contributors across the company to set technical direction and achieve the department’s objectives.
Current areas of focus
- Modular components and the Theseus platform vision: GitLab ships a single codebase to GitLab.com, Cells, Dedicated and Self-Managed. The Theseus platform vision defines one architectural contract for delivering components to all of these targets: applications declare what they need, and each platform supplies the implementation.
- Deployment interfaces: working with the Deployment Interfaces team on Runway and Fairway, and on how they relate to the Component Ownership Model.
- Platform standardization: shared CI components, project templates,
common-ci-tasksand Helm chart standards, which reduce duplication and let teams focus on feature delivery rather than boilerplate. - LabKit: I created LabKit to standardize logging, metrics, tracing and configuration across GitLab services. See the LabKit north star strategy and LabKit configuration design documents.
- Production-like local development: Caproni is a Kubernetes-native development environment, built on the Cloud Native GitLab Helm charts, that aims to give engineers a production-realistic local development loop. See Development Tooling.
- Delivering new services: helping teams building new services, such as Orbit, the Artifact Registry and the AI Gateway, decide how to deliver them to every deployment target through the shared platform rather than through bespoke delivery paths.
- Architecture coaching: I coach many design documents, particularly those covering infrastructure, Cells, observability and rate limiting.
Long-standing interests
- Service-level monitoring: I designed the metrics catalog that GitLab.com and GitLab Dedicated use to declaratively define their SLIs, recording rules, alerts and dashboards. It is the foundation of GitLab’s SLI/SLO framework. I was also one of the core committers on OpenSLO, a vendor-neutral, open-source SLO specification.
- Resource saturation and capacity planning: the cloud infrastructure that GitLab.com runs on is, for our purposes, almost infinitely expandable. Our ability to scale is limited by the bottlenecks and saturation points in our application and associated infrastructure. Tamland forecasts when we will hit these limits so that teams can prioritize the work ahead of time. See Capacity Planning.
- Observability: metrics, logs and traces. Having good observability allows engineers to better understand the internal state of the application. In turn, better understanding helps improve availability across our systems.
- Attribution & Ownership: GitLab’s monolithic architecture provides us with many advantages, but being able to attribute a change that leads to a particular
regression or bug is more challenging with this model. Attribution & ownership is about assigning responsibility of areas of the codebase to particular
teams (usually through
feature_categories). This is a two-way process: it’s about encouraging development teams to be involved in the execution of their code in production and infrastructure teams communicating back to the development teams. - GitLab.com Incidents: I frequently join incident calls. Primarily, I’m there to observe, so that we can iteratively improve our systems to avoid similar incidents in future.
Mentoring
Of all the attributes that go to make up a distinguished engineer, being the engineer that inspires and helps others is, to my mind, the single most important. A distinguished engineer is someone a team can build around for any project, a person who will spend time developing others and making them far better at their job then they were before.
Taken from On the Myth of the 10X Engineer and the Reality of the Distinguished Engineer, RedMonk.
Some of the ways I invest in developing other engineers:
- Staff+ and Principal+ engineering demos: I founded the Infrastructure Staff+ Engineering Demo, a bi-weekly community of practice for Staff+ engineers across Infrastructure Platforms. It later became the blueprint for the company-wide Principal+ Engineering Demo.
- Scalability Demo: for over four years, I ran a weekly public Scalability Demo on GitLab Unfiltered, which produced 180 episodes.
- Senior technical hiring: I designed GitLab’s Principal Engineering bar-raiser interview program for Staff+ technical hiring.
- WeThinkCode_: since 2022, I have mentored trainee engineers at WeThinkCode_, a South African non-profit that trains young people from underrepresented and disadvantaged backgrounds for careers in software engineering, as part of the GitLab and WeThinkCode_ partnership.
Customer Needs
A distinguished engineer will bring a relentless focus on delivering value for customers. They understand the customer need, and when it’s a new market they spend a lot of time trying to understand what the potential needs will be. They adjust course when necessary, and bring their team long with them.
Taken from On the Myth of the 10X Engineer and the Reality of the Distinguished Engineer, RedMonk.
During GitLab.com’s rapid growth phase from 2019 to 2021, I regularly joined customer and prospect engagements, including demos and pre-sales discussions, to share the lessons we learned while scaling GitLab.com.
North Star Principles
Inspiration taken from https://randsinrepose.com/archives/how-to-rands/.
A heavy bias towards action
Long meetings where we are endlessly debating potential directions are often valuable, but I believe starting is the best way to begin learning and make progress. This is not always the correct strategy. This strategy annoys those who like to debate.
Iteration
Related to the previous point, frequent, small, low-risk reversible incremental changes are almost always preferable to big risky changes. Compounding small incremental changes leads to big outcomes while keeping risk low. Big changes are nearly always a smell and I will encourage breaking them down.
Feedback
I am always willing to accept feedback, and will try my best to address it.
Talks
Occasionally I will talk at conferences. Here are some of the talks I have given:
SLOconf 2023
Tamland: How GitLab uses monitoring data for capacity forecasting
https://www.youtube.com/watch?v=EjrhMEvep-U
PromCon EU 2022
Tamland: How GitLab.com Uses Long-Term Monitoring Data For Capacity Forecasting
Tamland is GitLab’s open-source capacity planning tool. This talk demonstrates a data-driven approach to forecasting capacity challenges across hundreds of saturation points, using Thanos for long-term metric storage and Facebook’s Prophet library for predictive modeling.
https://www.youtube.com/watch?v=2R42jW98MXg
KubeCon + CloudNativeCon North America 2022
Tamland: How GitLab.com Uses Long-Term Monitoring Data For Capacity Forecasting
https://www.youtube.com/watch?v=Aa5IT3PWLO0
SLOconf 2022
Everyone can contribute to our SLO, with Bob Van Landuyt
How GitLab democratized SLO ownership across application teams, by embedding SLI definitions directly alongside the code that serves them.
https://www.youtube.com/watch?v=YXOm-8cpcyg
KubeCon + CloudNativeCon Europe 2021
How We are Dealing with Metrics at Scale on GitLab.com
Operating Prometheus at GitLab.com scale: the Jsonnet-based metrics catalog, and the SLI/SLO framework that powers GitLab.com observability.
https://www.youtube.com/watch?v=6sfr2IGJQXk
SLOconf 2021
GitLab’s journey to SLO Monitoring
GitLab’s migration from causal alerting to multi-window, multi-burn-rate SLO-based alerting, and its impact on GitLab.com reliability and on-call signal-to-noise.
https://www.youtube.com/watch?v=CbX1nZL7biQ
ScaleConf 2020
Good observability is critical to managing complex systems at scale; having quality metrics is key to this. But as system grow, the number of metrics they produce rapidly increases. Dashboards and alerts can become difficult to maintain and lead to technical debt. This talk describes the strategy we’re using at GitLab to tackle this.
https://www.youtube.com/watch?v=2zL9DymXi1E
PromCon EU 2019
Practical Capacity Planning Using Prometheus
https://www.youtube.com/watch?v=swnj6KTRg08
Devopsdays Cape Town 2019
GitLab.com’s monolithic Rails application experiences high week-on-week traffic growth. To ensure availability, GitLab’s Infrastructure team track and plan ahead in order to avoid hitting capacity limits in the application, whether these limits be CPU, database connection pools, memory, storage or any number of other finite resources. Hitting these limits could result in hours, or days, of degraded service while workarounds are put in place.
With this in mind, the team set about building a set of tools on top of Prometheus recording rules and alerts to provide them with the information they need to be sufficiently forewarned, up to a month in advance, of potential resource saturation issues.
If you’ve ever felt that you’re reactively responding to resource saturation issues, this session will provide practical examples of how we’re building resource planning into our SRE team workflow. We’ll be presenting our open-source solution and explaining how it works for us.
https://devopsdays.org/events/2019-cape-town/program/andrew-newdigate/
Monitorama PDX 2019
Practical Anomaly Detection using Prometheus
Monitorama PDX 2019 - Andrew Newdigate - Practical Anomaly Detection using Prometheus from Monitorama on Vimeo.
Google Cloud Next 2019
At Google Cloud Next ‘19, GitLab Staff Engineer Andrew Newdigate presented our migration experience and the steps we took to make it happen. Migrations seldom go as planned but we hope that others can learn from the process.
https://about.gitlab.com/blog/2019/05/02/gitlab-journey-from-azure-to-gcp/
Writing
Some of the posts I have written for the GitLab blog:
- Building GitLab with GitLab: How GitLab.com inspired Dedicated (2023)
- GitLab’s journey from Azure to GCP (2019)
- GitLab.com stability post-GCP migration (2018)
- We’re moving from Azure to Google Cloud Platform (2018)
- How a fix in Go 1.9 sped up our Gitaly service by 30x (2018)
216456f7)
