Experimentation Working Group

The GitLab Experimentation Working Group aims to define the process for Product Groups at GitLab to self-service the definition, running, and results analysis of Product experiments on GitLab.com.


Property Value
Date Created September 17, 2020
Date Ended October 30, 2020
Slack #wg_experimentation (GitLab internal) - archived
Google Doc Experimentation Working Group Agenda (GitLab internal)

Business Goal

Define the process for Product Groups to self-service the definition, running, and results analysis of Product experiments on GitLab.com.

In order to keep the working group short-lived and our discussions focused, we are explicitly & exclusively focusing on experimentation within GitLab.com. This means that we will consider the discussion of experimentation on self-managed, customers.gitlab.com, about.gitlab.com, and other such entities as being out of scope for this working group.

Exit Criteria (100%)

  • A short to medium term decision and rationale for build vs buy when extending experimentation capabilities for GitLab.com.
    • Epic preference is to dogfood
  • High level requirements analysis comparing improve, build, buy options.
  • An updated and documented experiment ideation process.
  • A guide to the types of experiments supported, how these can be rolled out by user/namespace/group, and the technical process to implement this rollout.
  • How to track experiments using frontend and backend events.
  • A guide to the types of data that can be collected, where, how, and for what use.
  • A code cleanup process for completed experiments.
  • A handbook page describing how and where to write up results for completed experiments.
  • A way to view currently active experiments on GitLab.com.
  • Summary of recent progress extending existing experimentation capability

Roles and Responsibilities

Working Group Role Person Title
Executive Sponsor Anoop VP Product
Product Lead Hila Qu Director, Growth
Engineering Lead Bartek Marnane Director of Engineering, Growth
Facilitator Phil Calder Engineering Manager, Growth
Member Sam Awezec Senior PM, Growth
Member Michael Karampalas Principal PM, Growth
Member Kenny Johnston Senior Director, Product
Member Jerome Ng EM, Product Intelligence
Member Nicolas Dular Senior Fullstack Engineer
Member Dallas Reedy Fullstack Engineer
Member Jeremy Jackson Senior Fullstack Engineer
Member Kathleen Tam Manager, Data