Tenant Controls Team

Within Production Engineering, the Tenant Controls team owns application-level rate limiting on GitLab.com: the framework that defines, counts and observes every application rate limit, and the migration of the existing implementations onto it. For the limits themselves, see Rate Limiting, which is the source of truth for operators.

Mission

Our mission is to give GitLab.com a single way to define, configure and observe every application rate limit, so that limits can be reasoned about during an incident and changed without a deploy.

Ownership and Responsibilities

The team’s scope is set by the Unified Rate Limiting Architecture and delivered as the workstreams listed under Boards:

1. Unified SDK: labkit as the shared rate limiting SDK, and conformance of callers to it.
2. Application unification: routing the existing implementations, Rack::Attack and ApplicationRateLimiter, through that SDK without breaking changes.
3. Externalized configuration: moving limit configuration out of the application and the database into a config file that labkit loads.
4. Dynamic rule service: an external service that returns rules per request, which is what makes per-customer and per-tier limits possible.
5. Tier-aware throttles: limits that vary by plan rather than applying one number to everyone.
6. Operational ownership: the rate limiting framework in production, including its observability, dry-run and bypass behavior.

Setting the value of an individual limit is a change-managed operation shared with operators, not something this team owns unilaterally; see Rate Limiting.

Getting Assistance

• Slack: #g_tenant-controls
• Rate limit settings for a user or group: use the request template.
• Bypass requests: follow the Rate Limit Bypass Policy.

Common Links

Team Members

The following people are members of the Tenant Controls team:

How work is tracked

The team plans and tracks its work as epics and issues in the gitlab-com/gl-infra group. Every board listed below is driven entirely by labels, and GitLab does not propagate labels from an epic to its child issues, so the author applies them when the epic or issue is created.

The labels and boards are named after the work, Rate Limiting, rather than after the team, so team::Rate Limiting is the Tenant Controls team’s label.

Two principles keep the boards accurate:

The epic defines the labels. The team:: and rate-limits:: labels on an epic are the labels its child issues are expected to carry. An issue created without them appears on no board and is therefore missed in planning and status reviews.

The epic board is the inventory. The epics in scope are those carrying team::Rate Limiting; no separate list is maintained. Applying the label brings an epic into scope, and removing the label or closing the epic takes it out.

Boards

One epic board, one team-level issue board, and one issue board per workstream.

Because rate-limits:: labels are scoped, an issue can carry only one of them. The workstream boards are therefore a partition of the Delivery board: the workstream counts sum to the team count. If they stop matching, an issue is missing a workstream label, usually because an epic went on the board without one.

Labels

These labels are defined at the gitlab-com/gl-infra group and are specific to this team. For the labels every Infrastructure Platforms team uses, such as workflow-infra::, group::, section:: and infra-category::, see Infrastructure Platforms Project Management.

Naming a new workstream label

Format: rate-limits::<workstream>, where <workstream> is a kebab-case short name for the workstream, such as rate-limits::config-externalization. The Boards table lists the labels currently in use.

• Name the outcome, not the artifact. Use config-externalization rather than yaml-file, because formats change and outcomes usually do not.
• Keep it short: one to three words, close enough to the workstream’s epic title and board name that the three can be matched at a glance.
• Use US spelling, so -ization rather than -isation, even where an epic title uses British spelling. Nobody should have to guess when filtering.
• One workstream label per epic. Two would break the partition, and because the labels are scoped GitLab silently replaces one with the other anyway.
• Never invent a label on an issue. If an epic carries no rate-limits:: label, create the label and put it on the epic first.

Adding a new workstream

Three steps, in order:

1. Create the label at gitlab-com/gl-infra: rate-limits::<name>, color #5843AD, with a description that links the epic.
2. Put it on the epic, together with team::Rate Limiting. That second label is what places the epic on the epic board.
3. Carry both labels down to the epic’s child issues, and keep applying them to new issues as they are filed.

Then create an issue board scoped to the new label, if the workstream warrants one.

Standing work that fits no workstream

Small improvements and keep-the-lights-on items that belong to no workstream go to the standing quarterly epic under rate-limits::ktlo. If an item fits an existing workstream, file it there instead. The standing epic is closed and rolled forward each quarter, so its contents get a forced review rather than accumulating indefinitely.

Known gaps

• Cross-group children cannot carry these labels. They are defined at gitlab-com/gl-infra, so applying one to an issue in another group, such as gitlab-org/gitlab, would create a stray project-level label there. Leave those issues unlabeled: they are still tracked in their epics, but they do not appear on the team’s boards.
• Child epics do not inherit either. If an epic has child epics, label each child epic and put it on the board in its own right.

View page source - Edit this page - please contribute.