Cloud cost attribution by feature category

Summary

Every billing line item from Google Cloud and AWS lands in the Engineering Data Warehouse, a BigQuery datawarehouse owned by Production Engineering, in the FOCUS 1.2 format (gitlab-infra-data.cloud_cost). A set of classification rules assigns each line item to a cost bucket. Inside the service_catalog bucket, the gl_service label further splits cost per service, as defined in the GitLab service catalog.

This cost allocation view is available in the Cloud Cost Utilization (CCU) Teamdeck (GitLab team members only).

That is enough to answer “what does this bucket cost?” and, through the catalog of services and teams (Engineering Portal) and Workday, “which team or department owns this cost?”. It is not enough to answer “what does a GitLab feature cost to run?”, because nothing maps cost to a product feature category.

As of October 2026, ~75% of GCP spend sits in the service_catalog bucket with a gl_service label, a smaller share sits in coarse buckets such as networking or gke_cluster, and a small long tail is unallocated.

The cost attribution approach described here covers GCP cost only and focuses on attributing service cost to feature categories. It therefore applies to the share of GCP spend that already carries a gl_service label; label coverage is being improved in parallel.

This document records how per-service cost is translated into per-feature-category cost. A service owner declares one of three models, or an explicit opt-out:

1. Service owners declare that a service maps wholly to one feature category (direct model).
2. Service owners declare a proportional model for their service instead: a usage metric that carries a feature_category label reports what share of the service’s usage each feature category consumed per day.
3. Service owners declare a weighted model where no usage metric captures the split: fixed shares per feature category, agreed by the service owner and recorded with a note.
4. Service owners declare that no feature category attribution model applies to their service, with a note saying why (no model). Its cost stays unattributed, but is reported as a deliberate decision rather than as a gap.

We introduce a declarative way to express the feature category attribution model in the metrics catalog in the runbooks repository: a service definition uses a new costAttribution field to define how service cost is broken down across feature categories, next to the SLIs and saturation points that already describe the service in PromQL.

On the analytical side, we use sqlmesh-catalog to ingest relevant metadata from Backstage (feature categories, service information). The analysis builds on the output of the existing line item classification pipeline, which produces per-bucket and per-service cost figures. The resulting cost per feature category is stored in BigQuery at daily resolution.

Motivation

The conceptual model behind this work is written up, with diagrams based on mock data, on the cost attribution concept page on the CCU Teamdeck (GitLab team members only). This document does not re-derive it.

The following vocabulary is used throughout this document:

Goals

1. A service owner can declare a feature category attribution model for their service in the metrics catalog.
2. Backstage serves as the single source of truth (SSOT) for the list of feature categories, ingested into the Engineering Data Warehouse and, eventually, used in CI validation. Backstage carried no feature categories when this work started, so building that sync was part of this project.
3. The Engineering Data Warehouse publishes a report model with daily effective cost per feature category, and explicitly reports unattributed cost.
4. The CCU Teamdeck renders that report on the existing cost allocation sankey diagram (GitLab team members only), or in another suitable form.

Non-Goals

1. Attributing whole-bucket boundaries such as networking or gke_cluster. They have no service definition to carry a model. Although the design leaves room to add this later, their cost remains unattributed.
2. Improving label coverage and allocation data quality: both are labeling and classification work, not attribution.
3. Models beyond the three specified here. The design leaves room for others, such as a model that derives shares from data the warehouse already holds, but specifies only the direct, proportional, and weighted models.
4. A second reporting dimension. Feature category is the only reporting dimension.
5. A SOX-scoped cost of goods sold (COGS) allocation. This report is an engineering view of what features cost to run, not a financial control.

Proposal

A costAttribution field on the service definition

The metrics catalog in the runbooks repository holds one jsonnet service definition per service. It is where SLIs and saturation points are declared, and where make generate turns PromQL about a service into the recording rules Mimir evaluates, checked by jsonnet validators, unit tests and the recording rule registry along the way. A feature category attribution model is a property of a service, not of the team that owns it, because one team can own several services with different models, and it is the same kind of thing as an SLI: a PromQL statement about the service that becomes a recording rule.

The design adds a costAttribution field to the service definition, carrying the service’s attribution model. The field is optional; a service without it is a boundary without a model, and its cost is unattributed. An owner who has decided that no model applies says so explicitly (see No model), so that an absent field always means that nobody has decided yet.

The declaration belongs to the service owner, and a service owner can merge a change to their own field without approval from anyone else. The contract above is a requirement on the declaration, not a review gate. The CCU team helps review declarations while the practice is new, but deliberately does not become the gatekeeper for them: an owner who has to queue behind another team to correct their own model will not keep it current. Confidence should come from tooling that checks a candidate declaration against the contract before it is merged, rather than from a required approver. The metrics catalog already supplies the first layer of that tooling: a declaration that names a missing SLI, an unknown feature category or an unsupported shape fails make generate rather than reaching Mimir.

How to write a declaration, including the validation rules and what make generate produces, is documented in runbooks under docs/metrics-catalog/cost-attribution.md. This document covers the models and their contract.

The metrics catalog is where the declaration starts, not necessarily where it stays. As service metadata moves closer to the service, into Backstage models and eventually the service’s own repository, the declaration is expected to move with it. The Engineering Portal design describes that path. The interface downstream of the declaration, the recorded metric, does not change when it moves.

The generator moves too. Under the Metropolis migration, recording rules are generated per service from service definitions held as Custom Resources, using the metrics catalog as a vendored jsonnet library, so cost attribution support has to migrate with it: the service definition resource optionally carries the costAttribution field. Metropolis is meant to be shippable to self-managed installations, which have no reason to attribute GitLab.com cloud cost. The field therefore stays optional there, and a service definition without it generates no cost attribution rules.

Direct model

Where a service serves exactly one feature category, the service owner declares a direct model:

// metrics-catalog/services/registry.jsonnet
metricsCatalog.serviceDefinition({
  type: 'registry',
  tier: 'sv',
  costAttribution: {
    model: 'direct',
    featureCategory: 'container_registry',
  },
  // ...
})

Proportional model

Where a service serves more than one feature category, the service owner declares a proportional model: a usage metric with a fixed contract.

The declaration names the metric and its selector as a metrics-catalog metric object, the same objects an SLI uses for its requestRate, and the generator derives the PromQL from it. Four forms are accepted:

Evaluated at any instant, the recorded expression returns samples that carry a feature_category label, each holding current usage intensity for that service: a per-second rate for a flow (such as query time, requests, or bytes transferred) or a gauge for the level of a stock (such as bytes stored). The declaration does not need to aggregate to one sample per category itself. The consumer does that: the generated recording rule wraps the usage in sum by (feature_category) (see the next section), so a counter that already carries the label, named with its selector, is a complete declaration.

The Mimir ruler records the summed result every five minutes, so the recorded metric holds one series per service and feature category whatever shape the usage metric has. The warehouse integrates the recorded series over an exact UTC day, which yields one usage figure per feature category per day, normalizes those figures into shares, and multiplies the boundary’s daily effective cost by them.

The contract places four requirements on the usage metric:

• Additive within a category, extensive across them. Because the consumer sums the samples within each feature_category, they must be additive: no two series may count the same usage twice. Summed across all categories, the result must then equal the service’s whole usage at that instant, so that each category’s figure is part of a meaningful total. Usage that cannot be assigned to a named feature category must be emitted as a remainder, without a feature_category label or with feature_category=unknown.
• Non-negative. Every value the metric returns must be non-negative.
• Scoped to exactly this cost boundary. The metric and selector must cover the given cost boundary and nothing else, which for a service definition means exactly one service.
• Cheap enough to evaluate every five minutes. What drives the cost is the cardinality of the underlying metric, not the size of the result. A narrow selector therefore matters more than aggregating in the declaration. The short range window is chosen by design to help, and fromSLIs avoids the raw scan altogether by reading the SLI’s already recorded rate.

The worked example is patroni, the main Postgres cluster, whose boundary is |service_catalog, (gl_service: patroni)|. Its usage statistic is client-side query time, read from controller_action:gitlab_sql_duration_seconds_sum:rate5m, an existing recorded series that rates gitlab_sql_duration_seconds_sum and sums it over pods while keeping the feature_category and db_config_name labels:

// metrics-catalog/services/patroni.jsonnet
metricsCatalog.serviceDefinition({
  type: 'patroni',
  tier: 'db',
  costAttribution: {
    model: 'proportional',
    usage: customRateQuery(|||
      controller_action:gitlab_sql_duration_seconds_sum:rate5m{db_config_name=~"main(_replica)?",env="gprd"}
    |||),
    note: |||
      Client-side Rails query time as a proxy for storage and compute together,
      covering the main primary and its replicas. Under-attributes categories
      that store much and query little, and sees no non-Rails database load.
    |||,
  },
  // ...
})

Client-side query time already carries the feature_category label, which makes patroni a good first candidate. It covers Rails web and Sidekiq processes alike. Reading the recorded series costs an instant lookup instead of a five-minute scan of every raw series, which is the cheapness requirement at work. The series it returns partition the raw counter by its remaining labels, so they are additive and the generator’s sum introduces no double counting. The example also shows that a model is usually a heuristic rather than an exact measure: vacuum, replication, and WAL work are invisible to Rails, and client-side stalls such as Ruby’s global interpreter lock count toward query time. A proxy that is good enough is the intended standard.

For the metric object forms, the generator merges the environment selector of the Mimir tenant the rule is recorded into, env="gprd" for gitlab-gprd (see the generated recording rules below). That scope is part of the model in the same sense: production usage by feature category is the proxy for the boundary’s entire cost, non-production spend included, because the cost side carries no environment dimension. This is intentional. customRateQuery is the one form the generator cannot scope, since it takes the PromQL as written, which is why the patroni declaration carries env="gprd" itself.

The db_config_name selector is where the scoping requirement bites. The patroni service is the whole cluster, primary and replicas, but db_config_name="main" selects primary traffic only, because Rails labels load-balanced replica connections main_replica. The regex main(_replica)? covers both and stays within the boundary: Prometheus regex matchers are fully anchored, and the other clusters use the ci, sec and registry config names.

Note also what the selector does not carry: no type="patroni". The usage metric is often emitted by another service, here Rails, so the generator injects no service matcher; the owner’s selector plus the environment is the whole selector. The recorded series is still keyed to the declaring service: the generated rule sets type: patroni as a static label, whatever service emitted the usage.

web shows the fromSLIs form:

// metrics-catalog/services/web.jsonnet
metricsCatalog.serviceDefinition({
  type: 'web',
  tier: 'sv',
  costAttribution: {
    model: 'proportional',
    usage: costAttribution.fromSLIs(['rails_request']),
    note: |||
      Provisional. Request count per feature category, from the same SLI that
      drives the web error budget. Treats every request as equal cost, so
      categories with few but slow requests are under-attributed. Replace with
      a duration-based proxy once one carries feature_category.
    |||,
  },
  // ...
})

Naming an SLI instead of a metric means the declaration holds no metric name or selector that can drift from the SLI, and the generated rule reads the SLI’s recorded sli_aggregations: series rather than rescanning the raw metric every five minutes. fromSLIs names SLIs of the declaring service only, and their selectors come with them: each SLI’s own requestRate selector, here job="gitlab-rails",type="web", plus the tenant’s environment. That is what keeps the declaration inside its boundary, because sli_aggregations:gitlab_sli_rails_request_total:rate_5m is recorded for several services, not only web. fromSLIs therefore takes no selector of its own. A model that needs another service’s SLI, or a narrower slice of its own, names the series and its selector explicitly through customRateQuery instead, as patroni does. Either way, the generated rule carries the declaring service as its static type label.

Reusing an SLI is a judgement call rather than a default: an SLI counts what matters for the SLO, which is not always what costs money. The note above says where this one is weak, and patroni deliberately does not reuse its SQL SLIs, because they count queries rather than query time and exclude Sidekiq, one of the largest database consumers.

Runbooks automation generates the recording rules from the declaration. Both the declared usage metric and the resulting recording rule can be inspected directly in Grafana.

Weighted model

Where no usage metric captures the split but the service owner can defend an estimate, they declare a weighted model: fixed shares per feature category.

// metrics-catalog/services/gitaly.jsonnet (illustrative, not a shipped declaration)
metricsCatalog.serviceDefinition({
  type: 'gitaly',
  tier: 'stor',
  costAttribution: {
    model: 'weighted',
    weights: {
      source_code_management: 0.7,
      continuous_integration: 0.3,
    },
    note: |||
      Agreed with the service owner in September 2026. No usage metric
      available today separates CI fetches from user-facing repository
      access. Revisit once Gitaly RPCs carry a feature category label.
    |||,
  },
  // ...
})

The weights are relative, not percentages: the warehouse normalizes them like any other usage figure, so they need only be non-negative and need not sum to one. An owner who can account for only part of a service puts the remainder on unknown, which folds into the unattributed total rather than inflating the named categories.

The note is required, as it is for a no model declaration, and it should carry the date of the estimate and say what would replace it. A weighted model is the only declaration that goes stale silently: it keeps reporting the same shares after the service has changed. Treat it as a step towards a proportional model rather than as a resting place. No service declares a weighted model yet.

The example also shows why the model belongs to the service rather than to a label on it: a weighted model splits the gitaly service across the categories it serves, rather than reading its cost as belonging to the gitaly feature category that shares its name. Today gitaly declares a direct model on the gitaly category, which is the starting point an estimate like this one would refine.

No model

Where a service owner has concluded that no feature category attribution model applies to their service, they declare that explicitly, with a note saying why:

// metrics-catalog/services/mimir.jsonnet
metricsCatalog.serviceDefinition({
  type: 'mimir',
  tier: 'inf',
  costAttribution: {
    model: 'none',
    note: |||
      Internal monitoring infrastructure that serves engineering, not a product
      feature category. Its cost is deliberately reported as unattributed.
      Revisit if Mimir starts serving a customer-facing observability feature.
    |||,
  },
  // ...
})

The note is required. It records the reasoning for the reader of the catalog, and it should say what would change the decision, because a none declared today can become a direct or proportional model later.

model: none is distinct from an absent field. An absent field means that nobody has decided yet; none means that the owner looked and decided against a model. The report is designed to keep the two apart (see the warehouse section), so the unattributed remainder reads as “still to do” and “decided” rather than as one undifferentiated total. It is also the sanctioned way to say that a service belongs to no product category: a service cannot declare a direct model on the not_owned pseudo-category, it declares none with a note instead.

The generator emits no recording rule for a none declaration, so nothing about it travels through the metric. It is meant to reach the warehouse as declaration metadata, alongside the service information the warehouse already ingests from Backstage. That path is not built yet. Until it is, the warehouse cannot tell a none declaration from an absent field, and reports both as unattributed with no model. Where declarations live and how the warehouse reads them is tracked in finops/team#539 (internal) and described in the Engineering Portal design, which moves declarations into Backstage.

Generated recording rules

make generate in the runbooks project reads every service definition that declares an attribution model and emits Mimir recording rules for all of them.

Rules are recorded into one of three Mimir tenants, gitlab-gprd, runway or gitlab-ops, which together hold the production cost that attribution applies to. A service records into exactly one of them: where a service runs in more than one, its declaration names the tenant in a tenant field. pulp, for example, records into gitlab-ops.

The rules record one metric, gitlab:feature_category:cost_attribution:usage:rate_5m, keyed by type and feature_category. Each sample is one service’s usage for one feature category at one instant. The rate_5m suffix denotes the five-minute recording cadence, not the kind of value: the sample is an intensity that may come from a counter rate, a gauge or a constant, and the warehouse treats all three the same way.

Each service gets its own rule group in its own rule file, the layout the metrics catalog already uses for SLI and saturation rules, so that services are isolated from each other. Rules inside one Mimir rule group evaluate sequentially, and a group that does not finish within its interval has its next evaluation skipped, which puts a gap in every series the group records. Since a proportional declaration is owner-authored PromQL of unbounded cost, one expensive declaration must not be able to gap another service’s samples.

The metric name follows the naming grammar runbooks already uses for recorded series. The service is identified by the type label, which is the label runbooks conventionally use for this purpose.

For patroni the generated file looks like this:

# mimir-rules/gitlab-gprd/patroni/autogenerated-gitlab-gprd-patroni-cost-attribution-usage.yml
groups:
  - name: 'Cost attribution usage: patroni'
    interval: 5m
    rules:
      - record: gitlab:feature_category:cost_attribution:usage:rate_5m
        labels:
          type: patroni
        expr: |
          sum by (feature_category) (
            controller_action:gitlab_sql_duration_seconds_sum:rate5m{db_config_name=~"main(_replica)?",env="gprd"}
          )

The sum by (feature_category) wrap is the generator’s, not the declaration’s. It collapses whatever the usage metric returns to one series per category and drops every other label, so that the recorded metric carries exactly the type label from the rule and the feature_category label from the data. It is also why the contract asks for additivity: the sum is applied to the usage metric’s output as is.

For web, declared through fromSLIs, the same rule reads the SLI’s recorded series instead of the raw metric, at the evaluation instant, in the same way the component aggregation sets read their sources:

      - record: gitlab:feature_category:cost_attribution:usage:rate_5m
        labels:
          type: web
        expr: |
          sum by (feature_category) (
            sli_aggregations:gitlab_sli_rails_request_total:rate_5m{env="gprd",job="gitlab-rails",type="web"}
          )

For direct models, the generator adds a recording rule that emits a constant value for the declared feature category. For registry, whose cost belongs wholly to container_registry, the rule reads:

      - record: gitlab:feature_category:cost_attribution:usage:rate_5m
        labels:
          type: registry
          feature_category: container_registry
        expr: vector(1)

vector(1) carries no labels of its own, so the rule’s labels block supplies both. The value of the constant is arbitrary: it is the only sample for that service and normalizes to a share of one. All three model types therefore reach the warehouse through one metric, with no separate path for the constant ones.

A weighted model is the same rule once per declared category, sorted by feature category, with the weight as the constant. For the gitaly example:

      - record: gitlab:feature_category:cost_attribution:usage:rate_5m
        labels:
          type: gitaly
          feature_category: continuous_integration
        expr: vector(0.3)
      - record: gitlab:feature_category:cost_attribution:usage:rate_5m
        labels:
          type: gitaly
          feature_category: source_code_management
        expr: vector(0.7)

For the three models that attribute cost, the recorded metric is the whole interface between the two sides. Everything upstream of it is the service owner’s to change; everything downstream of it is the same for every service. A none declaration bypasses the metric: it is catalog metadata, which the report reads once it reaches the warehouse:

flowchart TB
    subgraph owners["Declared and owned by the service owner"]
        direct["direct model<br/>a literal on one feature category"]
        prop["proportional model<br/>a usage metric carrying feature_category"]
        weighted["weighted model<br/>fixed shares, no usage metric"]
        none["no model<br/>an explicit opt-out with a note"]
    end
    ruler["Mimir ruler<br/>one generated rule per service, evaluated every 5m"]
    metric(["gitlab:feature_category:cost_attribution:usage:rate_5m<br/>keyed by type and feature_category"])
    catalog(["declaration metadata<br/>(not yet in the warehouse)"])
    subgraph framework["Owned by the cost attribution pipeline"]
        wh["Engineering Data Warehouse<br/>integrate the UTC day, normalize to shares,<br/>multiply by the boundary's daily cost"]
        rpt["Reporting<br/>cost per feature category, plus unattributed"]
    end
    direct --> ruler
    prop --> ruler
    weighted --> ruler
    ruler --> metric
    metric --> wh
    wh --> rpt
    none --> catalog
    catalog --> rpt

Adding a model for a service, or declining one, is therefore a change to that service’s definition and nothing else: no warehouse change, no new ingestion path, and no coordination with the team that owns the pipeline.

The attribution model, formally

This section states what the warehouse computes from the recorded metric and each boundary’s daily cost.

Write \(b\) for a cost boundary, \(\mathcal{M}\) for the set of boundaries that carry a direct, proportional or weighted model, \(\mathcal{N}\) for the disjoint set of boundaries whose owner declared no model, \(d\) for a UTC day, and \(\int_d\) for integration over that day. Let \(C_b(d)\) be the boundary's daily effective cost from the billing ledger. Let \(L_b\) be the set of feature_category label values the declared expression reports for that boundary, and \(x_{\ell,b}(t)\) the usage intensity of label value \(\ell\) at instant \(t\), which is what the recorded metric holds: the sum of the samples the expression returns with that label value, taking \(x_{\ell,b} \equiv 0\) for a category the boundary never reports.

Three members of \(L_b\) carry no attribution claim: unknown, which is what Rails emits when a transaction has no feature category, not_owned, which is a deliberate claim that the work belongs to no product group, and \(\varepsilon\), the empty value standing for a sample that carries no feature_category label at all, which is usage the declaration measured but could not attribute. Write \(P\) for those three and fold them onto a single unattributable element \(\bot\) by summing their intensity, leaving \(A_b\) attributable and \(K_b\) the full index set a share is defined over:

Accumulated usage over the day is the integral of that intensity, and the share normalizes it over the whole index set, the unattributable element included:

The cost attributed to one feature category on one day is then a usage-weighted split of the cost of every boundary that carries a model:

This is the whole approach:

1. the daily grain is the integration window,
2. the proportional model is the share, and
3. attribution is linear in the boundary’s cost.

Unattributable usage is summed into the denominator. The design deliberately does not distribute it pro rata across feature categories. This keeps the unattributable portion visible and explicit.

A direct model is realized as a constant expression on its declared category, the vector(1) rule shown earlier, so \(L_b = \{i_b\}\) and \(s_{i_b,b}(d) = 1\) with no special case anywhere. A weighted model is the same construction with one constant per declared category, so \(s_{i,b}(d)\) is the declared weight divided by the sum of the declared weights, constant over the day.

A boundary outside \(\mathcal{M}\) has no share vector at all, whether it sits in \(\mathcal{N}\) or in neither set.

When a boundary's usage for a day totals zero, the ratio is undefined. The convention is then \(s_{\bot,b}(d) = 1\), so a declaration that produced nothing reads as a wholly unattributed day rather than as a division by zero.

A boundary with a model that reports no samples at all for a day is, until declarations reach the warehouse, indistinguishable from a boundary without one, and is reported as unattributed with no model.

Conservation follows. The warehouse asserts it per day and service, and reconciles the attributed total to the service cost it started from. Shares are exhaustive inside a boundary, so attribution moves cost without creating or destroying it:

where the sum runs over the boundaries in scope, the services in the GCP service_catalog bucket, and the unattributed cost is reported rather than spread. Its three terms are the three kinds of unattributed cost that the report keeps apart, named below:

Until declarations reach the warehouse, the first two terms are reported together as unattributed_no_model.

Warehouse: storing and reporting the result

The attribution chain on the warehouse side consists of three models in the cloud_cost topic of sqlmesh-catalog, all running daily from 2026-09-21, the day the first recording rules landed:

1. Ingest (raw_feature_category_usage). One model reads the recorded metric once per day from each of the three tenants, with two instant queries evaluated at the midnight that ends the UTC day: sum_over_time(...[1d] @ T) and count_over_time(...[1d] @ T), summed by type and feature_category. The sum is the day’s integrated usage up to the constant five-minute step, which cancels in the share. The count records how many evaluations the day held, so a partial day is visible. Pinning to T makes the window one UTC day of evaluations regardless of when the query runs, and a re-run returns the same answer. Grain: one row per day, service, and feature category. The metric’s type label maps to the warehouse’s gl_service. This is another consumer of a path the warehouse already runs rather than a new ingestion surface: error_budget.raw_stage_group_availability reads Mimir recording rules into BigQuery daily.
2. Normalize (fct_feature_category_usage_share). A second model turns usage into shares. It folds the unattributable label values into one unattributable row, divides within each service and day so that the shares sum to exactly one, and gives that row the whole share on a day whose usage totals zero.
3. Cost (fct_feature_category_cost). A third model joins those shares to the existing daily cost per bucket and service and multiplies, for effective and billed cost. Grain: (usage_date, providername, gl_service, feature_category). It covers every service in the bucket, with or without a model. An attribution_state column carries attributed, unattributed_partial or unattributed_no_model, so unattributed cost is a row to read rather than a gap to notice. A fourth state, unattributed_declared, comes from a model: none declaration in the catalog metadata rather than from the metric, and waits on that metadata reaching the warehouse (see No model).

Only the cost model is a stable consumer contract; the two models behind it may change. The CCU Teamdeck reads it and renders the cost attribution breakdown by feature category as a sankey diagram, with unattributed cost shown alongside. A reporting view on top of it that adds feature category display names is a planned follow-up.

The ingest model is forward_only with restatement disabled, because it is the measurement. An incremental model otherwise restates earlier intervals when its logic changes, so an unrelated edit would re-query the recorded metric for days that are already published and either move a published figure or fail on series that have aged out of Mimir. This is what makes the forward-only guarantee in the known limitations hold in the models and not just in the recording rule. The two models after it are arithmetic over that durable table and stay restatable, so a bug in the fold or the division can be fixed for history without touching the measurement. The cost model also re-reads the last seven days, because billing data for a day keeps arriving after it.

Auditing the recorded usage

The recorded metric is input from outside the pipeline. Its shape is fixed by the contract above, but its content is authored by service owners and the warehouse cannot re-derive it. The pipeline therefore treats it as untrusted input, audits it daily, and publishes the findings next to the figures.

A contract violation does not stop the daily run. Fixing a bad declaration is a change in the metrics catalog, it takes effect from its deploy day forward, and it depends on the service team that owns the declaration. Halting the pipeline would withhold every other service’s figures for as long as that takes, and would not recover the affected day either way. Reporting the violation while leaving the figures visible is the better trade: the day is published, the defect is named, and the owner can act on it. One check does not follow this policy yet: negative usage is detected after folding and still blocks the run, until it moves to the individual sample (see the first check below).

The pipeline’s own invariants are a different matter and do stop the run: shares that do not sum to one, attributed cost that does not add up to the service’s cost, a service recorded into more than one tenant, or a sample without a type label. These are defects in the pipeline or the generated rules rather than in one declaration, and a figure built on them would be wrong for every service.

Four checks run against the recorded series and the catalog metadata:

1. Non-negativity. Any sample below zero is a contract violation. Today this is checked on usage after folding, and blocks the run; moving it to the individual sample, as a finding, is a follow-up.
2. Slug validity. Every reported feature_category must resolve in dim_feature_categories, with the two pseudo-categories allowlisted. This is the daily audit named in the next section.
3. Presence and continuity. A service that declares an attributing model but reports no samples for a day, or samples for only part of it, has a broken declaration, a selector that stopped matching, or a ruler that did not evaluate. Today a partial day is flagged from the evaluation count; a service that reports nothing at all can only be detected once declarations reach the warehouse.
4. Unattributable share. A high unattributable share for a service is not a violation, but it is the signal that a declaration measures usage it cannot assign, so it is reported per service rather than only folded into the total. Not implemented yet.

The findings are planned as a fourth model in the same topic, at a grain of one row per day, service, and check, carrying a severity and the offending value. The CCU Teamdeck surfaces them, so that a violation becomes a work item with an owner taken from the service entry rather than a number nobody questions. Until that model exists, the checks run as SQLMesh audits on the models themselves.

Checks in the warehouse find a problem a day late. Wrapping the usage metric in an SLI of its own would add absence alerts, and anomaly detection, in real time on the metrics side. That is a candidate follow-up rather than part of this design.

Feature categories: Backstage is the source of truth

categories.yml in gitlab-com/www-gitlab-com (data/categories.yml) remains the place where feature categories are authored. A sync projects them into the Backstage catalog at portal.gitlab.net, and the warehouse validates against Backstage rather than against the YAML: a non-blocking audit checks the slugs actually reported each day. runbooks validates each declared slug when make generate runs, against its own feature category map generated from data/stages.yml, until a validator there can read from Backstage.

In the Engineering Data Warehouse, we introduce a dimension table engineering_catalog.dim_feature_categories with slowly changing dimension (SCD) Type 2 semantics to model feature categories, sourced from the same Backstage catalog API the two existing Backstage dimensions read.

Backstage carried no feature categories when this work started, so the sync was built as part of this project. Feature categories are Domain entities in engineering-catalog, carrying both the categories from categories.yml and the maintained slugs from stages.yml, and the feature category dimension builds on the mechanics sqlmesh-catalog already used for its two existing Backstage dimensions.

Alternative Solutions

A bare featureCategory field on the service definition. Rejected. It can express only the direct model, so the proportional model would need a second, differently shaped field. It also reads as “this service is this category”, which is the conflation of resource and label that the concept page forbids: gitaly is both a gl_service and a feature category, and a bare field invites treating the two as the same thing. Nesting under costAttribution.model keeps all model types in one shape.

A cloud_cost: key in services/service-catalog.yml. Rejected after a first implementation. The service catalog is ownership metadata on its way to being generated from Backstage, and a PromQL declaration there sits outside the jsonnet validators, the unit tests, the recording rule registry and the per-tenant machinery that every other recording rule in runbooks goes through. It also forces the declaration to be a string that no tool understands until the generator runs. In the metrics catalog the usage metric is the same object an SLI is built from, so a declaration can reuse a service’s own SLIs, is checked at generation time, and lives next to the other PromQL statements about the service.

Treating an absent key as a deliberate opt-out. Rejected. Without an explicit none, a service that nobody has looked at and a service whose owner decided against a model look the same in the catalog and in the report, and the reason for the decision is recorded nowhere. The coverage view needs the two apart: the first is work still to do, the second is settled, and only the first should drive adoption. An explicit declaration with a required note costs the owner one merge request and keeps the reason next to the service.

Hard-coding proportional models in sqlmesh-catalog. Rejected. It would put a service owner’s model behind a warehouse merge request reviewed by a team that does not own the service, and it would split the two model types across two repositories with two review paths. The cost of declaring models in the catalog (a richer service definition schema) is paid once; the cost of hard-coding is paid on every model change.

The warehouse executing each declared query itself. Rejected. The warehouse would become an executor of PromQL it does not author, so a slow, wrong, or high-cardinality query would fail or delay a warehouse run and would be debugged by the warehouse team rather than by the service owner. Service owners could also not easily change their attribution model if it were defined directly in the data pipeline.

A daily recording rule. Rejected. interval: 1d is the direct route to one sample per day, but comes with significant drawbacks: the evaluation instant carries a stable but arbitrary offset within the day, hashed from the group name, so the recorded window does not align with the billing day it is multiplied against. There is one attempt per day, so any failure loses that day outright. The daily window is also the most expensive query shape over the raw metric, and the one already known to time out. Recording intensity and integrating at the consumer gives an exactly aligned day, many recording attempts instead of one, and a cheap query.

Normalizing into shares inside the ruler. Rejected. This would need two rules per service in one group, the second dividing the first by its own total, and it puts the arithmetic where it is hardest to audit and repair. This arithmetic is core to the cost attribution analysis and therefore belongs in the data pipeline.

Feature category as a resource label. Rejected. Shared infrastructure serves many categories at once, and feature ownership moves between teams. A label describes what a resource is, not who used it. The split has to come from runtime telemetry, which is what the proportional model does.

Spreading the unattributed remainder pro rata. Rejected. Distributing a residue in proportion to what is already attributed raises the coverage figure while making every number less accurate, because the unexplained part is rarely a scaled copy of the explained part. The design leaves the remainder explicitly unattributed and reports its share as the accuracy measure.

Known limitations

1. The declared, reported and validated vocabularies can drift. runbooks accepts the slugs in stages.yml, the warehouse validates against Backstage, and Rails reports whatever it is instrumented with. A slug that Rails reports before Backstage carries it, such as a newly added category, is attributed but fails the daily slug audit until the upstream vocabulary catches up. Resolving this requires a change to the input, not a warehouse allowlist.
2. Renames are not bridged. The dimension is keyed on the slug, and 13 category renames are tracked in services/stage-group-mapping-crossover.jsonnet in runbooks. A renamed category appears in the dimension as one row closed and another opened; history before the rename is reported under the old slug. Bridging renames would need the crossover map ingested as a second dimension, which is not designed here.
3. Two pseudo-categories. not_owned and unknown are emitted by telemetry and defined as a pseudo stage group in runbooks. Both are allowlisted where a slug is validated and both map to unattributed in the report, so a service cannot declare a direct model for either, though a weighted model may place a remainder weight on unknown. A service that belongs to no product category declares model: none with a note instead.
4. A weighted model is an estimate that does not decay. Its shares stay constant until the declaration changes, so a stale weighted model reports a confident number that no longer matches the service. The required note carries the date of the estimate, but nothing forces a review.
5. Whole-bucket boundaries are unmodeled. This design targets the service_catalog bucket. It leaves room to support similar attribution for other cost buckets, but does not implement it.
6. Per-category coverage is uneven. A category’s figure is complete only when all of its services carry a model, and no global coverage threshold is set. See the callout in the Summary.
7. Attribution is forward-only, by design. The generated rule produces samples from the day it is deployed, so a model’s history starts on the day its declaration is merged. The design turns that property into a guarantee: a published usage share is never restated. Dollar figures still settle over the cost model’s seven-day lookback as late billing arrives, which is the cost side settling rather than the attribution changing. A corrected declaration changes shares from its own deploy day forward, and the days already published under the old declaration keep the values they were published with. The trade is deliberate and it is not free: a declaration that was wrong leaves history that stays wrong. The auditing above is the mitigation, because it names the defective day next to the figure rather than silently repairing it later. The guarantee also settles what Mimir retention means here. Since no past day is ever recomputed, retention bounds nothing in the pipeline, and the daily figures in the warehouse are the record. Retention on gitlab:feature_category:cost_attribution:usage:rate_5m in Mimir is an operational parameter this design does not set, and none of the warehouse tables expire their partitions.
8. A none declaration is not visible in the report yet. Declarations do not reach the warehouse, so unattributed_declared is not populated and a declared opt-out reads as unattributed_no_model. How declarations reach the warehouse is open in finops/team#539 (internal).

References

1. teamdeck (internal project), the CCU cost dashboards. Its cost attribution concept page (GitLab Pages, private, mock data) carries the conceptual model this design implements, and its architecture decision records (ADRs) record the sankey semantics, the boundary between what is implemented and what is aspirational, and the decision to keep feature category as the single reporting dimension.
2. runbooks metrics-catalog/, the metrics catalog. This is where a service owner declares an attribution model on the service definition, and where the validators that check the declaration live. docs/metrics-catalog/cost-attribution.md documents how to write one. The service catalog in runbooks services/ supplies the gl_service names the cost side is split by, and its team entries carry cloud_cost.cost_owner.
3. runbooks mimir-rules/ and the jsonnet generators beside it. The naming grammar, the generate-and-commit pattern, and the CI jobs that check generated output and deploy it to Mimir all come from here.
4. sqlmesh-catalog (internal project), the Engineering Data Warehouse. It holds the billing classification this design builds on, the existing Backstage dimensions the feature category dimension is modeled after, and the three models added here (raw_feature_category_usage, fct_feature_category_usage_share, fct_feature_category_cost). error_budget.raw_stage_group_availability is the precedent for reading Mimir recording rules into BigQuery daily, and the ingest model here follows it.
5. Feature category vocabulary: data/categories.yml in www-gitlab-com.
6. engineering-catalog, the import that feeds the Engineering Portal at portal.gitlab.net. Its ADRs rule out custom entity kinds and require org-specific data to sit in a schema-validated spec.gitlab block, which shaped the entity model the feature category sync uses, and its entity model documents how services, groups and stages relate today. The modeling follow-up for product stages is production-engineering#29522; generating service-catalog.yml from Backstage is production-engineering#29526.
7. Attribution mechanism issue: finops/team#428 (internal).
8. Coverage risk and expected early outcome (the callout in the Summary): finops/team#474 (internal), responding to the coverage question raised on finops/team#428.
9. Other design documents in this section: Architecture.

View page source - Edit this page - please contribute.