Your Observability Bill Is a Multiplication Problem

Nobody ever shipped the change that tripled the observability bill. That is the whole problem.

Every individual instrumentation change looks harmless in review. Someone adds a pod label so they can tell which replica is slow. Someone adds customer_tier because a product manager asked. Someone swaps a counter for a histogram to get p99 latency. Each of these is one line of code, and each one is obviously correct in isolation.

The bill does not add these changes up. It multiplies them.

The arithmetic

A time series is identified by its metric name plus the exact set of label key-value pairs attached to it. Every distinct combination is a separate series, with its own index entry, its own chunk in memory, and its own line on the invoice. The Prometheus documentation puts it plainly: “every unique combination of key-value label pairs represents a new time series.”¹

That word combination is doing enormous work. The series count for one metric is the product of the cardinalities of its labels, not the sum:

series = ∏ cardinality(label_i)

Take a request-latency metric on a mid-sized service. Six HTTP methods. A hundred and twenty routes. Forty distinct status codes across the fleet. Sixty pods. None of those numbers is alarming on its own.

Twenty-four million series from one metric. The histogram is the final indignity: a classic Prometheus histogram is not one series but one series per bucket boundary, plus _sum and _count. Twelve le buckets means every one of those 1.7 million label combinations becomes fourteen stored series.

This is why cardinality budgets fail when they are expressed as “keep metrics under control.” Multiplication has no intuitive feel for it. A label with 60 values does not add 60 to anything — it multiplies everything upstream of it by 60.

Where the multiplication hides

Four sources account for nearly all of it, and only the first is widely understood.

Unbounded labels

The textbook case. Prometheus is explicit: “Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.”¹ Grafana’s example is a user_id label — one label that grows with your customer base rather than with your architecture.²

Most teams have internalised this rule for the obvious offenders. The ones that still slip through are the disguised ones: an un-normalised URL path (/orders/8823, /orders/8824, …), a raw error message used as a label value, or a version label carrying a full git SHA.

Histogram fan-out

A classic histogram trades one series for many. Native histograms, now specified in Prometheus, collapse the entire distribution back into a single series with sparse exponential buckets — “first class citizens in the Prometheus data model” rather than something “broken down into float components upon ingestion.”³

Churn

The quietest one. “Active series” is a snapshot; your index and your retained storage are a union over time.

A pod label on a Kubernetes deployment does not hold sixty values — it holds sixty values right now, and a fresh sixty after every rollout. Deploy ten times a day and that label has contributed six hundred distinct series to the retention window while every dashboard you look at reports sixty. The same applies to container_id, instance on autoscaled fleets, and any label derived from an ephemeral identity.

Datadog bills custom metrics as a monthly average of the distinct timeseries counted each hour,⁴ which means churn lands squarely on the invoice even though no single instant looks expensive.

Metrics that were never queried

Every default dashboard bundle, every library that helpfully instruments itself, every metric someone added for one incident in 2024. These carry the full multiplication cost of their labels and answer no question anyone is asking.

Why the invoice follows the index, not the bytes

It is tempting to model observability cost as a volume problem — bytes ingested, bytes retained. That intuition is wrong in a way that sends optimisation efforts in the wrong direction.

A metric data point is small. What is expensive is the identity: the inverted index entry, the label strings, the open chunk held in memory for as long as the series is active. Datadog states the consequence outright — a custom metric is “uniquely identified by a combination of a metric name and tag values (including the host tag),” and billable usage “is not impacted by data point submission frequency or the number of queries you run on your metrics.”⁴

Read that twice, because it inverts the usual optimisation instinct. Scraping twice as often is nearly free. Adding one label with 60 values is a 60× event.

The allotments make the scale concrete: Datadog includes 100 indexed custom metrics per host on Pro and 200 on Enterprise, counted across the whole infrastructure.⁴ A sixty-host Pro account is therefore entitled to 6,000 indexed custom metrics in total — against which the single 24-million-series metric above is roughly four thousand times over budget, on its own.

Self-hosted Prometheus bills you in RAM and index pressure instead of dollars, but the shape of the cost is identical.

The practical consequence: compressing or shortening your metrics will not save you. Removing a dimension will.

flowchart LR
A["Application SDK<br/>———————<br/>4 · high-cardinality context<br/>moves to exemplars"]
B["OTel Collector · free<br/>———————<br/>2 · drop what nothing queries<br/>3 · aggregate dimensions away"]
C["Metrics backend · metered<br/>———————<br/>1 · measure cardinality here"]
D["Dashboards<br/>and alerts"]
A --> B --> C --> D

Four moves, in order of leverage

1. Measure before you cut

Cardinality optimisation done blind is how teams delete the metric that mattered and keep six that do not. Prometheus exposes the answer directly at /api/v1/status/tsdb, which returns seriesCountByMetricName, labelValueCountByLabelName, and seriesCountByLabelValuePair — respectively, which metrics are expensive, which labels are unbounded, and which specific label values are doing the damage.⁵

The same question in PromQL, when you want it on a dashboard:

# Which metric names cost the most series?
topk(20, count by (__name__)({__name__=~".+"}))

# How many values does one label actually have?
count(count by (pod)(http_request_duration_seconds_bucket))

Run this before touching anything. In practice the distribution is brutally top-heavy: a handful of metrics usually account for most of the series count, which means the first three cuts do nearly all the work.

2. Drop what nothing queries

The bluntest and cheapest move. The OpenTelemetry Collector’s filterprocessor removes metrics before they are ever metered, using OTTL conditions:⁶

processors:
  filter/drop_unused:
    error_mode: ignore
    metrics:
      metric:
        - 'name == "runtime.jvm.buffer.count"'
        - 'IsMatch(name, "^rpc\\.server\\.duration.*") and
           resource.attributes["deployment.environment"] == "dev"'

The equivalent at the Prometheus scrape config, if you are not running a Collector:

metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'go_gc_duration_seconds.*'
    action: drop

This is reversible and low-risk, which is exactly why it should be first. The cost is that it is all-or-nothing per metric.

3. Aggregate the dimension away

More surgical, and where most of the real savings live. You usually do not need per-pod latency — you need latency, and the ability to find the bad pod when there is one. The Collector’s transformprocessor will strip an attribute and re-aggregate the datapoints behind it:⁷

processors:
  transform/reduce_cardinality:
    metric_statements:
      - context: metric
        statements:
          # Keep only the dimensions that dashboards and alerts actually use.
          - aggregate_on_attributes("sum", ["http.route", "http.response.status_code"])
            where name == "http.server.request.duration"

One warning that the documentation is emphatic about, and that bites people in production: deleting attributes without re-aggregating is not a cost optimisation, it is a correctness bug. Strip a label with delete_matching_keys and you are left with several datapoints sharing an identical identity — a violation of metric semantics that surfaces as identity conflicts or silently wrong sums downstream. Delete and aggregate are one operation, not two.⁷

Datadog’s Metrics without Limits offers the managed version of the same idea, letting you ingest everything while paying to index only an allowlist of tags.⁸

4. Move high-cardinality context out of metrics entirely

This is the move that resolves the argument rather than winning it. When someone insists they need user_id on a latency metric, they almost never want to graph per-user latency — they want to jump from a spike to the specific request that caused it.

That is what exemplars are for. An exemplar attaches a trace ID to a metric observation, and Prometheus stores them in a fixed-size circular buffer at roughly 100 bytes of memory per exemplar with a trace reference.⁹ A hundred bytes, once — versus an entire new time series, forever, multiplied by every other label on the metric.

flowchart TD
Q["New dimension<br/>requested"] --> B{"Bounded, and<br/>stable over time?"}
B -- No --> E["Exemplar → trace<br/>or a log field"]
B -- Yes --> C{"Used by an alert<br/>or a dashboard?"}
C -- No --> D["Do not add it"]
C -- Yes --> N{"Under ~100 values,<br/>and no churn?"}
N -- No --> A["Aggregate at<br/>the Collector"]
N -- Yes --> L["Make it a label"]

Applied to the 24-million-series metric from the opening, these moves compose multiplicatively in your favour for once:

The trace side of the same problem

Traces have the identical structure with different vocabulary: you cannot keep everything, so you decide what to throw away and when.

OpenTelemetry draws the line at head versus tail sampling. Head sampling decides “as early as possible,” knowing essentially nothing beyond a trace ID and a target percentage. Tail sampling decides after the spans complete, so it can keep exactly the traces you care about — the errors, the slow ones, the ones touching a specific service.¹⁰

Tail sampling is obviously better and is not free. The OpenTelemetry documentation is unusually blunt about the cost: tail samplers are “stateful systems that can accept and store a large amount of data,” potentially requiring “dozens or even hundreds of compute nodes” depending on traffic, and they may need to degrade to simpler techniques when overwhelmed.¹⁰ There is also a load-balancing constraint that catches teams out — every span of a trace must reach the same sampler instance, which means a routing tier in front of it.

So tail sampling does not eliminate the cost. It converts a storage bill into an infrastructure bill and an operational burden. That trade is usually worth making at scale and usually not worth making below it.

What breaks

Every technique above deletes information. Being specific about which information is the difference between cost engineering and self-harm.

A dropped dimension is a question you can no longer ask. Aggregating away pod means that when one replica degrades, your metrics will show a muted fleet-wide average instead of one bad host. That is an acceptable trade only if something else — traces, logs, or a lower-resolution per-pod metric with a short retention — can still answer it. Decide what that fallback is before you strip the label.

You cannot average percentiles. This is the most common way cardinality reduction quietly corrupts a dashboard. If you aggregate a pre-computed p99 across pods, the result is not the fleet p99 and has no statistical meaning at all. Percentiles must be computed from bucket counts after aggregation, never aggregated from percentiles. Native histograms and histogram_quantile() over summed buckets do this correctly; a avg(p99_latency) panel does not.

Sampling and rare events are in tension. Head sampling at 1% will miss an error that occurs in 0.1% of requests, most of the time. If your reliability model depends on catching rare failures, sampling policy is a reliability decision, not a cost decision.

Cardinality limits fail silently by default. Most backends enforce a limit by dropping what exceeds it — and what exceeds it is whatever arrived last, not whatever matters least. A limit you have not paired with an alert on rejected series is a limit that will delete your most important metric during your worst incident.

Takeaways

• Series count is a product, not a sum. A label with 60 values multiplies everything upstream of it by 60. This is why gradual instrumentation produces sudden bills.
• The index is the cost, not the bytes. Scrape faster if you need to; add a dimension only if you must. Compression saves nothing that matters.
• Measure before cutting. /api/v1/status/tsdb and topk(count by (__name__)) will tell you in two minutes what a week of guessing will not. The distribution is always top-heavy.
• Aggregate rather than delete, and never delete without re-aggregating — that is a correctness bug, not a saving.
• Most requests for a high-cardinality label are requests for a navigation path. Exemplars cost about 100 bytes and answer that need without touching the series count.
• Name the question you are giving up. Every reduction removes an answer. Write down which one, and what will answer it instead.

References