Why increase() Returns 27.5

A counter that only ever goes up by whole requests. A query asking how many requests arrived in the last minute. An answer of 27.5.

Every Prometheus user meets this eventually, usually while trying to reconcile a dashboard with a log count, and the usual reaction is to assume something is broken. Nothing is. The docs say so plainly: “it is possible to get a non-integer result even if a counter increases only by integer increments.”¹

That sentence explains that it happens. This post is about when — because the same mechanism that produces 27.5 also produces an exactly correct answer most of the time, and the difference between those two cases is something you can predict.

What rate() is actually computing

rate(), increase() and delta() are the same function with different switches: extrapolatedRate in the PromQL engine.² Given a range like [1m], it does not look at “the last minute” as a continuous thing. It looks at the samples that happen to fall inside that window — and samples land wherever the scrape interval puts them, which is almost never exactly on the window edges.

So the function measures what it can see, then scales that up to the width of the window:

// promql/functions.go
sampledInterval = float64(lastT-firstT) / 1000
durationToStart = float64(firstT-rangeStart) / 1000
durationToEnd   = float64(rangeEnd-lastT) / 1000

factor := (sampledInterval + durationToStart + durationToEnd) / sampledInterval

Extrapolating is the right default. A scrape landing 12 seconds before your window ends does not mean traffic stopped 12 seconds ago. But extrapolation is a guess, and the guess is only as good as the assumption behind it: that the series covers the whole window at a roughly constant rate.

Four windows, one counter

Take a counter incrementing exactly once per second, a 15-second scrape interval, and increase(...[1m]). I ran the published algorithm by hand for four situations — the arithmetic below is the engine’s, not an estimate.

The first two cases are the reassuring ones. With four samples spread across the window, the factor works out to (45 + 3 + 12) / 45 = 1.333, and 45 × 1.333 = 60 — the exact truth. Dropping a scrape from the middle changes the average spacing but not the endpoints, so the answer is still exactly 60. Missing scrapes do not by themselves corrupt the number.

The interesting cases are the last two, which are physically identical: a target that started 20 seconds ago, scraped twice. The only difference is what the counter read at the first scrape.

• From zero. The first sample is 0. The engine notices that a counter cannot have been negative before that, computes where the zero point must have been, and clamps the left-hand extrapolation to it.² The result is exactly 20 — correct.
• Already at 100. The first sample is 100, so the zero-point logic does not apply; the counter plainly existed before the window. The engine assumes the series extends past the first sample and extrapolates left by half the average sample spacing. Result: 27.5 for a window in which the counter demonstrably rose by 15.

That is the whole trick. increase() is not counting; it is estimating a slope and multiplying it by the window. When the series genuinely spans the window, the estimate is right. When the series begins inside the window — a new pod, a restarted target, a label value that only just appeared — the estimate has to invent the missing part, and it invents it at the rate it can see.

The clamps that stop it being worse

Two guards keep the extrapolation from running away, and both are worth knowing because they explain otherwise-baffling numbers.

flowchart TD
A["last − first<br/>+ reset correction"] --> B{"gap to boundary<br/>≥ 1.1 × avg spacing?"}
B -- "no" --> C["extrapolate to<br/>the boundary"]
B -- "yes" --> D["extrapolate only<br/>½ × avg spacing"]
D --> E{"counter, and<br/>zero point nearer?"}
E -- "yes" --> F["stop at the<br/>zero point"]
E -- "no" --> C
C --> G["× factor"]
F --> G

The first is the extrapolation threshold. If the gap between the window edge and the nearest sample is at least 1.1 times the average spacing between samples, the engine concludes the series does not cover the whole window and extrapolates only half the average spacing instead.² The source explains the reasoning: “we are assuming a more or less regular spacing between samples, and if we don’t see a sample where we would expect one, we assume the series does not cover the whole range.”² This is why case D extrapolated 7.5 seconds rather than the full 40.

The second is the zero-point clamp, which produced the correct answer in case C. Its comment is unusually direct about the motivation: “Counters cannot be negative … thereby avoiding extrapolation to negative counter values.”²

There is also a floor: with only one sample in the window and no start-timestamp information, the function returns nothing at all.² A range shorter than two scrape intervals does not produce a small number — it produces an empty result, and a panel that renders as a gap.

Counter resets, and the order that matters

Counters restart at zero when a process restarts. rate() handles that by walking the samples and adding the pre-reset value back whenever the series goes down:²

if currPoint.F < prevPoint.F {
    resultFloat += prevPoint.F
}

This only works while the individual series is still visible. Once you sum across series, a restart in one of them looks like a dip in the total — indistinguishable from traffic falling. The docs give the rule without hedging: “always take a rate() first, then aggregate. Otherwise rate() cannot detect counter resets.”¹

Two edges worth knowing

irate() is not a smoother rate(). It uses only the last two samples in the range, which makes it responsive and makes it noisy: a single slow scrape moves it a long way. The docs are blunt about where it belongs — “irate should only be used when graphing volatile, fast-moving counters. Use rate for alerts and slow-moving counters.”¹ An alert on irate() fires on sampling jitter, then resolves before anyone opens the dashboard.

A window containing both classic and native histogram samples returns nothing. During a migration a series can carry float samples before the cutover and histogram samples after, and for any window spanning that moment the engine drops the element entirely and attaches a warning rather than mixing the two.² The panel goes blank for exactly one range-width after the switch — not a broken exporter, just the one window that straddles both formats.

Why the line keeps going after the data stops

The other half of “the graph disagrees with reality” is staleness. When a query asks for a value at some instant, Prometheus looks backwards for the most recent sample within a lookback window — 5 minutes by default, set by --query.lookback-delta.³ The default lives in the engine as defaultLookbackDelta = 5 * time.Minute.⁴

Two consequences follow. A series whose target has gone away does not linger for five minutes, because Prometheus writes an explicit stale marker: “If a target scrape or rule evaluation no longer returns a sample for a time series that was previously present, this time series will be marked as stale.”³ After that, “no value is returned for that time series.”³

But a series that stops for a different reason — a pushgateway entry with its own timestamps, a metric that simply stops being incremented and then stops being exposed without the target disappearing — behaves the way people expect stale data to behave: it takes “the last value for (by default) 5 minutes before disappearing.”³

So a flat line at the right edge of a dashboard has two completely different meanings, and the graph looks the same either way.

What to do with this

Do not sum increase() for anything that has to balance. Billing, invoices, SLA credits, anything a person will reconcile against another system. Every window extrapolates independently, and windows containing new or vanishing series overshoot. Count events with logs or an exactly-once pipeline; use Prometheus to watch rates and trends.

Expect fractional answers when series are short-lived. Autoscaled pods, per-deployment labels, ephemeral job names — these produce series that begin and end inside your windows, which is exactly the condition under which extrapolation has to guess.

Keep the range at least four scrape intervals wide. Two samples is the minimum for any answer at all, and with exactly two the extrapolation has almost nothing to work with. Wider ranges make the measured portion dominate the guessed portion.

Rate first, then aggregate. Always. This one has no trade-off; the other order is simply wrong.

Remember which clamp saved you. The zero-point clamp only fires when the first sample in the window is 0. A counter you reset to zero on deploy will be extrapolated correctly. One that persists across restarts, or one whose series appears mid-window with a value already on it, will not.

Takeaways

• increase() estimates a slope and multiplies it by the window. It does not count. Fractional results are the expected output of that design.
• Extrapolation is exact when the series spans the whole window, including when scrapes are missed — the endpoints matter, not the gaps.
• It overshoots when a series starts inside the window with a non-zero value: in the worked example, 27.5 against a true 20.
• Two clamps limit the damage: extrapolate only half the average spacing when a sample is missing near the edge, and never past a counter’s zero point.
• One sample in the range returns nothing, not a small number — check your range against your scrape interval.
• Aggregating before rate() destroys counter-reset correction and turns a restart into an apparent traffic drop.
• A flat line at the edge of a graph is ambiguous: either a stale marker ended the series, or the last value is being carried for up to five minutes.

References