One fat record per request

There’s a particular silence that falls over a room when a graph goes red and everyone realizes the dashboard was built for a different disaster. Somebody scrolls. Somebody opens another tab. The dashboard keeps showing, very precisely, the thing that isn’t wrong.

I’ve been wondering why that happens so often, and I keep landing on the moment the data gets written. A metric is a summary made in advance. You decide up front which dimensions matter, the counter ticks, and everything you didn’t think to include evaporates. It’s like taking notes in a meeting by writing down only the words you expected people to say.

Logs don’t save you either, at least not the usual kind. A typical request leaves a trail of a dozen lines scattered across the output, “fetching cart,” “cache miss,” “calling payments,” each with its own timestamp and maybe a request ID if someone remembered. To answer anything you have to reassemble the request from its crumbs, and the crumbs were written as prose for a human who was going to read them one at a time. Grep is a lovely tool for one request. It’s a miserable one for a million.

The alternative that’s been gaining fans (some people are calling it observability 2.0, which I’ll mention once and then leave alone) is almost embarrassingly simple. For every request, emit one record. Make it wide. Pile on everything you might ever want to slice by, and do the adding up later, once you actually have a question. Something like this:

{
  "timestamp": "2024-11-19T17:42:03.118Z",
  "service": "checkout-api",
  "route": "POST /orders",
  "status": 502,
  "duration_ms": 1840,
  "build_sha": "9f3c2e1",
  "region": "us-east-1",
  "user_id": "u_81723",
  "account_plan": "team",
  "feature_flags": ["new_pricing_page", "fast_path"],
  "db_queries": 14,
  "db_time_ms": 1610,
  "cache_hit": false,
  "upstream": "payments",
  "upstream_status": 504,
  "retry_count": 2,
  "error": "upstream timeout"
}

Stare at that for a minute. Try to think of something you couldn’t ask of it. Just the new build? Just team plan accounts? Just fast path flag on and cache miss? If you had metrics you’d have had to guess at all of those. With the record you just filter/group and you can afford to be wrong about what matters and not suffer for it.

For me what really got it across was the cost story. One series per unique combination of label values in the time series db. Say you add route and status to a latency metric. Maybe a few hundred series. Who cares? Now add region and you multiply. Add build_sha and you multiply every time you deploy. Now add user_id and you have a series for each customer x each route x each status code and most of them are going to have one data point and live forever in the index. That’s why metric stuff either bans you for high cardinality labels or charges you a lot for them. It’s user ids and build hashes that you use for an incident. It’s like having a library where they charge you more for the books you actually use.

Events just don’t care. User id is just another column in the row and columnar storage rocks at that. It can compress each column independently so your region column (if you have five distinct values) gets super tiny. If you only read three columns for a query you only have to read three columns off disk. You pay for the number of events you keep and sort of the size of them. In the other world it seems like every time you have a new question you add a new metric, a new index, a new combination of tags, a new line on the bill that nobody knows how to explain in budget time.

Here’s kind of how it goes in practice. Raw sql. Most of them speak some form of sql.

.

SELECT build_sha,
       cache_hit,
       sum(sample_rate)                      AS requests,
       avg(duration_ms)                      AS avg_ms,
       sum(CASE WHEN status >= 500 THEN sample_rate ELSE 0 END)
         / sum(sample_rate)                  AS error_rate
FROM request_events
WHERE route = 'POST /orders'
  AND timestamp > now() - INTERVAL '1 hour'
GROUP BY build_sha, cache_hit
ORDER BY error_rate DESC;

If the top row is the new build with cache_hit = false, you’ve learned more in one query than in twenty minutes of dashboard scrolling, and you didn’t have to know in advance that cache misses on new builds were a thing to watch. Then you add upstream_status to the group by, and the story usually tells itself. (You’ll notice sample_rate sneaking in there. I’ll come back to it.)

---

It’s not as much work as you’d think based on the level of excitement. First of all, there’s some middleware that sets up an empty map at the beginning of the request. It stores it where the rest of the code can find it. Request context, thread local, what have you. As the request goes along, anything of interest updates a field. So the auth stuff puts the user in and the plan in. The flags client adds the flags it looked at. The db wrapper adds up the number of queries and the time spent. At the end of the request, the middleware adds the status and duration. And then it sends the whole map in a batch.

If you already have the OpenTelemetry libraries in use, the root span for a request is a good container for all of this. Span attributes are more formal but the same idea. And you’ll get the child spans for waterfall stuff when you need them. I see folks pit tracing vs wide events sometimes. I think it’s just the same thing at different zoom levels.

It’s mostly the dull stuff where things can go off the rails. Which is nice in a way. I’d probably monitor for the following:

• The emit call lives in the happy path, so the requests that crash or time out, the ones you most want to see, never produce an event. It belongs in a finally block or its equivalent, with whatever error information you have at that point.
• Fields drift. One service writes duration_ms, another writes latency, a third writes elapsed in seconds. A small shared library that owns the common field names saves a lot of quiet confusion later.
• Somebody adds the request body “just for debugging,” and now there are email addresses and street addresses sitting in your analytics store, plus events that are forty kilobytes each. Decide early which fields are allowed to hold anything personal.
• Work that continues after the response is sent (a background job, an async callback) gets lost unless it either emits its own event with the same request ID or the main event waits for it.

None of those are deep. They just need someone to care for a few weeks.

Some of the enthusiasm makes me squint, mostly because a lot of it comes from people selling tools built for exactly this. That doesn’t make them wrong. It just means the idea is worth holding apart from any product. A columnar store, open instrumentation, and a bit of middleware that builds the record and fires it off when the request ends will get you surprisingly far.

The old stuff doesn’t vanish, either. Alerts on a few clear service signals still do their job, and years of runbooks are wired to existing metrics. I’d keep all of it. Metrics are cheap, and they’re good at answering “is it on fire,” which you want to know every minute without scanning a billion rows. What changes is where you go to think. Dashboards tell you something is wrong, and the events are where you sit down and work out what.

Sampling is the part I’d take slowly. At some volume you can’t keep every event, and that’s fine, but the math has to survive it. Sample evenly at first, and write the sample rate onto every event, so an event kept at one in ten carries "sample_rate": 10 and counts as ten requests when you add things up. That’s what the sum(sample_rate) in the query above is doing. Averages come out right if you weight them the same way. Percentiles get fiddlier, and I’ll admit I’d have to look up how a given store handles weighted quantiles before trusting one.

Leve the clever stuff for after. First you’ve got to figure out what requests you really dont want to miss out on. Lot of errors, lot of slow requests, try to sample the uninteresting successes. Eventually it’s a really good idea to keep all the errors and all the slow requests. It’s also how you end up feeling good about analysing the dataset that mysteriously dropped all the errors, if the rules for “this is an error” happen before the status is available or if a bug skipped the sample rate in the events that get kept and so your counts are off but by an unnoticed factor. A tail sampling, where you make a decision after the request completes avoids the first case but then you have to buffer requests somewhere until you can find out what happened. It’s a wee system to keep running too.

Curiosity is the hardest thing to install. Wide events are only worth the effort if people actually go in to production and start poking it asking it weird stuff. If your team’s used to looking at a constant set of graphs for incident work. You need to give them a chance to mess around in a query tool on a peaceful afternoon. Twoam, page beeping time. Not a great way to get cozy with a new query language.

My guess is that the most interesting number a month down the line may not be any event. But the number of undialed dashboards.