Finding the one bad run in five thousand
An AI agent run is dozens of steps: model calls, tool calls, retrievals, retries. When one goes wrong, someone has to find it among thousands, see where the time went and read what failed. The Run explorer is my answer, and this is how I got there.
- 5,000
- runs, 27,865 steps (sample data)
- <30
- rows in the page at once, measured
- <100 ms
- to search every run
- 91
- tests, 28 for the explorer
01The problem
Three jobs, in order
I started from what someone debugging agents actually does, in the order they do it. First, triage: which runs failed, and are they the slow or expensive ones? Then diagnose: inside one run, where did the time go, and which step broke? Finally, read: the exact input, output and error of that step.
That order became the layout, left to right: the list for triage, the run’s steps for diagnosis, the inspector for reading. Summary numbers sit above, because they answer the question before the question: is today worse than usual?
02Principles
What I held myself to
- Dense, not noisy. Show a lot at once, but give every pixel a job. No decoration on data.
- The keyboard is the fast path. People who debug for a living live on it.
- Never colour alone. About one in twelve men can’t rely on red versus green.
- Honest data. It is sample data, generated from a fixed seed, and the page says so.
03Decisions
What I chose, and what it cost
Steps and their timing on the same row
- Chose
- A tree of steps on the left with each step’s bar on the same row, sharing one time axis.
- Instead of
- A flame graph, the usual way to draw traces: steps stacked by depth with names inside the bars.
- Because
- A flame graph is compact, but names get cut off inside short bars, and two tool calls running in parallel at the same depth land on the same row, where one can hide the other completely. On a shared row, the eye goes from a name straight to its bar, nesting is plain indentation, and parallel calls show as overlapping bars on separate rows.
- Trade-off
- It takes more vertical space: one row per step instead of one per level. For agent runs of 5 to 15 steps that is cheap; for traces with hundreds of steps it would need collapsing by default.
Failure that survives colour blindness
- Chose
- Failed steps are red, striped and labelled “▲ Error”. Run status uses a different icon shape per state, always with a text label.
- Instead of
- Red bars on their own, which is what most dashboards do.
- Because
- Under deuteranopia, the red of a failed step and the orange of a tool call become nearly the same colour. The stripes and the label carry the meaning without hue.
- Trade-off
- Slightly busier bars. I limited stripes to failures, so they stay rare enough to stand out.
Validate the palette, don’t eyeball it
- Chose
- Step colours from a categorical palette I checked with a validator against the site’s own light and dark backgrounds: worst colour-blind separation ΔE 9.2 (light) and 9.4 (dark), above the target of 8.
- Instead of
- Picking colours that looked right on my screen.
- Because
- Colour-blind safety is computable, so I computed it. The check also flagged that the light-mode aqua is only 2.82:1 against white, below the 3:1 guideline.
- Trade-off
- That aqua needs relief: every step shows its type as text next to the colour, so contrast never carries meaning alone. It also limits the chart to three categorical colours before it must fall back to labels.
Render only what is on screen
- Chose
- A virtualized list: rows are absolutely positioned inside a spacer as tall as all 5,000, and only the ones in view, plus a small buffer, exist in the page.
- Instead of
- Rendering all rows, or paginating.
- Because
- All 5,000 rows makes scrolling and filtering sluggish; pages break the flow of scanning and make “jump to the slowest” awkward. Virtualized, search filters every run in under 100 ms, including typing.
- Trade-off
- The browser’s find-in-page can’t see rows that aren’t rendered, so search has to live in the app, which it does. Screen readers would also lose their place, which is why each row carries its position (“row 37 of 5,000”) and the list announces the active row.
~25 rows in the page for 5,000 runs. Scroll or press End.
One Tab stop per region
- Chose
- The run list is a single Tab stop driven by arrow keys (a listbox), and the steps follow the WAI-ARIA tree pattern.
- Instead of
- Making every row a link or button.
- Because
- Five thousand Tab stops would make the page unusable from the keyboard. With one per region, Tab moves between list, steps and inspector, and the arrows move within them.
- Trade-off
- Custom keyboard handling had to be built and tested, and it must match what screen-reader users expect from a listbox and a tree.
| / | Search, from anywhere on the page |
|---|---|
| ↑ ↓ or j k | Previous or next run |
| Page Up / Down, Home, End | Jump through the list |
| Tab | Move from the list into the run's steps |
| → ← | Expand or collapse a step; go to its child or parent |
| Esc | Clear the search |
Connect the summary to the single run
- Chose
- A duration histogram with a marker showing where the selected run falls.
- Instead of
- A chart that only describes the whole set, or no chart.
- Because
- “Is this run slow?” only has an answer relative to the others. The marker answers it at a glance. It is a single series, so it has a title instead of a legend, a tooltip on every bar, and a hidden table for screen readers.
- Trade-off
- Equal-width bins make the long tail of slow runs look thin; that is honest, but the slowest outliers are easier to find with the Slowest sort than in the chart.
The selected run lives in the URL
- Chose
- Selecting a run updates ?run= in the address bar.
- Instead of
- Keeping the selection only in memory.
- Because
- Debugging is a team activity: “look at run_00062” should be a link. It also survives a reload.
- Trade-off
- The server renders without a selection, so the first frame shows the newest run until the browser reads the URL. With static pages that is a deliberate trade for speed.
04What review caught
Five things I got wrong first
I review my own work the way I would review someone else’s: tests first, then a script that loads every page in light, dark and at phone width, measures overflow and collects console errors, then screenshots I actually look at. These came out of that.
01 A third of runs were failing
The first data generator scored runs so harshly that 36% failed evaluation. A test I wrote to assert a believable mix of outcomes failed before I ever looked at the screen.
Fix: Reshaped the score distribution into a long tail: 79% pass, 14% fail evaluation, 7% error.
02 The header overflowed by 3 pixels
Adding the Runs link made the header 393 px wide on a 390 px phone, so the theme switcher was clipped. I only saw it because the review script measures scroll width on every page.
Fix: Tightened the link spacing on small screens; every page measures exactly 390 px again.
03 The header didn't line up with the page
The explorer is wider than the rest of the site, but the header stayed at the narrow width, so its edges didn't match.
Fix: Pages can mark themselves wide; the header follows with a :has() selector, no JavaScript.
04 Step names were cut to “clas…” on phones
The LLM / TOOL labels took the space the names needed in the narrow step column.
Fix: Below the sm breakpoint the labels are visually hidden but still read by screen readers; the colour key carries the type.
05 Timeline labels collided on phones
While reviewing this case study at phone width, the timeline’s axis labels ran together, and the full-width bar’s duration landed on top of its step name. The same bug was in the Run explorer; my earlier phone screenshots had cropped it out.
Fix: On phones the axis shows only its start and end; a duration that fits on neither side of its bar now sits inside the bar’s end.
05What’s next
What I would build next, and why
- Zoom and pan on the timeline. A 50 ms tool call next to a 5 s timeout is a sliver; you should be able to zoom into it.
- Compare two runs. The most common debugging question is “what changed between the run that worked and the one that didn’t?”
- Real traces. Recording the Returns assistant’s runs into the explorer would replace sample data with real behaviour.