Designing for an agent you can’t fully predict

An agent takes seconds to answer, calls tools on your behalf, can be wrong, and in this case deals with money. The Returns assistant is a chat interface for my Relay Returns agent, designed around those four facts.

9 / 9
of Relay’s original test cases pass after the port
5
visible steps for an eligible refund
2 vs 25
screen-reader updates per reply, shipped vs naive
0
refunds the agent can issue on its own

01The problem

Three moments where chat interfaces break

While it waits, a spinner says nothing: is it working, stuck, or about to time out? When it answers, the user has to decide whether to trust it, with nothing to go on but confident prose. When it fails, the usual red toast disappears along with the context.

On top of that, a returns agent touches money. Relay’s own policy, RET-01, already says eligible requests need human approval, so the interface has to make “approved by the rules” and “money has moved” impossible to confuse.

02Principles

What I held myself to

  • Show the work, then get out of the way.
  • The agent never moves money. A person does.
  • Failure is information, not an apology.
  • Calm for assistive technology. Streaming is a visual effect, not something to read aloud.
  • Honest about what it is. It runs rules, not a model, and the page says so.

03Decisions

What I chose, and what it cost

Steps appear live, then fold away

Chose
Each tool call appears as it runs, in plain language (“Looking up order RX-104”, then what it found). When the answer arrives, the steps fold into one line, “Checked · 5 steps”, that can be reopened.
Instead of
A spinner, or steps that stay expanded after the answer.
Because
A spinner gives no evidence of progress and nothing to judge the answer by. Steps that stay open push the answer down and turn every reply into a log. Folding keeps the evidence one click away without making everyone read it.
Trade-off
Reopening costs a click, and on a very fast run the steps flash past before anyone can read them.

My headphones arrived damaged. Can I get a refund for RX-104?

Press Play, then switch modes and replay to compare.

The same real request at the assistant’s real pace. Play it in each mode: with a spinner you learn nothing until the end; always expanded, the answer sits under five finished steps.

A person approves the money

Chose
An eligible refund becomes a draft card marked “Awaiting human approval”, with “No money has moved” spelled out underneath.
Instead of
Issuing the refund as soon as the rules pass.
Because
Rules can be right and the outcome still wrong: fraud, a customer with history, a mistake in the order data. The policy already requires approval; the interface just has to make that state unmistakable.
Trade-off
Customers wait for a person, so the card has to answer “is it done?” at a glance. That is why the state has its own indicator and plain words, not just a status colour.

Failure is a state you can act on

Chose
When the order service is down, the reply says what couldn’t be checked, the steps show which tool failed, and “Try again” reruns the request in place, replacing the failed reply.
Instead of
A toast that disappears, or silently retrying in the background.
Because
A toast loses the context. Silent retries hide a flaky service and make waits unpredictable. A visible failure tells the user what happened and gives them the next step.
Trade-off
In this demo the outage is permanent, so retrying fails again. That is honest about the service, but it means the retry button can’t show a recovery here.

One announcement per turn

Chose
A separate status line for screen readers says “Checking your request…” when a turn starts, and the full reply once it’s done.
Instead of
Marking the reply itself as a live region, which is the default way to make streaming text accessible.
Because
With a live region on the reply, every step label and every streamed word is a new update: 25 for a single 20-word answer. That is noise, not access.
Trade-off
Screen-reader users hear the answer only when it’s complete. They can still read the conversation log at any time.
The “Damaged headphones” request, replayed

Live region on the reply0

    One announcement per turn (shipped)0

      What reaches a screen reader for the same reply. Counts are updates, not exact speech; screen readers queue and voice them differently.

      Stop is always one key away

      Chose
      Esc, or the Send button turning into Stop, ends a reply mid-stream. A stopped reply is marked “Stopped. The reply above is incomplete.”
      Instead of
      Making people wait out an answer they no longer want.
      Because
      Agents are slow and sometimes head the wrong way. Stopping should be instant, and partial text should never look finished.
      Trade-off
      Every reply needs a “stopped” state, in the data and in the design, alongside working, streaming and done.

      Honest about what it is, built for what it could be

      Chose
      The assistant runs Relay’s policy engine, ported to TypeScript, in the browser, and the page says “no AI model, nothing leaves the page”. The UI consumes a typed stream of events (step started, step finished, text, done).
      Instead of
      Pretending it is a model, or wiring up a paid model API for a portfolio piece.
      Because
      A deterministic engine is free and safe to host publicly, and it makes every outcome testable: all nine of Relay’s original test cases pass, and the UI tests assert exact replies. A model-backed stream can emit the same events, and the interface would not change.
      Trade-off
      It only understands what the rules understand. Ask without an order ID and it can only ask you for one.

      04What review caught

      Three things that surprised me

      1. 01 The step icons were cut in half

        The steps open and close with a height animation, which clips anything outside the list. The status icons sat on the timeline line just outside the list’s edge, so mid-run they showed as half circles. Only a screenshot taken mid-run showed it; the finished state looked fine.

        Fix: Drew the line inside the list, so the icons sit fully within the clipped area.

      2. 02 Adding “Assistant” pushed the theme switcher off phones

        The header grew to 400 px on a 390 px screen, so the switcher was 10 px off the edge on every page, not just this one.

        Fix: On phones the header keeps just the name and drops “Design engineer”, so the three links and the switcher fit.

      3. 03 Enter didn’t send while a reply was streaming

        My review script typed a second message and pressed Enter before the first reply finished. Nothing was sent, and I first took it for a bug.

        Fix: It was the behaviour I wanted: the draft is held, not lost, until the reply finishes or you press Esc to stop it. I kept it and the Send button turns into Stop while busy, so the state is visible.

      05What’s next

      What I would build next

      • A real model behind the same events. Stream a model’s tool calls through the same interface and compare it with the rules on Relay’s nine test cases.
      • Record every run into the Run explorer, so the two projects share real traces instead of sample data.
      Try the assistantRead the code