Making Vantle, part 1

TL;DR

• I audited 134 of my Claude Code sessions. 47% of the moments where I had to correct or check the agent came after it had finished its work.
• 93 of those moments reported defects only my live testing caught. In 16, the agent had just told me the tests passed or it had verified the change live.
• I built a “receipt” that checks an agent’s “done” against examples agreed before the work. In a replay of 6 past bugs, it caught none of them.
• If checking an agent’s work requires reading the code, many of the people now building with agents can’t check it at all.

Why I looked

For about three months I’ve been building RiftSight as a solo dev. It’s a browser extension for Riftbound, a popular trading card game, that uses computer vision to recognize cards in gameplay videos and streams. Viewers can hover a card to see a large, sharp picture of it, which is useful for learning cards or double-checking what they do.

RiftSight started as a passion project at the request of a streamer learning the game. It has since spawned an open-source project for the card-recognition pipeline underneath (riftsight-vision), along with sister extensions for Yu-Gi-Oh! (DuelSight) and the One Piece Card Game (Kenbunshoku).

Claude wrote nearly all the code while I directed. The RiftSight repo alone has 1,901 commits (before I started squashing) and 464 merged pull requests as of November 4, 2026. Working this way opened my eyes to the sheer tsunami of what’s possible with agentic development along with the opportunities it enables.

Somewhere along the way my role changed. I stopped being the programmer and became the reviewer, the tester, the release manager and the person who restarts the stream at 1 a.m. to see whether a card tooltip moved after the latest commit. Nobody announced the promotion. There was no pizza party (or usage reset).

This isn’t just me. In April, Google said 75% of its new code is now AI-generated and approved by engineers [1]. Anthropic says that as of May, more than 80% of the code it merges is written by Claude, and that “human code review has become a new bottleneck” [2]. Its typical engineer merged 8 times as much code per day in the second quarter of 2026 as in 2024, which Anthropic itself says almost certainly overstates the real productivity gain. Writing code got cheap. Checking it didn’t.

So I asked a simple question. Where do I actually have to step in when I work with an agent, and what would it take for “done” to mean done?

Method

The data is 134 of my main Claude Code sessions from July 28 to October 22, 2026, with 2,260 messages I typed across 104 of them. I left out subagent transcripts (agents talking to agents, possibly plotting an AI uprising) and documents I pasted in.

A script pulled out my messages, with the end of the agent’s previous message for context, and matched 792 of them against 16 families of keywords such as “I meant”, “didn’t work” and “are you sure”. Each match was labeled with a category and a phase (before, during or after the agent’s work), and 594 turned out to be real correction or check moments. Keywords miss things, so a random sample of 150 unmatched messages was labeled too. 35 of them were relevant, mostly operations chores, live-testing bugs, and UI notes. Apparently I’m too polite for a “you got this wrong” detector to work on me.

That makes 629 labeled moments in 71 sessions. Extrapolating the sample to the 1,373 unmatched messages (not counting pasted documents) gives about 900 moments in total. So the keyword search alone missed roughly a third of them, and the shares below lean toward the kinds of matches it finds.

A Claude research session did all of the labeling in one pass. I set the question and spot-checked some of the labels, but nobody did a second full labeling, so there’s no measure of agreement. Treat the categories as one careful reader’s judgment, not as ground truth. Yes, I used AI to audit how much I have to check my AI. I’m aware of how that looks.

For defects, cost means my turns and the elapsed hours from my first report until a later message of mine said it worked. Elapsed hours include nights and other work. Only 29 defect episodes could be traced that far, so these are rough cost figures.

Finding 1: 10% of my steering came before the work, 47% after

Nearly half of the labeled moments came after the agent had finished its work and reported back. Only 10% came before the work started, which is when a misunderstanding is cheapest to fix.

Some of these are just life with software. Plenty of copy was technically fine and still “WAY too verbose” (my words). Others are about trust. Today I check the agent’s work in four ways.

• Screenshots. About 22% of my messages include one, mostly to show the agent what’s wrong.
• “Are you sure?” About 6% of my messages ask the agent to double-check or verify something.
• Second opinions. 42 times, I passed along another model’s review of a plan or a design.
• The merge button. I merge every PR myself, and the evidence I’m shown is mostly test counts and “build clean” lines.

The only thing standing between the agent and production is a person who mostly sees “1,454 passed” and a green check. “That’s not oversight. That’s a ceremony.” is what Claude would say. I hate to admit it, but he’s right.

Finding 2: the tests checked what the agent understood

The category the audit rated most expensive was defects that only my live testing found. By “live testing” I mean a real game, a real Twitch or YouTube stream, and a second browser window pretending to be a viewer. The audit labeled 93 messages as reports of these defects, and several can be about the same bug. 58 reported new defects, 22 reported something “fixed” that was still broken when I tested again, and 14 reported regressions.

Yes, 58 + 22 + 14 is 94. The labels can overlap. I promise I can do math (with 95% confidence).

In 16 of the 93, the agent’s previous message had just cited passing tests or a live verification.

These weren’t cheap. Of the 29 defects I could trace to a fix, the median took 5 of my turns and about 1.7 elapsed hours. One in four took 15 turns or more, one in four took about 20 hours or more, and the worst took 37 turns. A retest is a ritual of eight to ten steps. I rebuild, reload the extension, refresh the game, restart the stream, open a viewer window, take a screenshot, describe it to the agent, lose some sanity, and repeat.

Early on I asked for bigger tooltips. The agent made some tooltips bigger, just not the ones I meant, and reported the change as “verified live”. It had verified something live. Just not my thing.

The agent’s tests checked what it understood, and my test checked what I meant. When those two differ, a green check is accurate and useless at the same time.

What I built: the receipt

The idea is a side project called Vantle, and it started with a chart. You may have seen the infamous flowchart of how Slack notifications work [3].

I wanted a tool that could draw a chart like this for any codebase, straight from the source code. Each box would be a node in a graph, grouped automatically or however you like. Each chart would show one layer of abstraction, such as behavior, at whatever level of detail you need. It would come with sensible default layers, and you could customize any of them. Every diagram would be checked against the code and could be diffed when the code changes.

But a diagram of the code is only as trustworthy as the checks under it. So in October I set the charts aside to focus on the foundation—checking that the work does what was agreed. I’m still working out what shape it should take. For now it’s the receipt, and it works in four steps.

1. Before the work starts, I approve a few example rows. Each says “given X, expect Y”, or says what must stay the same.
2. The agent does the work and says it’s done.
3. A “runner” script checks each row on fresh copies of the repo and flags edited tests and files changed outside the agreed scope, so that no AI decides the verdict. (In the two trials so far, an agent did this step by hand, and the replay’s runner missed one edited test.)
4. The receipt lists each row as matched or mismatched, with its evidence and any flags.

The rows are ordinary acceptance tests. What the receipt adds is the agreement before the work, and a single record of the examples, the check results and the exceptions, made by something other than the agent that did the work.

Here’s what a simplified receipt looks like from the replay later in this post. The feature is RiftSight’s hover cards on Twitch.

In the replay, an agent wrote the rows from my request and the code, and by design I didn’t review them, so the test measured what an agent catches on its own. One row hides a miss, which I’ll get to.

Before the replay, I ran the receipt on two real changes. Neither trial found a mismatch. In the first, all 8 rows matched. The second was the first with rows I approved myself, which took me about 4 minutes. 5 of its 6 rows matched, and one live row was left for me to run by hand. The first receipt took under 5 minutes and the second about 6. Both were well-tested, predictable changes, where a receipt mostly confirms that the work is right.

With an agent doing the checking, a receipt cost 132,000 to 142,000 tokens. Running fixed checks with a script needs no model tokens, so the tokens can go into writing checks instead. We have the technology. We (mostly Claude) just need to build it.

0 of 6: the bugs lived next door

The replay was the real test, and maybe the most important part of this post.

For the replay, I took 6 past bugs where an agent had said “done” and I later found a defect. Five came from RiftSight. I added 10 controls, which were accepted changes I hadn’t corrected within a week. An agent wrote rows from my original request and the code as it stood before the task, and another agent turned them into check scripts. A separate Claude instance acted as a judge of whether each script tested what its row said. Then the runner ran the scripts on the “done” version and on the fix. A bug counts as caught only if a row fails on “done” and passes on the fix.

This tests agent-written rows alone. I set the bar before any row was written. Catching 3 or more would count as promising, as long as there were no more than 2 false alarms in the 10 controls and each case cost at most $3 and 15 agent-minutes. Catching 2 or fewer would mean rethinking the approach.

It caught 0 of the 6.

The controls went better. 5 of the 10 had a failing row. One was excluded under a rule set before the run, because the AI judge found its row stricter than anything I’d asked for. 2 were false alarms where the rows invented details, right at that limit. The other 2 were real gaps, shipped changes that hadn’t done what I asked and had gone unnoticed for weeks. The plumbing worked too. The runner got through all 96 row checks in about 6 minutes, with 7 marked live and not run automatically. Each case took 3 to 11 agent-minutes, and the harness didn’t record dollars.

So why did it miss all six? Every bug case had failing rows, but each failure was about something else, and it failed the same way on the fix. No check told the buggy version apart from the fixed one. Most bugs sat just outside what the rows covered, and in one case the row was right but its check tested the wrong layer.

That case stopped one hop short. A check meant to confirm that an update “reaches viewers” only confirmed that it reached the streamer side’s connection to the server. The real bug dropped the update one hop later, in the viewer’s buffer. The judge rated this check faithful. I overruled it.

Here’s where the six bugs were.

• Two were one step downstream: a debug view that drew a hitbox rotated twice, and the viewer’s buffer that dropped the update.
• One was on a side path: a saved setting that didn’t apply until the next update, in a task whose rows were about something else.
• One was where two layers meet: a revealed card in the pop-up window couldn’t be hovered where it overlapped a card on the board.
• One was a “must not happen” nobody wrote down: a plain click in my card-labeling tool started drawing boxes, though the old help text promised it wouldn’t.
• One predated the task entirely. Without it, the score is 0 of 5.

The two-layer bug is the one from the receipt figure earlier. Rows 2 to 4 checked board cards under the window, clear of it and half under its edge, and all three matched. Row 4 came closest. It said a revealed card should show if one sat over the covered half, but its example never put one there, so its check accepted nothing at all. That overlap was exactly where the bug was. Row 5’s mismatch was about something else, and it failed the same way on the fix.

In this pilot, rows that an agent wrote from my request, with the old code in hand, couldn’t reach the bugs.

Checking an agent’s work is only as good as the description of what “done” means. Running the checks is the easy part. Writing the right examples up front, including the boring “this must stay the same” ones, is the hard part. The next version needs a step that deliberately traces what the changed code feeds into and what it already promised to do, and writes rows for those.

I’m glad I wrote the bar down in advance. Otherwise I might have found a way to call 0 of 6 “promising”, the same way a pre-revenue and pre-product startup is “promising”.

Two smaller lessons from the trials

The checker got its own timesheet wrong

In the first trial, the agent doing the checking reported its own cost. It said about 120,000 tokens and 24 tool calls, and that one of its scripts took “~10 min”. Claude Code’s usage numbers, which actually count, recorded 132,000 tokens, 31 tool calls, and under five minutes for the whole run.

No agents were harmed in the process, but the episode gives a clean rule. Anything an agent says about its own work needs the same evidence as everything else it says. From the next version on, receipts will take cost and timing only from the usage numbers or the runner, never from the agent. Self-reports are a tricky thing to lean on at any scale.

CodeQL caught what planted bugs couldn’t

In the second trial, a test for a failure case passed. Great. Then a CodeQL code-scanning alert pointed out that a line in the test’s setup that was meant to change the input string actually left it unchanged. The input did fail, but for a different reason than the test meant to check, and the receipt had already marked the row as matched. Planting bugs in the code didn’t catch it either, because that changes the code under test, not the test’s own setup. The test was a smoke detector beeping because its battery is low.

There was an upside too. The second trial’s receipt found 3 places where the agent’s tests didn’t assert what the agreed rows said. The code was right but the evidence was weak, and the agent closed all three gaps in a test-only commit.

Why this matters beyond my side project

I’ve written software professionally for five and a half years, and I can read every line my agents write with a reasonable degree of confidence. I still ended up in “fixed, still broken” loops that took hours to resolve. Someone who can’t read code can get stuck in the same loops without even knowing it. We both need a way to break out of them.

There’s also evidence that this could get worse. In a preliminary study Anthropic published in January 2026 [4], 52 developers learned Trio, a Python library new to them, with or without AI assistance. They were quizzed right after, and the AI group averaged 50% while the other group averaged 67%. The biggest gap was on debugging questions, which asked them to find and diagnose errors in Trio code. The people who grow to rely on agents the most could become the ones least able to check their work.

Checking an agent’s work (verification), or even just making sure you’re both on the same page (alignment), is a hard problem. It breaks into smaller pain points, like checks that pass while the real feature fails, an agent’s claims about its own work, and writing down what “done” means before the work starts. Solving just one of them well could spawn a billion-dollar company. Not financial advice, but if you have any ideas, hit me up.

As agentic development continues to accelerate, we need a way to verify an agent’s work that doesn’t require reading the diff. The receipt is my first attempt at one, and it rests on three ideas.

• The human and the agent agree on examples of the task before any work starts.
• The checks run independently of the agent that did the work.
• A task counts as complete when there’s evidence for it, not when the agent says so. That includes the agent’s claims about its own performance.

If self-assessments were enough, I would never be single.

Limitations

• One person, one project: 98% of the moments come from RiftSight, which depends on live streams, so live-testing pain is likely inflated compared with backend or command-line work.
• Keyword search misses: The search missed about a third of the moments, mostly ops chores, live-testing bugs, and UI notes, so those categories are undercounted.
• One AI labeler, one pass: I spot-checked it, but there was no second labeling and no agreement measure, so category edges are soft. The labeler is also Claude, the same model family as the agent being audited.
• Counting messages: The audit counted messages instead of bugs or hours spent. Cost comes from 29 traceable defect episodes and measures elapsed time.
• A pilot-sized replay: 6 bugs and 10 controls. Agents wrote its rows and checks, the AI judge sometimes saw notes that predicted the outcome, and a single case moves the catch rate by about 17 points.
• No baseline: I don’t know how often I’d step in after a human colleague said “done”, so I can’t say how much of the 47% is specific to agents.
• Blind spots: The audit can’t see screenshots, pasted documents, subagent transcripts, or anything I did outside Claude Code.

What’s next

The next test is the one that matters. I’ll approve the rows myself before the work starts, on new tasks rather than these six, since I’ve now seen their answers. The open question is whether reviewed rows catch misunderstandings, which purely agent-written rows have failed to. A few smaller changes are planned too.

• Every row for a failure case asserts the reason it fails, not just that it fails.
• Static-analysis alerts, like the CodeQL one above, attach to the receipt as evidence.
• For behavior a row marks as unchanged, a reusable check runs the same command on the old and new code and compares the outputs, instead of a bespoke 100-line script each time.

I’ll write up the next result whichever way it goes, especially if it’s 0 out of X again. That will be part 2, about the approaches I try until one sticks.

References

[1] Sundar Pichai, “Cloud Next ’26: Momentum and innovation at Google scale”, Google, April 22, 2026.

[2] “When AI builds itself”, Anthropic, 2026.

[3] Johnny Rodgers, Charlie Hess, Raissa Largman, Jamie Scheinblum and Chris Sullivan, “Reducing Slack’s memory footprint”, Slack Engineering, March 2017.

[4] Judy Hanwen Shen and Alex Tamkin, “How AI Impacts Skill Formation”, arXiv:2601.20245, January 2026. Summary: How AI assistance impacts the formation of coding skills, Anthropic.