- I.Four years of writing agents that closed the books§1
- II.The four-step ladder§2
- III.What stuck§3
- IV.What didn't§4
Four years of writing agents that actually closed the books.
When I started PC Financials in 2022, “AI accounting” was a sales-deck phrase and a list of integrations nobody had wired together. The pitch was always some version of “let the model read your bank feed and do your books.” The reality was that the model would read the bank feed, hallucinate a vendor that didn’t exist, classify a Stripe payout as a refund, and the bookkeeper would spend longer cleaning up the output than the original work would have taken. The first generation of these tools shipped to early adopters, ran for six weeks, and got quietly turned off.
Four years later, the slice that survives daily contact with a real general ledger is much narrower than the marketing suggested — and the patterns that do work have very little to do with the models themselves. The model is the easy part. The hard part is the scaffolding around it: the deterministic checks that catch the failures, the review queues that surface them, the rollback paths when something goes sideways at 3am. None of that scaffolding is glamorous, and none of it shows up in a demo.
This essay is a write-up of what I’ve actually shipped across client engagements, what I’ve torn out and rewritten, and the four-step ladder I now use to decide whether a piece of close-time work belongs in an agent or stays human. The ladder is the thing I would have wanted handed to me in 2022. It’s boring. It works.
Working rule“If the only person who can verify the output is the person who wrote the prompt, it’s a demo. If a CPA can spot-check the output in under two minutes per client per day, it’s an agent.
”
The rule above is the constraint that does most of the real work. Once you accept that the CPA’s two-minute review budget is the bottleneck — not the model’s accuracy, not the input volume, not the engineering cost — the design space narrows fast. Most of the things you might want to automate become indefensible at two minutes per day. The ones that survive are the ones the ladder picks out.
The four-step ladder.
Each rung is a question. If the answer is “yes,” the work is allowed to climb to the next rung. If “no,” it stays human until something material changes — either the work itself, the tooling, or the client’s data shape. I run every candidate piece of automation through these four questions before I write a line of code.
By “shape” I mean: does the transaction look roughly the same in January as it does in November? Bank-feed lines are stable; the descriptor format from a given processor doesn’t change week to week. Payroll journal entries are stable; the same accounts get hit in the same direction every cycle. Vendor invoices in the AP queue are mostly stable, with a long tail of one-offs. Anything where the input shape drifts — a CRM export whose columns get renamed, a Stripe API response that adds a field, an internal CSV someone hand-edits before sending — fails this rung. Drift kills classifiers.
This is the most-skipped step in the industry, and the one that separates real automations from theatre. By a “deterministic check” I mean: there is a programmatic way to confirm the model’s output is correct, without invoking another model. For transaction classification, the check is whether the proposed entry balances and whether the account exists in the chart. For three-way reconciliation, the check is whether the bank, processor, and GL all agree to the cent. If the output is free-text — a draft email, a narrative explanation, an executive summary — there is no deterministic check, and the agent is not allowed to climb.
This is the operational constraint. If the agent ships a thousand classifications a day and the CPA can review them at sixty per minute, the engagement absorbs fifteen minutes per client per day on review alone. Across a full roster that compounds fast — over a month it’s the entire payback of running the agent in the first place. The threshold I’ve settled on is a hundred per minute, which in practice means the output has to be glanceable — a row, a flag, a color, a single number out of place. Anything that requires reading a paragraph fails this rung even if it passes the first two.
The failure mode I lose sleep over is the silent one — the agent that ran fine for eleven months and started quietly mis-classifying for two weeks before anyone noticed, because the chart of accounts got a new line and the classifier hadn’t seen it. The fix is to instrument every agent with an out-of-distribution detector: a confidence threshold, a per-account volume diff against the trailing 30-day baseline, a “novel vendor” flag that fires on first sight. The agent’s job isn’t only to do the work; it’s to know when it shouldn’t.
The four rungs are not independent. An agent that passes rungs 1 and 2 but fails rung 3 is almost always salvageable by changing the review format — collapse the output into a queue, sort by exception, present the CPA with the eight items that matter instead of the thousand that don’t. An agent that fails rung 1 is almost never salvageable; if the input shape is unstable, no amount of model-tuning catches up.
What stuck.
Five agents are running across the roster today, all of them named, scoped, and on a schedule. I named them so I can talk about them on Slack without ambiguity, and because un-named software is harder to retire when it stops earning its place.
pcf-categorize handles transaction classification against the client’s chart of accounts. On a chart it’s seen for at least ninety days, it’s right on the large majority of transactions, and — more important — it’s reliably uncertain about the rest. The disagreements cluster in three places: vendors with ambiguous memos that could plausibly be one of two accounts, transactions whose right answer requires context only the operator has, and new vendors with no prior history. The first two go to the CPA; the third resolves itself within a billing cycle. We log every disagreement between the agent and the CPA and track accuracy as a rolling 30-day average rather than a single snapshot.
pcf-reconcile does three-way matching across the bank statement, the processor settlement file, and the GL. It runs nightly and surfaces mismatches as a queue the controller works through in the morning. The thing that makes this work is not the model — it’s the deterministic check. Every proposed match has to tie to the cent, in the same period, with consistent memos. The model’s job is to propose; the check decides.
pcf-anomaly compares the current period against a rolling 90-day baseline and flags account-level movements more than two standard deviations off trend. The suspected driver is attached to each flag. About 70% of flags are real signal (a new vendor, a missed accrual, a duplicate invoice); the rest are seasonality or one-offs the operator already knows about. Either way, the conversation on the variance-review call is meaningfully better when the anomalies are pre-surfaced.
pcf-sales-tax normalizes taxable-transaction data, routes by jurisdiction, and prepares filings + remittance. It’s the most rules-heavy of the five — model usage is minimal, deterministic logic is most of the work — but it goes here because it ships under the same engagement and on the same cadence as the others.
pcf-cashflow assembles the 13-week rolling forecast from AR, AP, payroll, recurring revenue, and seasonal patterns. It refreshes weekly. Most operator-readable, least mechanically impressive: it’s mostly bottom-up modeling with a small model in the loop for cohort assignment.
The pattern across all five is that the output is structured, the validation is mechanical, the review is glanceable, and the failure mode is loud. The ladder built itself out of the survivors.
What didn't.
An incomplete list of things I shipped, tried for a quarter, and either tore out or quietly demoted from “agent” to “internal tool with a human always in the loop.” Each one failed at least one rung of the ladder.
Free-text journal-entry drafting. The pitch is irresistible — describe the transaction in plain English, get a draft JE you can post. The reality is that the model is right about 80% of the time and wrong in ways that take longer to catch than writing the JE yourself. Failed rung 2 (no deterministic check) and rung 3 (review-per-output too slow because each draft has to be read in full). Not coming back until either the model gets a lot more conservative about uncertainty or someone figures out how to make a JE auto-verify against the underlying source documents.
Anything that requires reading a contract. Revenue recognition, lease classification, capex versus opex calls, deferred revenue schedules — all of these are downstream of a document that someone has to actually read. Models are increasingly good at extracting structured data from contracts, but the structured data isn’t the bottleneck. The bottleneck is the judgement of which clause matters, which is exactly the work a CPA gets paid for. Failed rung 4 (no clear out-of-distribution signal — the agent’s wrong answers look identical to its right answers).
Revenue recognition for SaaS with usage-based pricing. Standard subscription rev rec is mostly mechanical and the existing accounting platforms handle it well enough. Usage-based rev rec, with consumption that varies by month and discount tiers that kick in at thresholds, is a different shape of problem — and it’s exactly the kind of work where the input shape isn’t stable enough for a classifier and the output isn’t deterministic enough for a checker. Failed rungs 1 and 2 simultaneously. Still hoping someone solves this; the answer is going to be specific tooling, not a general agent.
Auto-drafted board-packet commentary. Tempting because the inputs (the close packet) and the outputs (a one-page operator summary) are right there. The problem is that the commentary needs to know what changed in the operator’s life, not just in the GL. We tried this for two months across three clients and the drafts were uniformly worse than what the CPA wrote in fifteen minutes. The CPA writes the commentary; the agents prepare the inputs.
The pattern across the failures“The pattern in what didn’t stick is the same pattern as in what did, just inverted. The work that stuck has structured outputs and glanceable review. The work that didn’t has free-text outputs, requires reading something that isn’t in the GL, or both.
”
The interesting failure mode I expected and have not seen across four years is the CPA losing intuition for the books because the agents handle so much of the day-to-day. The opposite has happened: reviewing exception queues turns out to be a much higher-leverage way to stay close to the GL than re-keying transactions. The CPA spends less time on rote work and more time on the edge cases, which is exactly the time the engagement is being paid for.
- The four-year window starts in early 2022 with the first GPT-3.5-based classifier; the current generation of pcf-categorize has been running on Claude with a deterministic checker since mid-2024.
- ”Two-minute review budget per client per day” is observed, not designed. It’s roughly what the engagement can absorb before the unit economics break against the §1 retainer fee.
- The rolling 30-day accuracy measurement is the only one I trust. Single-snapshot accuracy numbers in this space are nearly meaningless; the question is what the agent does in the third month after a chart change.
