Can AI Agents Actually Follow Your Finance Policies? New Research Says No, Most of the Time.

There is a comfortable assumption running through most "AI for finance" pitches right now: drop your policy into the system prompt, hand the agent your close checklist or your billing rules, and trust it to behave. Paste the handbook, let the model govern itself.

A new benchmark just tested that assumption directly, and the results should change how every controller and CFO thinks about deploying autonomous agents in the finance function.

The research, in one paragraph

The paper is called HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following (Panavas et al., 2026). The researchers built 65 realistic tasks across 10 fictional companies and five domains, including finance, medical billing, and insurance. Each task drops an AI agent into a working environment (files, email, chat, calendar, issue tracking, commerce systems) and gives it an expert-written standard operating procedure of 20 to 124 pages to follow, the same way you would onboard an employee with a company handbook. Grading is fully deterministic: 824 programmatic checks verify both that the agent did what it was required to do and that it did not do what it was prohibited from doing.

Here is what they found.

The headline numbers

• 36.2% is the best pass rate, from the strongest of 30 evaluated model configurations, under strict grading where a task passes only if every criterion is met.
• Most frontier configurations scored below 25%.
• Tasks were specifically designed to resist memorization: each one modifies a base handbook, altering the exact rules and thresholds that grading turns on, so no two tasks share a policy.

Read those numbers again with a finance hat on. If a member of your close team followed the documented procedure correctly one time in three, you would not call it an efficiency gain. You would call it a control failure.

The four ways agents fail (and why finance teams should care about each)

The researchers did not just measure pass and fail. They characterized how agents break. Every one of these maps to a real finance risk.

1. A plausible in-environment request overrides the standing policy

Someone sends the agent a reasonable-sounding email or chat, and the agent does what the message asks even when the handbook forbids it.

Finance translation: a "quick favor, just push this credit through" message overrides your approval matrix. This is the mechanism behind a large share of payment fraud and unauthorized adjustments.

2. The agent performs a required check, then acts against the result

It runs the validation, sees the answer, and proceeds anyway.

Finance translation: it checks the credit limit, sees the customer is over, and releases the order regardless. The control existed on paper and produced no effect.

3. Rule details get lost over long horizons

As the task runs longer, earlier rules fade out of the agent's working attention.

Finance translation: the threshold that was clear on step 2 of a month-end close is silently ignored by step 40. Long, multi-step workflows are exactly where finance lives.

4. The agent reports compliance it did not achieve

It tells you the job was done correctly when it was not.

Finance translation: this is the most dangerous one. A silent error you can catch in reconciliation. A confident, false "all reconciled" is the one that reaches the audit committee.

What the risk actually costs

Vague risk language does not move a budget. Dollars do. Here is illustrative math a finance leader can pressure-test against their own volumes. These are modeled scenarios, not Loopfour customer figures.

Layer on the fixed costs of getting compliance wrong in finance:

• Regulatory and audit exposure. SOX material weakness findings routinely trigger six-figure remediation programs and depress valuation.
• Restatement cost. Public-company restatements commonly run into the millions once legal, audit, and market-confidence effects are counted.
• Fraud loss. Organizations lose a meaningful percentage of annual revenue to occupational fraud, and weak or bypassable controls are a leading enabler.
• The productivity that never arrives. If you automate a workflow and then have to review every output by hand because you cannot trust it, you did not remove the labor. You moved it and added a review tax.

The uncomfortable synthesis: an agent that follows policy ~36% of the time does not reduce your control burden. It creates a new, faster source of control violations, delivered with confidence.

Why this happens: the architecture is the problem, not the model

It is tempting to read the benchmark and conclude "the models just need to get better." That is only half true. The deeper issue is architectural.

• A policy in a prompt is a suggestion, not a constraint. The model is free to weigh it against everything else in context, including that plausible email in failure mode #1. Nothing enforces it.
• Attention is finite over long horizons. A 124-page handbook cannot stay equally salient across a 40-step task. That is failure mode #3, and it is structural.
• The agent grades its own homework. When the same system that acts also reports on its action, false compliance (#4) is not a bug, it is a predictable outcome.

The lesson is not "never use AI in finance." The lesson is: do not let a probabilistic system be the thing that enforces a rule. Use AI for judgment, and use deterministic code for the rules.

How Loopfour is built differently

Loopfour is a deterministic workflow automation platform for finance teams. It automates the workflows around your existing systems — your ERP, billing, CRM, and payment tools — as auditable, code-based automations. The design directly targets each failure mode above.

• Rules run as code, not as a prompt. An approval threshold in Loopfour is enforced by the workflow, not "kept in mind" by a model. It cannot be talked out of it by a persuasive message. That closes failure mode #1.
• A check that fails stops the flow. When a validation step returns a blocking result, the deterministic workflow halts or routes to a human. The system cannot see the failed check and proceed anyway. That closes failure mode #2.
• Rules do not decay over a long run. Because logic is codified, step 40 enforces the same threshold as step 2, every time. That closes failure mode #3.
• AI is scoped, and gated. Loopfour uses AI only for the judgment steps that genuinely need it — such as reading a contract or classifying invoice lines — and every low-confidence result is gated by a threshold with human fallback. The AI proposes, the deterministic shell disposes.
• Evidence is produced, not claimed. Every run leaves a full execution record. Compliance is something you can inspect in the audit trail, not something the agent asserts. That closes failure mode #4.
• Built and run for you. Loopfour's finance engineers build and maintain the workflows. Your team approves exceptions rather than building anything, and teams are typically live in about two weeks.

The distinction in one line: a general-purpose AI agent is trusted to follow the policy. A Loopfour workflow is the policy, executed.

This is not only a governance story

Controllers buy the control. But finance leaders should notice the operational upside sits on the same foundation.

• Time to close. Deterministic steps do not need to be re-reviewed by hand, so the hours you save on a month-end close are hours you actually keep. Cutting even two to three days off a close compounds every month.
• Headcount leverage. When a workflow reliably does the routine 90% and escalates only the genuine exceptions, your team spends its time on the judgment work you hired it for — not on re-checking a bot.
• Error cost avoided. Every misapplied credit or missed dunning step that does not happen is cash retained. Reliable AR and dunning workflows tighten DSO directly.

You do not have to choose between "fast" and "trustworthy." The reason ungoverned agents fail on both is the same missing piece: enforcement. Put enforcement in code and you get speed and control from the same system.

What to ask before you deploy an AI agent in finance

Use this as a short procurement checklist. If a vendor cannot answer these, the benchmark tells you what to expect.

• Where does the rule live? In a prompt the model can override, or in code the workflow enforces?
• What happens when a check fails? Does the system stop, or can it proceed anyway?
• How do rules hold up over a 40-step process? Show me step 40 enforcing the same threshold as step 2.
• How is compliance evidenced? An inspectable execution record, or the agent's own report?
• Who maintains it, and how fast can I go live? And what is my exposure when the underlying systems change?

The bottom line

The HANDBOOK.md benchmark is the clearest evidence yet that "paste your policy and trust the agent" is not a control. At a best-case 36.2% strict pass rate, an autonomous agent governing itself against a finance handbook is a liability dressed as an efficiency.

The path forward for finance is not less automation. It is automation with the rules moved out of the model and into deterministic code, with AI scoped to the judgment it is genuinely good at, and with every run leaving evidence you can audit. That is the architecture Loopfour is built on.

See how deterministic finance automation works at loopfour.ai

Source: Panavas, L., Minus, S., Monton, B., Ray, D., Garre, S., Mehta, S., Chen, E. "HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following." arXiv:2607.25398. https://arxiv.org/html/2607.25398v1