Between a diligence score and a usable deal file

In a sample Westworld Finance Diligence Bench redline, early notes ask for 24 users and on-premise setup; the rubric requires 24 plus 6 read-only users, free hosting until customer hardware is available, and preserved terms. That progression captures why live transaction work is harder than extracting a figure from a static corporate filing. During an acquisition or commercial negotiation, business terms move across email threads, meeting notes, and draft agreements while discussions are still underway. Before touching the document, an analyst—human or software—has to establish which communication carries authority to change the deal, translate the agreed compromise into precise contract language, and leave settled provisions intact.

Scoring a task for model development and clearing a file for release answer different questions. Consider an independent hypothetical rubric with ten equally weighted requirements. A run that satisfies eight requirements and fails two receives a 0.80 score, which registers substantial progress when comparing systems under development. If a team treats those two failed requirements as mandatory for sign-off, however, it will withhold acceptance until the defects are repaired, because credit on the other eight items does not offset an unfulfilled required condition. An additive rubric measures how much of a specification an agent completes, whereas a release gate asks whether the file is ready for use.

Partial credit and the acceptance gap

Halluminate’s August 11, 2026 report (not independently rerun here) grades nine models across two harnesses on 88 tasks (three attempts, 264 per configuration, no best-of-three) using mean grades and ≥0.50 attempt rates. When an evaluation averages fractional scores across repeated trials, it blends two distinct properties of an agent: how much of a prompt it typically satisfies, and how consistently it executes.

In an independent two-trial hypothetical, a system scoring 0.60 and 0.60 posts the exact same 0.60 mean as a system scoring 1.00 and 0.20, yet the first clears a 0.50 cutoff twice while the second clears it once. While an attempt clearing 0.50 can still miss half of a prompt's weighted criteria, partial-credit averages remain useful when engineering teams are iterating on scaffolding, because strict all-or-nothing grading would collapse near-misses to zero; operational sign-off, by contrast, depends on how often a run finishes with zero critical errors.

In the published chart data (checked October 20, 2026), native Gemini 4 Argon leads single configurations at 0.60747 (0.01373 ahead of native Opus 5.5), while Opus 5.5 posts nearly identical grades across both harnesses and GPT-6.1 Sol edges higher in CU + TU at a much steeper run cost. Because the release omits task-clustered confidence intervals across repeated prompts and potentially shared source deals, a 1.4 percentage-point gap cannot establish a statistically conclusive ranking on its own. More practically, which model looks strongest depends on the deployment constraint: selecting a single model-and-CLI pairing asks a different question from asking how a model behaves when security policies forbid local shell access.

Native CLI harnesses permit code execution whereas CU + TU blocks it, and reported costs measure observed spend or token estimates instead of assigned compute budgets. A team's deployment permissions therefore determine which harness rows are relevant for comparison.

Keeping those execution contracts explicit is essential when moving from open-book filing benchmarks such as FinanceBench, which grade answers retrieved and calculated from public corporate disclosures, to workplace tasks that require editing files in place. To evaluate file-producing agents for production, operators need to separate task-level repeatability—defined in tau-bench as pass^k, the probability of succeeding across every one of k trials rather than at least once (pass@k)—from the unit economics of review. All-k repeatability measures how consistently an agent succeeds across repeated trials under the evaluated conditions—not whether it can run without supervision—while mean partial credit cannot price accepted work.

The relevant financial metric is cost per accepted deliverable: total spend on candidate model runs, tool infrastructure, and human verification divided by the count of deliverables that actually pass sign-off.

What the benchmaxxing record shows

Whenever a benchmark in a high-value domain shows tight margins at the top or sharp swings across tool environments, practitioners reasonably ask whether the leading systems were benchmaxxed—deliberately optimized to score well on that specific evaluation. Answering that question requires separating four distinct mechanisms: leaking held-out test prompts or reference answers into training corpora; training in related simulated environments to build domain competence; tuning prompts, harnesses, or checkpoint choices against a validation split; and exploiting weak graders at runtime without solving the task.

Across the public record reviewed on October 20, 2026, no model-specific evidence was located showing that any evaluated finance model was trained on the benchmark's tasks, exposed to hidden reference answers, or tuned through repeated submissions against the final test set. Because the public materials omit dated pre-evaluation access logs, training-set intersection audits, and submission histories, that result remains an inconclusive public audit rather than a certificate of clean training.

On the earlier Westworld web simulator benchmark, Halluminate noted that Yutori trained directly in Westworld environments. Yutori’s Navigator disclosure describes reinforcement learning in simulated sites including Noodle Flights, reporting 86.0% on the Westworld web benchmark and transfer to live Google Flights without real Google Flights RL data. In a separate shopping sim-to-real study, Halluminate trained small vision-language adapters on 257 tasks per site, selected checkpoints using 63 held-out tasks, noted prototype partial-credit reward hacking and limited cross-domain transfer, and reported headline rates across all 320 tasks.

Neither web-agent study implicates the frontier models on the finance leaderboard, and shared Westworld branding does not establish shared tasks or transferable competence. What the shopping study illustrates is a general rule of evaluation hygiene: once a held-out split is consulted to select checkpoints or prompts, it functions as a validation set, and pooling those tasks with training data cannot measure untouched generalization.

The report’s QA probes, illustrative keyword and one-cell exploits, and disagreeing post-run trajectory judges are diagnostics, not training or observed entrant exploits. Similarly, the page’s interactive trajectory explorer displays an earlier run set alongside golden trajectories built after inspecting run traces and verified with executor agents.

A trace-informed golden trajectory demonstrates that the workspace tools can execute a valid solution, whereas measuring autonomous discovery requires blind runs on held-out tasks. Moving from an inconclusive public audit to a specific finding requires matching each claim to relevant evidence: a reachable answer file inside an evaluation container or a reproduced grader exploit would show a specific environment or verifier flaw, whereas a weight-contamination claim would need evidence connecting test tasks to the actual training of the evaluated model.

What has to happen to the file

Its trajectory annotations (normalized within tagged errors rather than task failure rates) highlight stale sources and delivery mistakes, echoing the sample redline’s shifting terms. Addressing those failure modes in practice calls for architectural controls—offered here as untested engineering proposals—that follow what must happen to a deal file from initial discovery to downstream handoff. When a data room holds dozens of folders and message threads, reading every document up front can exhaust context and budget before editing begins, yet stopping at the first plausible file can miss later renegotiations.

Resolving conflicts by picking the newest timestamp also fails in transaction work: an informal afternoon coworker chat does not supersede an earlier signed term sheet or an explicit commercial approval. Bounded, task-directed retrieval needs to populate an authority-aware requirements ledger that records each active constraint alongside its source document, effective date, relevance to the deliverable, and governing authorization, escalating unresolved conflicts between authorized sources to a human reviewer instead of guessing.

Once the governing terms are identified, applying them to a workbook or agreement requires separating input selection from execution. When an agent copies a summary figure directly into a memo or rewrites a contract section from memory, any unit mismatch, rounding slip, or dropped condition becomes opaque to reviewers. Storing each chosen input with its source locator, reporting period, unit, and accounting definition allows scripts or spreadsheet formulas to compute derived schedules, while automated tie-out checks confirm that prose tables match workbook cells. The same separation applies when editing legal text: modifying a pricing tier or service transition has to target the affected clause while leaving settled notice periods, liability remedies, and baseline schedules untouched.

Even a sound edit fails if the artifact saved on disk does not match what the agent reports in its final completion message. A write call can return a success code after saving to the wrong directory, leaving #REF! errors in a spreadsheet, or placing required contract language inside a deleted markup span. Before an agent marks a deliverable ready, an independent verification pass should reopen the target file from disk, recalculate workbook formulas, and confirm that required contract edits survive in the clean readout alongside preserved clauses.

When runtime friction blocks that step—such as restricted local execution permissions or a locked office file—a transient lock may clear, but unbounded retries of the identical command without checking whether the environment changed can drain budget that unaffected requirements need. Checking available tool capabilities at startup, capping retries on non-idempotent file writes, reserving budget to save completed work, and emitting a structured handoff note that lists the verified files on disk, the blocked requirement, and the decision needed to resume keeps partial progress usable.

Testing whether files survive handoffs

Published task aggregates do not establish whether prior-stage actual outputs propagate across a deal. When transaction work spans connected stages, deliverables form a chain: extracted records feed financial schedules, analytical adjustments reshape underwriting assumptions, and approved commercial terms govern draft agreements. Summary averages alone cannot show how early errors travel through that sequence, because dependent stage handoffs allow an unflagged mistake in an upstream file to pass directly into the inputs that downstream work inherits.

Measuring that handoff gap requires a discriminating experiment on fresh holdout deals partitioned at the source-transaction level, with the model checkpoint, harness permissions, and retry policy frozen before runs start. For each transaction, run the sequence of dependent deliverables across two parallel tracks: a chained track where each step receives only the files the agent actually saved in the preceding step, and a reference track seeded with verified ground-truth inputs at every boundary.

Because altered intermediate files can change which documents an agent inspects next, comparing the two tracks serves as a practical diagnostic rather than a pure causal isolation. What that comparison reveals is the operational consequence that single-task averages hide: whether an early source-authority mistake or broken formula stays contained within a local draft, or quietly compounds across handoffs until a downstream valuation workbook or closing agreement fails its acceptance gate.