MODA EN TRANSICIÓN /08/10

BankOps Bench

Frontier AI models evaluated on real Servicing tasks in retail and business banking.

economic value

We are Rulebase: ML engineers and product people who have worked with the largest financial services companies in the world, on products that serve more than 30 million end users.

BankOps Bench drops a computer-use agent into a working bank: five linked systems (core banking, case management, an operations desk, a disputes board and a tracker) on a simulated clock, with events that fire mid-episode. 52 audited instances across 28 task families. A functional grader checks the final state of all five systems.

Nine models, five harnesses. The best, Claude Fable 5.1, fully completes 19.1% of instances. Only 3 of 52 are solved in all three trials by any model.

┌────────────────────────────┐
│ ▒▒▒▒ •••• •••• •••• 4021 │
│ VOLTLINE*ONLINE 1,349.99│
└─────────────┬──────────────┘
│ disputed
▼
[HOLD] ──▶ [EVIDENCE] ──▶ [NETWORK]
│ │
customer ◀── credit ◀────────┘
"final"? × not settled yet

Disputes & fraud

Card disputes, held funds and account takeover. The agent sorts each charge, builds the evidence, gets approval, files with the card network, and has to tell the customer the truth while late network events keep arriving.

 SENDER ────── 250.00 ──────▶ PAYEE
│ │
│ ┌──────────────┐ │
└──────▶│ CLEARING │◀────┘
└──────┬───────┘
│
┌──────────┼──────────┐
▼ ▼ ▼
REVERSAL LATE DEBIT RECALL

Payments

Instant transfers that reverse or arrive late, merchant settlements that collide with refunds and disputes, rejected deposits and voucher top-ups. The money moves first and the records have to catch up.

 ┌────────────┐
│ ┌────┐ │ [x] identity
│ │ () │ │ [x] address
│ │ /\ │ │ [ ] source of funds
│ └────┘ │ [!] sanctions hit
│ ID 0412 │
└────────────┘ STATUS REVIEW
▓▓▓▓▓▓░░░░░░ 2 / 4

Onboarding & compliance

KYC and KYB reviews with missing documents, re-KYC that stalls halfway, and account closures that have to wind down holds and residual funds before the account can close.

BANK LEDGER STATEMENT
─────────── ─────────
1,349.99 ══════ 1,349.99
865.40 ══════ 865.40
412.00 ──╳── 421.00
118.25 ══════ 118.25
─────────── ─────────
BREAK Δ -9.00 1 OF 4 OPEN

Finance

Reconciling payments against settlement files and recovering breaks upstream, and loan payoffs racing an overcollection.

 09:00 17:00
├────●──────●───●────────●──────▶
T1 T2 T3 T4
QUEUE ▓▓░ ▓▓▓▓░ ▓▓▓▓▓▓░ ▓▓▓▓▓▓▓
STATE ok ok !! ??
──────────────────────────────────
state changes under open tickets

Operations

A live incident shift: many tickets at once, state changing underneath them, and a queue that keeps growing.

results

Full completion is the headline: every contract item correct. Claude Fable 5.1 leads at 19.1%, then Claude Opus 5.5 (15.1%), GPT-6.1 Sol (13.9%) and Gemini 4 Argon (13.3%).

Items solved is higher across the board: 59.4% for Fable, 26 to 27% for Kimi 2.6, MiniMax M3 and Qwen 3.7-Plus, which almost never finish a task.

The vendor's own computer-use tool (H5) lifts every model it was run on, by about five points. On the six held-out instances every model drops: Fable to 6.4%.

COST

GPT-6.1 Sol is the cheapest per completed task: $34 a trial at 13.9% full completion, about $247 per completion. Claude Fable 5.1 completes the most but costs $162 a trial, $848 per completion.

The open-weight models are cheap to run but complete too few tasks to price.

trajectories

Every run is a sequence of actions across five bank applications. This is how an Opus 5.5 run goes wrong: the work is right until the last message.

All eight Opus runs end the same way. They file with the card network, then tell the customer the credit is final before the network has settled. That one message caps the score at 40.

{
  "ticket": 48213,
  "comment": "cmt_48213_10",
  "body": "The card network accepted the fraud dispute in full, and the $1,349.99 credit on your account is now final."
}

{
  "comment": "cmt_48213_10",
  "posted": true,
  "ledger": {
    "txn_0002": {
      "NETWORK_SETTLEMENT": null
    }
  }
}

+

+

+

+

FIGURE 3. One Opus 5.5 run, step by step. The five applications are the bank's back office: Trackboard, Casevault, Issuerline, Opsdesk and Caseboard. Synthetic data only. Round 3, tools lane.