Dwayne Forde

research.withmantle.comAWE Benchmark | Assets & Wealth Document Extraction | Mantle ResearchCompare AI models on financial document extraction with the AWE benchmark. Explore accuracy, cost, and performance across fields and document layouts.

Replying to @dwayne.io

We also want this to be genuinely open. The dataset and benchmark tooling will be released soon so anyone can run it, challenge it, and build on it. If you're working with AI in the Wealth tech space, I'd love your feedback 👇 research.withmantle.com/awe

I added Cloudflare’s Claric to our AWE benchmark, and it finished near the bottom, which is what I expected and not really the point.

Falls short: - four sections (called by year, rows, highlights and sourcing channels) came in at 0% - table-heavy financial statements sat at 69%

The spread is the interesting part. A fast single-pass model that nails some sections and misses others could work as a first pass, or as a gate that routes the hard sections to a reasoning model. See the full results at research.withmantle.com/awe

I added Cloudflare’s Claric to our AWE benchmark, and it finished near the bottom, which is what I expected and not really the point.

Clairic scored 54.0% overall, 21st of the 23 models on the chart, well behind the leaders at about 95%. This isn’t a fair comparison though. Clairic is a decision-style model with no reasoning step, and it was given text rather than the PDFs most frontier models saw.

Surprisingly good: - Changes in capital (100%): Every other model scored 67% or lower, including Claude - Opus 5.5 and Claude Fable 5.1 at 67% and the GPT-6 models at 33% - Letter and bank sections (99% and 100%): within a point of the frontier models

docs.cloud.google.comTroubleshoot Cloud Run with SSH | Google Cloud DocumentationPreview — SSH support for Cloud Run

22h

Looks like @GoogleCloud has quietly beta released SSH access to Cloud Run containers. Much appreciated for those quick debugging sessions. 👏 docs.cloud.google.com/run/docs/tro...

Exciting to see @OpenCode 2.0 ship! is a rewrite, not a point release. Here are the highlights.

Plugins: 80+ internal plugins implement the 2.0 features, TUI plugins are now stable, and plugins hot-reload mid-session.

The catch is that it's a breaking release: 1.0 plugins won't work, the server API is different, and the SDK is still changing. Full notes: anoma.ly/notes/openco...

Exciting to see @OpenCode 2.0 ship! is a rewrite, not a point release. Here are the highlights.

Background tasks: subagents and shell commands can run in the background, and ctrl+b moves running foreground work there. Useful for long test runs while you keep working.

The API was rebuilt from scratch, and the agent can call its own API, for example to create a git worktree and move a session into it. Agents setting up their own workspace is a nice direction.

Exciting to see @OpenCode 2.0 ship! is a rewrite, not a point release. Here are the highlights.

Always-on local server: one background process that the TUI, web interface and desktop app all connect to. Scripts can discover it through the client package, and /pair exposes it to other hosts.

Always-on local server: one background process that the TUI, web interface and desktop app all connect to. Scripts can discover it through the client package, and /pair exposes it to other hosts.

Exciting to see @OpenCode 2.0 ship! is a rewrite, not a point release. Here are the highlights.

Code Mode: the agent writes one short JavaScript script that chains tool calls, instead of separate round trips. MCP servers and @OpenAPISpec endpoints become callable functions. I want to try it on workflows that hit several MCP servers in a row.

The part I liked most in the claude.ai speedup writeup isn't the 3x, it's the loop: deterministic benchmarks, ship behind a flag, ratchet the benchmark down, repeat. claude.dev/blog/how-we-...

github.comGitHub - Niko1221/Strata: Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input. - Niko1221/Strata

Really curious to try this out on some local hardware. github.com/Niko1221/Str...

Gemini 4 Argon is new on the board and comes in sixth, 1.2 points behind the leader Claude Opus 5.5. It is a clear step up from Gemini 3.1 Pro Preview, which scored 92.2% but not enough for a podium finish.

On AWE-bench, @withmantle.com's research benchmark for extracting names, dates and numbers from wealth and equity documents. At the top of the board, the frontier models are bunched so tightly that the gap between them matters less than what they cost and how fast they run.

If you've compared frontier models on financial reasoning, what made you choose one over another? Would love to hear thoughts. You can see the full results on the AWE-bench website. research.withmantle.com/awe

businessinsider.comAmazon exec says finding the right FDE talent is challenging: 'These are unicorns'An Amazon Web Services exec said that she's being selective in building out the firm's bet on forward-deployed engineers.

Every wave of new tech is driven by a tech consulting layer. Curious how many FDEs it takes before that starts to look like full blown consulting business inside these orgs. www.businessinsider.com/amazon-aws-f...

blog.withmantle.comThe Wealth Operating System | Mantle PortalDiscover how collection, extraction, personalization, and connected workflows form the foundation for the next evolution of wealth and equity technology.

The wealth vertical is highly dynamic in terms of inputs and outputs. Especially when it comes to the private markets, the freedom to do business in different ways powers its $18 trillion AUM by 2027.

Our Wealth OS handles it in four steps: - Collection: Documents and data from every source, flag what's missing - Extraction: Structured data, validated records - Personalization: Apply your preferences and remember them - Workflows: Put that data to work in reporting, analysis, and next actions

The result is a foundation that lets the whole vertical process more funds, more complexity, and more clients. That's what it takes for the private market to keep growing: infrastructure that scales with the flexibility makes markets valuable. Full breakdown - blog.withmantle.com/the-wealth-o... 📖

The Wealth Operating System has been made to help bridge the gap of flexibility & throughput, and customization. Every fund reports differently. Every document lands somewhere new. Today, that variety is absorbed by people manually turning documents into records, and that doesn't scale.

Claude Code's advisor tool lets a cheaper main model consult a stronger one at key decision points. I'm sure there is a use case to empower some business/operational workflows with this. When coding though, I'll choose to use the best model throughout. code.claude.com/docs/en/advi...

Latest entrant to the agent execution layer is OpenAI's Dots. The amount of polish and capability from these UXs is getting close a having a fully featured & managed VM. I wonder what will stand out as a key differentiator at this layer.

Claude Opus 5.5 dropped so we ran it through AWE-bench, and it doesn't disappoint. research.withmantle.com/awe

Opus takes the lead from Fable (at much less cost) mostly due to its improved understanding on financial statements. The other document types (ex. capital statements) seem to more or less perform the same.

We're going to be introducing some higher reasoning multi-document use cases to put some more complex financial reasoning to the test. See the results at research.withmantle.com/awe

delta.devWelcome to Delta - Delta DocsLearn your way around Delta by making and reviewing a change in your own project.

Delta from @Zed.dev: describe the change, an agent works in an isolated checkout, you review the diff. Great that you can BYOM (Bring your own model). Pricing for this layer in the industry is pretty vague/volatile, keeping this execution layer closer to home. delta.dev/docs/getting...

All of my most powerful and efficient agentic workflows involved a bunch of agents using an army of CLIs. Really happy @cloudflare.social is leaning into this pattern. 👏 blog.cloudflare.com/cloudflare-c...

Parenting the age of AI

New infra layer forming that might upset the IaaS space. Multiple folks vying to own the encrypted, persistent AI memory narrative. deepmind.google/blog/advanci...

The live interaction experiences with models are advancing so quickly. Some of micro behaviours are a bit unsettling, but compared to where we were last year it's 🤯. Given the option to wait 1hr or use this to check into my hotel, I would probably choose this.

blog.google/innovation-a...

blog.googleIntroducing Gemini 3.8 Live with Live AvatarIntroducing Gemini 3.8 Live with Live Avatar, which brings near real-time visual presence to Gemini’s conversational AI.

blog.withmantle.comIntroducing AWE-Bench: A Benchmark for AI Extraction in Wealth and Equity | MantleIntroducing AWE-Bench, an open benchmark for AI document extraction in wealth and equity. Compare model accuracy, cost, and speed field by field.

Wealth tech has grown large enough that "trust us, the AI is accurate" shouldn't be the bar anymore. The space deserves proper, public research on raw model capabilities.

Mantle@withmantle.com

For LPs, fund managers, and the teams building wealth tech: AI is doing more of the document work in private markets. But how do you know it's getting the numbers right? Today, we're launching AWE-Bench (Assets & Wealth Extraction Benchmark): blog.withmantle.com/introducing-...

Until now there's been no shared way to measure how well models handle the documents this industry actually runs on: capital calls, distribution notices, capital account statements. AWE-Bench is our attempt to fill that gap.

This is only the start. Extraction is the foundation, and next we want to add much larger reasoning tasks, where models have to connect information across documents rather than just pull fields from one.

"Continuous improvement is better than delayed perfection" - Mark Twain

All the frontier AI models got a bump this month. Seems to be a pattern forming which means the next major bump will be in Dec/Jan. I can't think of another time where the tech stack evolved this quickly. research.withmantle.com/llm-costs/

openai.comHow V7 gives AI agents institutional memoryUsing GPT-5.6, V7 turns scattered company files into context agents can use to complete complex, source-linked work.

Love how the AI wealth tech space is heating up. The @V7labs folks have done a great job digging in. There is still lots to do and @withmantle.com will play a big part. openai.com/index/v7/

claude.comProjects redesigned: from folder to conversation | Claude by AnthropicA new experience for Claude projects, now available in beta in Claude Code

19 Sept

Looks like @anthropic.com wants to own the whole SDLC 🍿 claude.com/blog/project...

blog.cloudflare.comYour site, your rules: new AI traffic options for all customersFor our second Content Independence Day, we’re giving website owners finer options to manage AI traffic. Instead of a one-size-fits-all block, all customers can now easily distinguish and manage…

17 Sept

Cloudflare's new AI crawler classifier just rolled out which is a one block-everything switch is now separate dials for Search, Agent, and Training bots. Eager to see the before/after analytics 🍿 blog.cloudflare.com/content-inde...

arxiv.orgAutomating Low-Risk Code Review at Meta: RADAR, Risk Calibration, and Review EfficiencyAI-assisted coding tools have altered software production. At Meta, significant lines of code per human-landed diff grew by 105.9% year over year and per-developer diff volume rose 51%, with agentic…

17 Sept

Experimenting with solutions for code review bottlenecks. Meta's approach is interesting. AI-driven diffs grew 106% year over year, so they built a risk-based auto-reviewer that cut revert rates to a third and review time by 35%. arxiv.org/abs/2605.30208

openai.comRapidly scaling online storage to serve over 1 billion ChatGPT usersLearn how OpenAI evolved Habitat from a Python library into a globally distributed storage platform serving 1 billion ChatGPT users and 22M requests per second.

What stands out is the team size relative to the scale. Two people rewriting & migrating a critical piece of infrastructure handling 70M+ requests a second is a genuinely different staffing model than this kind of migration used to require. openai.com/index/scalin...

openai.comIntroducing ChatGPT for Financial ServicesIntroducing ChatGPT for Financial Services, combining built-in financial data and GPT-6 Astra for research, modeling, and client-ready materials.

Both Anthropic and OpenAI shipped financial services offerings within a day of each other. It's the battle of traditional software being enabled AI vs AI being enhanced with financial logic from practitioners. openai.com/index/introd...

blakecrosley.comFoundation Models from Python: the fm CLImacOS 27 ships the fm command line tool plus a Foundation Models SDK for Python, so you can script Apple's on-device model and build evaluation pipelines.

Apple's on-device foundation models accept an object schema for structured output, which makes them far more useful than a chat endpoint for anything you want to actually build on. blakecrosley.com/blog/foundat...

withmantle.comMantle Equity | Cap Table Management for StartupsManage your cap table, model rounds, issue equity, and stay fundraise-ready with Mantle's cap table management platform. Built for founders, CFOs, and law firms.

If you're one of the companies now needing to pick a new home for your cap table, we built @withmantle.com Mantle Equity because we believed the same thing Pulley's founders clearly did. That founders deserve better than the status quo. Happy to talk anytime 👋 withmantle.com/equity