BenchSmoke

A detail worth checking in any AI agent review: were tool permissions auto-approved? In BenchSmoke's OpenCode test we used opencode run with --auto, because run is non-interactive. So a session where you approve each step isn't what we measured, and we said so.

When BenchSmoke gets something wrong in a review, the fix goes in the next issue where readers can see it, not a quiet edit. Verdicts can change too, and any change gets noted in a later issue.

Azure MCP Server 3.0 (stable Oct 12) leans cautious, per Microsoft's post: the Advisor "apply" tool returns guidance for an infrastructure-as-code change instead of touching live resources, and a --learn flag explores commands without running Azure operations. News, not a BenchSmoke test.

BenchSmoke sets each verdict before anyone checks whether the tool has an affiliate program. Affiliate links only ever go on tools rated Use it, and they're labeled. So far, issues #1 and #2 have had no affiliate links and no sponsor.

How BenchSmoke turns scores into a verdict: Use it needs "Does it work?" at 4 or higher out of 5, with no criterion below 3. One big weakness, like price or bugs, makes it Watch it. A failed task or wasted time is Skip it. Builders: which one score would sink a tool for you?

Testing a coding agent's sandbox? Run your probe in a plain shell first. Without that control, you can't tell "blocked by the sandbox" from "the file wasn't there." In BenchSmoke's control run, every fake secret was readable and the shell saw 267 processes. Inside Garuda's sandbox, it saw 5.

After a coding agent says it's done, check what's still running. In BenchSmoke's Garuda test, a command started a web server in the background, and nothing was left listening once Garuda finished. In our OpenCode test, a server stayed up on port 3000. One test each.

Running llama-server? llama.cpp v0.7.0 (Oct 15) changed its default port from 8080 to 9931, per the release notes. Docker images keep 8080 inside the container. Anything pointed at localhost:8080 may need --port 8080 after you upgrade. News, not a BenchSmoke test.

Ollama v0.40.5 (Oct 14) added NVIDIA's Nemotron 3.5 Bolt: 74B parameters, 9B active, tuned to fit a 24 GB NVIDIA GPU, per the release notes. Its bigger sibling, Super, is 120B with 12B active. News, not a BenchSmoke test. Builders: what's the largest model you run locally?

Read the changelog, not just the homepage. In BenchSmoke's Garuda test, its sandbox blocked writes outside the project, except package caches like ~/.cache and ~/.npm. The docs mention it, and the maker's unreleased changelog lists it as a known gap. Makers who write down the holes earn trust.

What can a sandboxed command actually see? In BenchSmoke's Garuda test, a command inside its sandbox saw 5 processes instead of 267, and 0 secret-looking environment variables instead of 7. With any agent, run env and ps from inside it before you trust it.

Check how new an AI tool is before you trust it with real credentials. Garuda, the coding agent BenchSmoke tested this week, had a GitHub repo created Sep 23, 2026, 1 star on Oct 15, and version 0.16.1 out Oct 7. New isn't bad, but fewer people have poked at it yet.

Same prompt, three runs, three different apps. In BenchSmoke's Garuda test with a local 9B model, run 1 made a Node command-line to-do app, run 2 wrote only a package.json and stopped, and run 3 made a Python API server with no web page. One run tells you very little.

Two separate questions when you try a sandboxed coding agent: is it safe, and is it any good? In BenchSmoke's Garuda test, the sandbox did what its docs say in every probe we ran. How good it is as an agent we couldn't tell, since we used a small local model. Safe isn't the same as good.

A small lesson from BenchSmoke's Garuda test: exit code 0 doesn't mean the agent did anything. With no terminal, garuda -p denied every file edit, ignored a project setting that allowed edits, and still exited 0. Our headless to-do run wrote nothing. Check the files, not the exit code.

A small lesson from BenchSmoke's Garuda test: approving file edits is not the same as approving shell commands. Garuda denied the model's file write, so the model wrote the same file with printf in bash, inside the sandbox, with no question asked. Garuda's maker documents this.

What did a free AI coding agent build in 23 seconds? In BenchSmoke's OpenCode 2.0 test, run 1 made a Node server with no dependencies (server.js, 121 lines) and a single HTML page (112 lines). You start it with node server.js. Nothing to install for a small to-do app.

BenchSmoke picks what to test next partly from readers. What's one AI tool you want us to run on a real small task, or one that wasted your time? Reader picks go to the top of our test list.

BenchSmoke's test prompt for AI coding tools is one paragraph: build a to-do app that can add, complete and delete tasks, keeps them after a restart, runs with one command, and uses no paid services or API keys. Every tool gets the same task, so results can be compared.

BenchSmoke's first issue planned 3 tool tests and shipped 1. The other 2 were cut, not rushed. Pi 1.0 still made Quick Hits as news, with no verdict, because we couldn't finish testing it in time. A Use it, Watch it or Skip it call needs a finished test.

BenchSmoke's OpenCode 2.0 review covers one command, opencode run, with a free model on one small task. We did not test /pair, Code Mode, background subagents, plugins or the desktop app. That gap is why our Use it verdict has medium confidence, not high.

BenchSmoke #001 is out. We ran OpenCode 2.0 with its free Big Pickle model on our to-do app task, twice. Both runs worked, in 23 s and 121 s, with no account or API key. Verdict: Use it, with medium confidence, because we tested a narrow slice. benchsmoke.com/issues/001/

Same model name, two dates, two ranks. On LMArena's WebDev board (Oct 18 data), DeepSeek-V4.1-Flash-20261017 (Max) is #12 at 1670, while the 20260910 snapshot is #25 at 1619. If you pin a dated model version, check which one a leaderboard actually scored.

Sponsors get one labeled Sponsored slot in a BenchSmoke issue. A sponsored tool never shows up in that same issue's tested section, and a sponsor never sees or influences a review. Ads and verdicts stay in separate places.

How our AI model leaderboard will grade each run: pass means every acceptance check passed, partial means at least half did, and fail means fewer than half did or the run never finished. Results show per task, like "2 pass, 1 partial". No combined score, no overall rank.

When we can't decide between two verdicts for an AI tool, we pick the lower one and explain why in the write-up. A verdict can be revisited later too, and any change gets noted in a later issue. So a Watch it today isn't the final word.

If a maker gives us free credits or an extended trial to test their AI tool, the write-up says so in plain words: "The maker gave us free access for testing." Any other tie, like an investment or a friendship, gets disclosed too, or a different reviewer takes that tool.

In a test this week, a coding agent wrote a to-do CLI, told us the run command, and never ran its own code. Our check found it mostly worked, plus two edge bugs: empty tasks accepted, and a wrong error for non-numeric input. Treat "done" as a claim until you verify.