AI-generated work qualifies for a delivery decision only after execution and observable checks; before that it is a candidate output.
The AI says it is “done.” How do I know it is actually deliverable?
Do not treat an agent stopping as a delivery state. Define completion first, supervise the execution boundary, then connect code, runtime results, the target environment, and the exact version with layered evidence.
For: Independent developers and small teams using AI to write code, build products, or complete technical research
This page takes you through
- 1Understand the problem
- 2Compare approaches
- 3Inspect evidence
- 4Complete the check
- 5Review BitShovel practice
- Approach paths
- 3
- Product mappings
- 8
- Developer signals
- 4
- Evidence sources
- 11
- Verification card
- 5 min
01 / UNDERSTAND
“Done” is an agent conversation state, not your acceptance decision
Delivery requires four things at once: a bounded artifact, observable acceptance conditions, executed changes, and evidence tied to the correct environment and version. If any layer is missing, the result is only a candidate—not a completed delivery.
Within the same compute and execution environment, a fixed scope, budget, and evaluation metric make consecutive agent experiments comparable and support keep-or-revert decisions.
Critical tool calls cannot rely solely on the model voluntarily following a format; validation, strict constraints, and repeatable feedback belong in the execution environment.
specgit's official site connects visual document editing, diffs, comments, and approvals with GitHub commits and pull-request workflows.
02 / COMPARE
Three paths, not one universal tool
Compare the method first, then see which part each product supports. All six profiles are partially checked; none should be read as a BitShovel hands-on validation of the core flow.
PATH 1
Define what “done” means first
Turn the request into a reviewable spec, a bounded change set, and repeatable acceptance conditions. Give the agent a finish line before execution instead of explaining the outcome afterward.
- Use when
- When the request is still ambiguous, several people must agree on scope, or a false “done” would create rework or release risk
- Trade-off
- Up-front definition costs a few minutes. If acceptance conditions remain slogans, it only creates more documentation overhead.

specgit
Partially checkedSpecification and review record
It places specs, diffs, comments, and approvals in commit and PR context, helping teams freeze scope before execution and compare changes afterward. BitShovel has not independently tested its repository permissions or merge behavior.
Still unverified
'Docs stay in GitHub' does not mean zero processing. The GitHub App needs read/write contents and pull-request permissions on granted repositories, and realtime collaboration state may persist temporarily for up to seven days. When a user invokes AI, document text and relevant comments go to xAI, whose API data may be retained for abuse monitoring for up to 30 days; explicitly shared AI conversations may be kept for up to 90 days. The current privacy page also discloses allowlisted, content-free analytics through PostHog Cloud EU: region- and consent-gated, with raw events retained up to 12 months and operational logs up to 30 days. These boundaries are more specific than older 'no analytics' summaries and deserve review before adoption.

JetBrains Air
Partially checkedPlanning, parallel execution, and human review
Its official material puts planning, parallel agent tasks, and diff review in one workspace, making it a useful reference for “inspect the plan, execute, then review.” This remains a maker claim in this record.
Still unverified
Air is still a public preview. Available models, permissions, execution location, and cost depend on JetBrains AI, BYOK, or an external agent provider; an isolated worktree does not mean code or prompts never leave the device. Agents may edit files, run commands, or consume quota, so each diff, permission, and network path needs review. The live download page should take precedence over early platform plans; this review did not independently verify stability, privacy, speed, or agent compatibility.
PATH 2
Make execution visible, stoppable, and reversible
You need to see what the agent is doing, which files changed, and which permission it is waiting for—and stop it before it crosses a boundary. Parallelism matters less than clear task ownership, state, diffs, and recovery points.
- Use when
- When work changes a repository, calls external systems, operates browser accounts, or runs several agents
- Trade-off
- A control plane adds operational overhead. Auto-approval is faster, but visibility must not be mistaken for security isolation.

JetBrains Air
Partially checkedCentral view of plans, task state, and code diffs
It is useful as a reference for combining multi-agent execution and final review in one interface. It cannot replace a project's own permissions, tests, or release gates.
Still unverified
Air is still a public preview. Available models, permissions, execution location, and cost depend on JetBrains AI, BYOK, or an external agent provider; an isolated worktree does not mean code or prompts never leave the device. Agents may edit files, run commands, or consume quota, so each diff, permission, and network path needs review. The live download page should take precedence over early platform plans; this review did not independently verify stability, privacy, speed, or agent compatibility.

CodeAgentSwarm
Partially checkedControl plane for multi-agent tasks, terminal state, and diffs
Its official page presents a task board, terminal state, file diffs, history, and permission controls, making it useful for studying supervised parallel agents. Turbo Mode should still not be treated as a security sandbox.
Still unverified
'Code and prompts do not pass through the CodeAgentSwarm backend' does not mean the product has no data flows. The current policy says desktop usage events go to its backend; crash reports are forwarded to Sentry and stack traces may contain local file paths; optional sign-in stores OAuth profile data, tokens, IP address, and device information; and the Swarmi help assistant sends questions plus recent history to the backend and stores the exchange. The policy gives no fixed deletion period for analytics, errors, or support messages, and does not explain storage location, encryption, portability, or deletion for the complete local conversation history. The Turbo Mode guide also says Ask effectively behaves like Allow while confirmation skipping is on, so actions that must never auto-run need Deny. Conflicts from several agents editing one repository, guardrail coverage, and bypass paths were not independently tested.

AgentLimb
Partially checkedObserve actions and permission boundaries in a real browser
The official repository demonstrates an agent acting on real pages through an extension and local bridge, useful for reasoning about step confirmation, account side effects, and browser permissions. BitShovel has not installed or tested it.
Still unverified
The official pages use “90% fewer tokens,” “85% fewer tokens,” and an 85.7% estimate from one Reddit posting task; none should be treated as a general benchmark. The Chrome listing has no rating and the product is still early. Its privacy page discloses activeTab, scripting, debugger, nativeMessaging, clipboardWrite, and <all_urls> permissions, and lets a local agent evaluate short JavaScript expressions in the page; local execution is not the same as low risk. The repository uses Business Source License 1.1: free for personal use, with commercial use subject to the license.
PATH 3
Confirm the result with layered evidence
Record “code changed,” “tests passed,” “the target environment works,” “risk is controlled,” and “users benefited” as separate layers. Evidence for one layer does not automatically prove the next.
- Use when
- Before release, when real data is involved, when comparing candidate solutions, or when you must explain why a change is safe to approve
- Trade-off
- Evidence becomes more costly and risky as it approaches the real environment. Start with low-risk checks and advance layer by layer instead of immediately operating production accounts.

traceAI
Partially checkedTrace, evaluate, and compare AI-system runs
Its official open-source repository combines tracing and evaluation, addressing the evidence layer of what the agent actually ran and how outcomes compare. Performance and coverage still need reproduction in your own system.
Still unverified
The docs explicitly say full prompts, completions, tool arguments and results, and error context are captured by default; every hide flag for inputs, outputs, messages, images, embeddings, and model parameters defaults to false. Even maximum redaction keeps model names, token counts, timing, span types, and error messages, which can themselves contain sensitive material. The default environment guide requires Future AGI API and secret keys and points HTTP and gRPC collectors to its cloud. Custom collectors and bring-your-own TracerProvider are documented but were not exercised here. Uneed copy and the first launch image say MIT, while the current repository LICENSE, README badge, and NOTICE all say Apache-2.0. Uneed claims four-language full parity and 35+ frameworks, the images and README say 50+, the repository's own matrix contains many blanks, and cloud pricing says only 30+ libraries. The cloud free tier currently advertises 50 GB of tracing per month with 30-day retention and overage from USD 2 per GB. The data-region page says US is available to all plans and EU Frankfurt only to Enterprise, while the pricing card presents US and EU side by side. Security pages self-report SOC 2 Type II and ISO 27001 certification and HIPAA support through a BAA on Scale or Enterprise; they list AWS, ClickHouse Cloud, Cloudflare, Stripe, plus platform-wide HubSpot, PostHog, Sentry, and Intercom. No downloadable certificate, audit report, or penetration-test summary was public, so procurement must request them. The open-source instrumentation, Future AGI hosted platform, and its 'Self-host it' claim are three separate boundaries and should not be treated as interchangeable.

HeimWall
Partially checkedReview AI-generated code risk before approval
Its official page positions security checks before AI-generated code enters delivery, adding a risk-evidence layer beyond functional tests. BitShovel has not independently audited its coverage or false-positive rate.
Still unverified
This is an early version, not a finished enterprise control plane. The team product is labeled early access. The Individual page still says v0.0.5 and 3.9 MB, while the download API pointed to v0.0.6 and the publisher screenshot also shows v0.0.6. No public changelog, installer SHA-256, reproducible build, or source code was found to map the binary. Feature definitions drift as well: the maker's Uneed copy says 47 hand-written rules while the current site says 25+; Individual says a pretrained classifier runs locally while the architecture labels it 'architecture shipped · model pending'; FAQ says 'no browser extension, no proxy' while Home and How it works include a local TLS proxy; and pages plus screenshots differ on coverage for Cursor, Claude Code, Copilot, Windsurf, and ChatGPT. The privacy boundary needs precision. Individual claims no account, backend, or network. Team mode may upload tool, category, severity, rule, timestamp, a redacted snippet up to 500 characters, and a correlatable hash. Privacy names Investigation Mode as the sole exception to no raw-prompt upload and retains investigation records for a contractual legal-hold period, while the architecture says the server stores an unreadable sealed payload. Public material does not explain whether that payload contains raw prompt text, when it is produced on the endpoint, who holds decryption keys, whether an employee can refuse or revoke it, what happens if notice fails, or the guessing risk of deterministic hashes over low-entropy secrets. Team events are queryable for 90 days, cold-stored for another 180, then retained as org aggregates. A public subprocessor page is still in preparation; the draft Privacy page names Supabase, ClickHouse Cloud, Clerk, Vercel, Fly.io, Resend, and Anthropic. The public site bundle also includes a PostHog page-view provider configured with memory persistence and session recording/autocapture disabled, while the current Privacy draft does not separately explain website analytics. Compliance and contracts are incomplete: Privacy and Terms are both April 22, 2026 drafts, and the binding terms and MSA are still being prepared with counsel. FAQ says SOC 2 Type II is in progress and HIPAA eligibility arrives with a future v1.5, while Pricing directly lists an Enterprise SOC 2 report, HIPAA eligibility, BYOK, and on-prem and also marks some items 'soon' in its comparison table. GDPR, signing/notarization, <50ms p95, 95% soft notifications, 99.9% SLA, the blog's three teams / 312 engineers, and enforcement of the no-performance-review covenant are team statements. No audit report, corpus, false-positive/negative results, notarization ticket, customer reference, or final contract was independently checked. Team and Business are annual per-seat subscriptions, and FAQ describes a $15K deposit for a 30-day paid POC. Before production procurement, confirm the available version, seat minimums, refunds, legal entity and address, lawful basis for employee monitoring, DPA, regions, subprocessors, keys, and deletion flow.

AgentLimb
Partially checkedObserve real action outcomes on target web pages
A browser task cannot be accepted from the agent's narration alone. Page state, permission prompts, and account side effects are target-environment evidence. AgentLimb shows this surface, but BitShovel has no hands-on conclusion yet.
Still unverified
The official pages use “90% fewer tokens,” “85% fewer tokens,” and an 85.7% estimate from one Reddit posting task; none should be treated as a general benchmark. The Chrome listing has no rating and the product is still early. Its privacy page discloses activeTab, scripting, debugger, nativeMessaging, clipboardWrite, and <all_urls> permissions, and lets a local agent evaluate short JavaScript expressions in the page; local execution is not the same as low risk. The repository uses Business Source License 1.1: free for personal use, with commercial use subject to the license.
03 / DEVELOPER PUBLIC PRACTICE
What leading developers actually do
These are concrete public actions with first-party sources. BitShovel's reading is labeled separately; individual practice is not presented as universal proof.
Simon Willison
Start each agent session on an existing project by having it run the tests.
BitShovel reading
Tests do more than approve the final result: they teach the agent how the project is verified and make a post-change rerun more likely.
Mitchell Hashimoto
Turn recurring agent mistakes into fast, automatic, repeatable checks and project instructions.
BitShovel reading
A manual correction fixes one result. Encoding the feedback in the harness changes the default quality of the next delivery.
Armin Ronacher
Reproduce tool-call failures and compare strict constraints with permissive repair behavior.
BitShovel reading
A stronger model does not make every tool call more reliable. Critical parameters and irreversible actions need harness-level enforcement.
Andrej Karpathy
Fix the editable file, time budget, and evaluation metric; log every experiment and keep only improvements.
BitShovel reading
An agent can iterate autonomously, but what it may change, how success is measured, and when to revert must be executable rules first.
04 / CLAIMS & EVIDENCE
See the ground beneath every conclusion
Source claims, checked facts, editorial judgment, and unknowns stay separate; every linked item returns to a primary source.
AI-generated work qualifies for a delivery decision only after execution and observable checks; before that it is a candidate output.
- First run the tests
- Source:
- Simon Willison's personal weblog
- Type:
- Developer primary source
- Checked:
First-party practice note on using automated tests to verify AI-generated code and orient an agent in an existing project.
- My AI Adoption Journey
- Source:
- Mitchell Hashimoto's personal site
- Type:
- Developer primary source
- Checked:
First-party account of clear tasks, verifiable outcomes, and harness engineering as ways to improve agent results.
Within the same compute and execution environment, a fixed scope, budget, and evaluation metric make consecutive agent experiments comparable and support keep-or-revert decisions.
- autoresearch project README
- Source:
- Andrej Karpathy's official GitHub repository
- Type:
- Developer primary source
- Checked:
First-party project description of a bounded edit surface, fixed five-minute experiment budget, and val_bpb comparison across runs.
- autoresearch agent operating rules
- Source:
- Andrej Karpathy's official GitHub repository
- Type:
- Developer primary source
- Checked:
First-party operating rules: establish a baseline, log every metric, keep improvements, and discard or revert equal, worse, or crashing runs.
Critical tool calls cannot rely solely on the model voluntarily following a format; validation, strict constraints, and repeatable feedback belong in the execution environment.
- Better Models: Worse Tools
- Source:
- Armin Ronacher's personal weblog
- Type:
- Developer primary source
- Checked:
First-party reproduction of newer-model failures on a tool schema and the effect of strict invocation constraints in his tests.
- My AI Adoption Journey
- Source:
- Mitchell Hashimoto's personal site
- Type:
- Developer primary source
- Checked:
First-party account of clear tasks, verifiable outcomes, and harness engineering as ways to improve agent results.
specgit's official site connects visual document editing, diffs, comments, and approvals with GitHub commits and pull-request workflows.
- specgit official product page
- Source:
- specgit
- Type:
- Maker primary source
- Checked:
Maker material used to confirm the publicly presented workflow, not independent BitShovel testing of permissions, saves, approvals, or merges.
JetBrains Air's official material presents planning, parallel agent tasks, and code-review workflows.
- JetBrains Air official product page
- Source:
- JetBrains Air
- Type:
- Maker primary source
- Checked:
Maker material confirming the publicly presented planning, parallel-task, and review interface, not an independent product test.
CodeAgentSwarm's official page presents multi-terminal state, a task board, file diffs, history, and permission features.
- CodeAgentSwarm official product page
- Source:
- CodeAgentSwarm
- Type:
- Maker primary source
- Checked:
Maker material confirming public features and UI, without treating Turbo Mode or permission features as independently verified security boundaries.
AgentLimb's official repository shows a Chrome extension and local bridge working together so an agent can act on real browser pages.
- AgentLimb official open-source repository
- Source:
- AgentLimb
- Type:
- Maker primary source
- Checked:
Maker README and demo proving the published architecture and demo flow, not that BitShovel installed the extension, ran the bridge, or reproduced outcomes.
traceAI's official open-source repository combines tracing, evaluation, and monitoring for AI applications in one toolchain.
- traceAI official open-source repository
- Source:
- Future AGI / traceAI
- Type:
- Maker primary source
- Checked:
Maker material confirming published tracing and evaluation capabilities, not BitShovel reproduction of performance or coverage.
HeimWall's official site positions the product as a security-checking layer before AI-generated code is delivered.
- HeimWall official product page
- Source:
- HeimWall
- Type:
- Maker primary source
- Checked:
Maker material confirming its public positioning, not a BitShovel audit of detection coverage, false positives, or security conclusions.
05 / CHANGE HISTORY
How this problem brief evolves with evidence
Public updates record new sources, boundary revisions, and accepted patches; chronology is not a quality ranking.
First problem brief and five-minute verification path published
BitShovel organized “the AI says it is done—how do I know it is deliverable?” into understanding, comparison, evidence, a verification card, and practice.
06 / YOUR CHECK
Turn reading into a reusable delivery decision
Your answers stay in this browser. You will receive one of three verdicts: ready, more verification needed, or stop.
Complete on this device
5-minute delivery verification card
Answer each item and retain reviewable evidence. If any critical item is unclear, do not promote “agent finished” to “ready to deliver.”
07 / BITSHOVEL PRACTICE
See how we separate finished, live, and impactful
The same work can be complete at the publication layer, incomplete as hands-on product testing, and still unsupported by user-impact evidence.
How BitShovel verified and published the AgentLimb profile
This practice completed bilingual source review and public content publication, but did not install AgentLimb, complete a real product task, or demonstrate new user impact. The three conclusions are recorded separately so “article published” never becomes “product validated.”
- Content publication: completed
- The bilingual AgentLimb profile is published on a public BitShovel page and appears in the public content feed.
- Hands-on product validation: not completed
- This was a read-only review of public material. We did not install the Chrome extension, start the local bridge, grant browser permissions, or run a task in a real account.
- User impact: not completed
- There is currently no evidence attributing new visits, outbound clicks, feedback, or retention to the AgentLimb profile.