Skip to content
PROBLEM BRIEF / v12026-07-26

The AI says it is “done.” How do I know it is actually deliverable?

Do not treat an agent stopping as a delivery state. Define completion first, supervise the execution boundary, then connect code, runtime results, the target environment, and the exact version with layered evidence.

For: Independent developers and small teams using AI to write code, build products, or complete technical research

This page takes you through

  1. 1Understand the problem
  2. 2Compare approaches
  3. 3Inspect evidence
  4. 4Complete the check
  5. 5Review BitShovel practice
Approach paths
3
Product mappings
8
Developer signals
4
Evidence sources
11
Verification card
5 min

01 / UNDERSTAND

“Done” is an agent conversation state, not your acceptance decision

Delivery requires four things at once: a bounded artifact, observable acceptance conditions, executed changes, and evidence tied to the correct environment and version. If any layer is missing, the result is only a candidate—not a completed delivery.

01Editorial judgment

AI-generated work qualifies for a delivery decision only after execution and observable checks; before that it is a candidate output.

02Checked fact

Within the same compute and execution environment, a fixed scope, budget, and evaluation metric make consecutive agent experiments comparable and support keep-or-revert decisions.

03Editorial judgment

Critical tool calls cannot rely solely on the model voluntarily following a format; validation, strict constraints, and repeatable feedback belong in the execution environment.

04Source claim

specgit's official site connects visual document editing, diffs, comments, and approvals with GitHub commits and pull-request workflows.

02 / COMPARE

Three paths, not one universal tool

Compare the method first, then see which part each product supports. All six profiles are partially checked; none should be read as a BitShovel hands-on validation of the core flow.

PATH 1

Define what “done” means first

Turn the request into a reviewable spec, a bounded change set, and repeatable acceptance conditions. Give the agent a finish line before execution instead of explaining the outcome afterward.

Use when
When the request is still ambiguous, several people must agree on scope, or a false “done” would create rework or release risk
Trade-off
Up-front definition costs a few minutes. If acceptance conditions remain slogans, it only creates more documentation overhead.

specgit

Partially checked

Specification and review record

It places specs, diffs, comments, and approvals in commit and PR context, helping teams freeze scope before execution and compare changes afterward. BitShovel has not independently tested its repository permissions or merge behavior.

Still unverified

'Docs stay in GitHub' does not mean zero processing. The GitHub App needs read/write contents and pull-request permissions on granted repositories, and realtime collaboration state may persist temporarily for up to seven days. When a user invokes AI, document text and relevant comments go to xAI, whose API data may be retained for abuse monitoring for up to 30 days; explicitly shared AI conversations may be kept for up to 90 days. The current privacy page also discloses allowlisted, content-free analytics through PostHog Cloud EU: region- and consent-gated, with raw events retained up to 12 months and operational logs up to 30 days. These boundaries are more specific than older 'no analytics' summaries and deserve review before adoption.

Inspect sources, interface, and limits

JetBrains Air

Partially checked

Planning, parallel execution, and human review

Its official material puts planning, parallel agent tasks, and diff review in one workspace, making it a useful reference for “inspect the plan, execute, then review.” This remains a maker claim in this record.

Still unverified

Air is still a public preview. Available models, permissions, execution location, and cost depend on JetBrains AI, BYOK, or an external agent provider; an isolated worktree does not mean code or prompts never leave the device. Agents may edit files, run commands, or consume quota, so each diff, permission, and network path needs review. The live download page should take precedence over early platform plans; this review did not independently verify stability, privacy, speed, or agent compatibility.

Inspect sources, interface, and limits

PATH 2

Make execution visible, stoppable, and reversible

You need to see what the agent is doing, which files changed, and which permission it is waiting for—and stop it before it crosses a boundary. Parallelism matters less than clear task ownership, state, diffs, and recovery points.

Use when
When work changes a repository, calls external systems, operates browser accounts, or runs several agents
Trade-off
A control plane adds operational overhead. Auto-approval is faster, but visibility must not be mistaken for security isolation.

JetBrains Air

Partially checked

Central view of plans, task state, and code diffs

It is useful as a reference for combining multi-agent execution and final review in one interface. It cannot replace a project's own permissions, tests, or release gates.

Still unverified

Air is still a public preview. Available models, permissions, execution location, and cost depend on JetBrains AI, BYOK, or an external agent provider; an isolated worktree does not mean code or prompts never leave the device. Agents may edit files, run commands, or consume quota, so each diff, permission, and network path needs review. The live download page should take precedence over early platform plans; this review did not independently verify stability, privacy, speed, or agent compatibility.

Inspect sources, interface, and limits

CodeAgentSwarm

Partially checked

Control plane for multi-agent tasks, terminal state, and diffs

Its official page presents a task board, terminal state, file diffs, history, and permission controls, making it useful for studying supervised parallel agents. Turbo Mode should still not be treated as a security sandbox.

Still unverified

'Code and prompts do not pass through the CodeAgentSwarm backend' does not mean the product has no data flows. The current policy says desktop usage events go to its backend; crash reports are forwarded to Sentry and stack traces may contain local file paths; optional sign-in stores OAuth profile data, tokens, IP address, and device information; and the Swarmi help assistant sends questions plus recent history to the backend and stores the exchange. The policy gives no fixed deletion period for analytics, errors, or support messages, and does not explain storage location, encryption, portability, or deletion for the complete local conversation history. The Turbo Mode guide also says Ask effectively behaves like Allow while confirmation skipping is on, so actions that must never auto-run need Deny. Conflicts from several agents editing one repository, guardrail coverage, and bypass paths were not independently tested.

Inspect sources, interface, and limits

AgentLimb

Partially checked

Observe actions and permission boundaries in a real browser

The official repository demonstrates an agent acting on real pages through an extension and local bridge, useful for reasoning about step confirmation, account side effects, and browser permissions. BitShovel has not installed or tested it.

Still unverified

The official pages use “90% fewer tokens,” “85% fewer tokens,” and an 85.7% estimate from one Reddit posting task; none should be treated as a general benchmark. The Chrome listing has no rating and the product is still early. Its privacy page discloses activeTab, scripting, debugger, nativeMessaging, clipboardWrite, and <all_urls> permissions, and lets a local agent evaluate short JavaScript expressions in the page; local execution is not the same as low risk. The repository uses Business Source License 1.1: free for personal use, with commercial use subject to the license.

Inspect sources, interface, and limits

PATH 3

Confirm the result with layered evidence

Record “code changed,” “tests passed,” “the target environment works,” “risk is controlled,” and “users benefited” as separate layers. Evidence for one layer does not automatically prove the next.

Use when
Before release, when real data is involved, when comparing candidate solutions, or when you must explain why a change is safe to approve
Trade-off
Evidence becomes more costly and risky as it approaches the real environment. Start with low-risk checks and advance layer by layer instead of immediately operating production accounts.

traceAI

Partially checked

Trace, evaluate, and compare AI-system runs

Its official open-source repository combines tracing and evaluation, addressing the evidence layer of what the agent actually ran and how outcomes compare. Performance and coverage still need reproduction in your own system.

Still unverified

The docs explicitly say full prompts, completions, tool arguments and results, and error context are captured by default; every hide flag for inputs, outputs, messages, images, embeddings, and model parameters defaults to false. Even maximum redaction keeps model names, token counts, timing, span types, and error messages, which can themselves contain sensitive material. The default environment guide requires Future AGI API and secret keys and points HTTP and gRPC collectors to its cloud. Custom collectors and bring-your-own TracerProvider are documented but were not exercised here. Uneed copy and the first launch image say MIT, while the current repository LICENSE, README badge, and NOTICE all say Apache-2.0. Uneed claims four-language full parity and 35+ frameworks, the images and README say 50+, the repository's own matrix contains many blanks, and cloud pricing says only 30+ libraries. The cloud free tier currently advertises 50 GB of tracing per month with 30-day retention and overage from USD 2 per GB. The data-region page says US is available to all plans and EU Frankfurt only to Enterprise, while the pricing card presents US and EU side by side. Security pages self-report SOC 2 Type II and ISO 27001 certification and HIPAA support through a BAA on Scale or Enterprise; they list AWS, ClickHouse Cloud, Cloudflare, Stripe, plus platform-wide HubSpot, PostHog, Sentry, and Intercom. No downloadable certificate, audit report, or penetration-test summary was public, so procurement must request them. The open-source instrumentation, Future AGI hosted platform, and its 'Self-host it' claim are three separate boundaries and should not be treated as interchangeable.

Inspect sources, interface, and limits

HeimWall

Partially checked

Review AI-generated code risk before approval

Its official page positions security checks before AI-generated code enters delivery, adding a risk-evidence layer beyond functional tests. BitShovel has not independently audited its coverage or false-positive rate.

Still unverified

This is an early version, not a finished enterprise control plane. The team product is labeled early access. The Individual page still says v0.0.5 and 3.9 MB, while the download API pointed to v0.0.6 and the publisher screenshot also shows v0.0.6. No public changelog, installer SHA-256, reproducible build, or source code was found to map the binary. Feature definitions drift as well: the maker's Uneed copy says 47 hand-written rules while the current site says 25+; Individual says a pretrained classifier runs locally while the architecture labels it 'architecture shipped · model pending'; FAQ says 'no browser extension, no proxy' while Home and How it works include a local TLS proxy; and pages plus screenshots differ on coverage for Cursor, Claude Code, Copilot, Windsurf, and ChatGPT. The privacy boundary needs precision. Individual claims no account, backend, or network. Team mode may upload tool, category, severity, rule, timestamp, a redacted snippet up to 500 characters, and a correlatable hash. Privacy names Investigation Mode as the sole exception to no raw-prompt upload and retains investigation records for a contractual legal-hold period, while the architecture says the server stores an unreadable sealed payload. Public material does not explain whether that payload contains raw prompt text, when it is produced on the endpoint, who holds decryption keys, whether an employee can refuse or revoke it, what happens if notice fails, or the guessing risk of deterministic hashes over low-entropy secrets. Team events are queryable for 90 days, cold-stored for another 180, then retained as org aggregates. A public subprocessor page is still in preparation; the draft Privacy page names Supabase, ClickHouse Cloud, Clerk, Vercel, Fly.io, Resend, and Anthropic. The public site bundle also includes a PostHog page-view provider configured with memory persistence and session recording/autocapture disabled, while the current Privacy draft does not separately explain website analytics. Compliance and contracts are incomplete: Privacy and Terms are both April 22, 2026 drafts, and the binding terms and MSA are still being prepared with counsel. FAQ says SOC 2 Type II is in progress and HIPAA eligibility arrives with a future v1.5, while Pricing directly lists an Enterprise SOC 2 report, HIPAA eligibility, BYOK, and on-prem and also marks some items 'soon' in its comparison table. GDPR, signing/notarization, <50ms p95, 95% soft notifications, 99.9% SLA, the blog's three teams / 312 engineers, and enforcement of the no-performance-review covenant are team statements. No audit report, corpus, false-positive/negative results, notarization ticket, customer reference, or final contract was independently checked. Team and Business are annual per-seat subscriptions, and FAQ describes a $15K deposit for a 30-day paid POC. Before production procurement, confirm the available version, seat minimums, refunds, legal entity and address, lawful basis for employee monitoring, DPA, regions, subprocessors, keys, and deletion flow.

Inspect sources, interface, and limits

AgentLimb

Partially checked

Observe real action outcomes on target web pages

A browser task cannot be accepted from the agent's narration alone. Page state, permission prompts, and account side effects are target-environment evidence. AgentLimb shows this surface, but BitShovel has no hands-on conclusion yet.

Still unverified

The official pages use “90% fewer tokens,” “85% fewer tokens,” and an 85.7% estimate from one Reddit posting task; none should be treated as a general benchmark. The Chrome listing has no rating and the product is still early. Its privacy page discloses activeTab, scripting, debugger, nativeMessaging, clipboardWrite, and <all_urls> permissions, and lets a local agent evaluate short JavaScript expressions in the page; local execution is not the same as low risk. The repository uses Business Source License 1.1: free for personal use, with commercial use subject to the license.

Inspect sources, interface, and limits

03 / DEVELOPER PUBLIC PRACTICE

What leading developers actually do

These are concrete public actions with first-party sources. BitShovel's reading is labeled separately; individual practice is not presented as universal proof.

Developer public practice2026-07-26

Simon Willison

Start each agent session on an existing project by having it run the tests.

BitShovel reading

Tests do more than approve the final result: they teach the agent how the project is verified and make a post-change rerun more likely.

First run the tests
Developer public practice2026-07-26

Mitchell Hashimoto

Turn recurring agent mistakes into fast, automatic, repeatable checks and project instructions.

BitShovel reading

A manual correction fixes one result. Encoding the feedback in the harness changes the default quality of the next delivery.

My AI Adoption Journey
Developer public practice2026-07-26

Armin Ronacher

Reproduce tool-call failures and compare strict constraints with permissive repair behavior.

BitShovel reading

A stronger model does not make every tool call more reliable. Critical parameters and irreversible actions need harness-level enforcement.

Better Models: Worse Tools
Developer public practice2026-07-26

Andrej Karpathy

Fix the editable file, time budget, and evaluation metric; log every experiment and keep only improvements.

BitShovel reading

An agent can iterate autonomously, but what it may change, how success is measured, and when to revert must be executable rules first.

autoresearch agent operating rules

04 / CLAIMS & EVIDENCE

See the ground beneath every conclusion

Source claims, checked facts, editorial judgment, and unknowns stay separate; every linked item returns to a primary source.

01Editorial judgment
High confidence

AI-generated work qualifies for a delivery decision only after execution and observable checks; before that it is a candidate output.

  • First run the tests
    Source:
    Simon Willison's personal weblog
    Type:
    Developer primary source
    Checked:

    First-party practice note on using automated tests to verify AI-generated code and orient an agent in an existing project.

  • My AI Adoption Journey
    Source:
    Mitchell Hashimoto's personal site
    Type:
    Developer primary source
    Checked:

    First-party account of clear tasks, verifiable outcomes, and harness engineering as ways to improve agent results.

02Checked fact
High confidence

Within the same compute and execution environment, a fixed scope, budget, and evaluation metric make consecutive agent experiments comparable and support keep-or-revert decisions.

  • autoresearch project README
    Source:
    Andrej Karpathy's official GitHub repository
    Type:
    Developer primary source
    Checked:

    First-party project description of a bounded edit surface, fixed five-minute experiment budget, and val_bpb comparison across runs.

  • autoresearch agent operating rules
    Source:
    Andrej Karpathy's official GitHub repository
    Type:
    Developer primary source
    Checked:

    First-party operating rules: establish a baseline, log every metric, keep improvements, and discard or revert equal, worse, or crashing runs.

03Editorial judgment
High confidence

Critical tool calls cannot rely solely on the model voluntarily following a format; validation, strict constraints, and repeatable feedback belong in the execution environment.

  • Better Models: Worse Tools
    Source:
    Armin Ronacher's personal weblog
    Type:
    Developer primary source
    Checked:

    First-party reproduction of newer-model failures on a tool schema and the effect of strict invocation constraints in his tests.

  • My AI Adoption Journey
    Source:
    Mitchell Hashimoto's personal site
    Type:
    Developer primary source
    Checked:

    First-party account of clear tasks, verifiable outcomes, and harness engineering as ways to improve agent results.

04Source claim
Medium confidence

specgit's official site connects visual document editing, diffs, comments, and approvals with GitHub commits and pull-request workflows.

  • specgit official product page
    Source:
    specgit
    Type:
    Maker primary source
    Checked:

    Maker material used to confirm the publicly presented workflow, not independent BitShovel testing of permissions, saves, approvals, or merges.

05Source claim
Medium confidence

JetBrains Air's official material presents planning, parallel agent tasks, and code-review workflows.

  • JetBrains Air official product page
    Source:
    JetBrains Air
    Type:
    Maker primary source
    Checked:

    Maker material confirming the publicly presented planning, parallel-task, and review interface, not an independent product test.

06Source claim
Medium confidence

CodeAgentSwarm's official page presents multi-terminal state, a task board, file diffs, history, and permission features.

  • CodeAgentSwarm official product page
    Source:
    CodeAgentSwarm
    Type:
    Maker primary source
    Checked:

    Maker material confirming public features and UI, without treating Turbo Mode or permission features as independently verified security boundaries.

07Source claim
Medium confidence

AgentLimb's official repository shows a Chrome extension and local bridge working together so an agent can act on real browser pages.

  • AgentLimb official open-source repository
    Source:
    AgentLimb
    Type:
    Maker primary source
    Checked:

    Maker README and demo proving the published architecture and demo flow, not that BitShovel installed the extension, ran the bridge, or reproduced outcomes.

08Source claim
Medium confidence

traceAI's official open-source repository combines tracing, evaluation, and monitoring for AI applications in one toolchain.

  • traceAI official open-source repository
    Source:
    Future AGI / traceAI
    Type:
    Maker primary source
    Checked:

    Maker material confirming published tracing and evaluation capabilities, not BitShovel reproduction of performance or coverage.

09Source claim
Medium confidence

HeimWall's official site positions the product as a security-checking layer before AI-generated code is delivered.

  • HeimWall official product page
    Source:
    HeimWall
    Type:
    Maker primary source
    Checked:

    Maker material confirming its public positioning, not a BitShovel audit of detection coverage, false positives, or security conclusions.

05 / CHANGE HISTORY

How this problem brief evolves with evidence

Public updates record new sources, boundary revisions, and accepted patches; chronology is not a quality ranking.

Browse all verified updates

06 / YOUR CHECK

Turn reading into a reusable delivery decision

Your answers stay in this browser. You will receive one of three verdicts: ready, more verification needed, or stop.

Complete on this device

5-minute delivery verification card

0/9 · 0%

Answer each item and retain reviewable evidence. If any critical item is unclear, do not promote “agent finished” to “ready to deliver.”

01Deliverable

Name the exact files, pages, APIs, documents, or external states included in this delivery.

Stop condition: There is only a “done” message and no identifiable artifact.

02Observable acceptance conditions

Describe what success and failure look like in a way another person can repeat.

Stop condition: The standard is still “looks good,” “should work,” or the agent's own summary.

03Change diff

Confirm the diff stays within scope and identify new dependencies, configuration, migrations, and deletions.

Stop condition: The change contains unexplained edits, mixed-in generated output, or an unbounded rewrite.

04Actual execution

Record the checks, tests, or tasks that actually ran, including raw outcomes and failures.

Stop condition: The code never ran, or success is inferred only from static inspection.

05Target environment

State the device, browser, operating system, account tier, or deployment environment where the result was verified.

Stop condition: Only local or simulated evidence exists, but the result is described as production- or user-verified.

06Version evidence

Tie test results, build artifacts, and deployment state to the same commit, version, or content record.

Stop condition: You cannot show that the tested content is the content being delivered or deployed.

07Permissions and side effects

List account permissions, external writes, messages, data handling, costs, and irreversible actions.

Stop condition: The agent can use high-risk permissions without step confirmation, least privilege, or outcome auditing.

08Rollback point

Identify where to restore, who performs the rollback, and how long it should take.

Stop condition: The change is irreversible, or the recovery path has never been checked in a comparable environment.

09Unknowns

List what remains unverified, comes only from a maker statement, or requires real-user data.

Stop condition: Unknowns are omitted or hidden behind the word “done.”

Evidence text stays in this browser for 7 days. Do not enter secrets or personal data; reset after use on a shared device. Anonymous metrics include the verdict plus page, session, and attribution context—not your text.

The verdict uses only the status and evidence you record. It is a local self-check, not a BitShovel audit or delivery guarantee.

07 / BITSHOVEL PRACTICE

See how we separate finished, live, and impactful

The same work can be complete at the publication layer, incomplete as hands-on product testing, and still unsupported by user-impact evidence.

BitShovel Practice2026-07-26

How BitShovel verified and published the AgentLimb profile

This practice completed bilingual source review and public content publication, but did not install AgentLimb, complete a real product task, or demonstrate new user impact. The three conclusions are recorded separately so “article published” never becomes “product validated.”

Content publication: completed
The bilingual AgentLimb profile is published on a public BitShovel page and appears in the public content feed.
Hands-on product validation: not completed
This was a read-only review of public material. We did not install the Chrome extension, start the local bridge, grant browser permissions, or run a task in a real account.
User impact: not completed
There is currently no evidence attributing new visits, outbound clicks, feedback, or retention to the AgentLimb profile.
Inspect the full practice
Browse all BitShovel practices