patchpilot fonts motion repair core

Bug report in. Tested patch out. Brakes built in.

PatchPilot AI is an autonomous repair agent for approved Python repositories. The model only reads and proposes. Allowlists, patch rules, budgets, a locked-down sandbox, hidden-test grading and a human approval gate are enforced in code outside it.

View on GitHub Explore the architecture

Built and verified offline one real-model repair, local qwen3:8b

inj-001-cachekit · repair mode scripted
  1. proposal 1edit src/cachekit/lru.py +1 · new test
  2. sandboxpatched code: test fails ✗ rejected
  3. feedbackown checks only · 2 proposals left
  4. proposal 2move_to_end(key) +1
  5. sandboxfails before ✓ passes after ✓ existing ✓
  6. gradinghidden tests 3/3 · 14/14 → resolved
Recorded offline run. The “model” is a script of hand-written answers.
scroll

the workflow · seven scenes

Generating a patch is the easy part. Trusting it is the work.

  1. 01 · the problem

    A bug report arrives.

    Issue text is untrusted input. A plausible change can be written in seconds, but plausible is not correct, and the repository itself may be hostile.

    Frequently updated cache entry gets evicted

    One sensor reports far more often than the others, so its entry is rewritten constantly with put(). Even so, that entry keeps disappearing from the cache…

    untrusted textinj-001-cachekit

  2. 02 · investigation

    Read a pinned snapshot, never a live checkout.

    The repository must be on an allowlist. Its commit is pinned and read from git objects. The model explores with four read-only tools and no shell.

    • cachekit/ tree 5410f9e6…
    • README.md
    • src/cachekit/
    • __init__.py
    • lru.py put()
    • tests/
    • test_lru.py
    • list_files
    • read_file
    • search_code
    • outline

  3. 03 · first proposal

    One fix, one regression test.

    Edits are search-and-replace and must match exactly once. PatchPilot applies them and computes the diff itself. This one touches the right method and changes nothing that matters.

    src/cachekit/lru.py
  if key in self._entries:
      self._entries[key] = (value, expires_at)
+     self.hits += 0
      return

    + tests/test_rewrite_recency.py

  4. 04 · failure

    The sandbox rejects it.

    Both versions run in a container with no network and no credentials. The new test fails before the patch, as it should, and still fails after it. Not verified.

    original code · test failsheld

    patched code · test failsdid not hold

    unverified

  5. 05 · repair

    Feedback from its own checks, within a budget.

    In repair mode the model is told what its visible checks found, nothing from hidden tests, and tries again. At most five proposals, three by default, each applied to the original files.

    2 proposals left

  6. 06 · verification

    Verified by the agent. Then judged by someone else.

    The agent’s checks give only candidate_verified. After the run, hidden tests the agent never saw are added to a tree rebuilt from the pinned start. Only they decide resolved.

    +     self._entries.move_to_end(key)

    fail-to-pass3/3

    pass-to-pass14/14

    resolved

  7. 07 · human approval

    A person, not the agent, authorizes the write.

    The proposal shows every changed file in full. A person approves it by typing the start of its patch digest. Only then can one draft pull request be opened. Nothing is ever merged.

    3f9c2a7be1d40c86a5e2f71b…

    approved by a persondraft PR

    validated once on a private synthetic repository

interactive architecture

Three trust zones. One direction of trust.

The host orchestrator is trusted and never runs repository code. The sandbox is where untrusted code runs. The evaluator holds hidden tests the agent never sees. Pick a flow to trace it, or select a component for its rule and the tests behind it.

scripted offline replay

A wrong patch, caught and repaired.

The repair-mode run of scenario wrong-then-right on task inj-001-cachekit. Every diff, check and feedback message is taken from the recorded offline run. The answers were written by hand: this shows the pipeline at work, not a live model.

    source · offline scripted run · inj-001-cachekit · wrong-then-right · repair mode

    engineering

    Every safety property is code, and every control has a test.

    The values below are the defaults in code. The security tests attack a real container, not configuration flags.

    01 · containment

    Sandbox isolation

    Repository code, including the candidate’s tests, runs only in a container. The host never imports or executes it, and test commands come from the task manifest, never from model output.

    • Network, writes outside the workspace, privilege, resource limits and leftover processes are attacked by tests in a real container
    • The container shares the host kernel
    • Tests can forge their own results in the same process (kept as a known, expected failure)
    host · trusted
    container · untrusted
    • --network none
    • --read-only
    • --user 10001:10001
    • --cap-drop ALL
    • no-new-privileges
    • --pids-limit 128

    memory512 MB

    cpu1 core

    time60 s

    output64 KiB

    default per run · bar: share of the maximum the settings allow

    internetblocked

    02 · bounds

    Tool boundaries, patch rules and budgets

    The model gets four read-only tools and no shell. Paths are confined to the snapshot. Proposals are checked against the patch policy before anything runs. Every loop has a ceiling, enforced outside the model.

    calls from the scripted benchmark scenarios

    1. list_files {"prefix": "internal"}error · nothing returned
    2. read_file {"path": "../internal/hidden_tests/README"}error · outside snapshot
    3. search_code {"query": "fail_to_pass"}error · nothing returned
    4. add tests/conftest.pypolicy: rejected
    5. edit src/cachekit/lru.py · 1 lineallowed
    model turns
    20
    tool calls
    60
    proposals
    3max 5
    files / lines per patch
    5/ 300
    ledger cap per run
    $0.75
    Gemini pacing
    4per 60 s

    03 · memory

    Feedback, retries, checkpoints and resume

    The repair loop is plain functions over a small JSON state machine. State is saved before every model request and after every step. A resume is refused if the spend ledger shows any request after the checkpoint, so a request whose outcome is unknown is never repeated.

    • JSON with a schema version, a SHA-256 over the state, atomic replace and fsync, 16 MB at most
    • Limits and counters are restored, so restarting cannot reset a budget
    • Resume is offline only for now, replaying recorded answers
    proposeevaluatefeedbackdone

    running · checkpoint written after each step

    { "schema_version": 1, "step": "evaluate", "attempt": 1, … }

    04 · separation

    Hidden-test evaluation and secret redaction

    Hidden tests never enter the agent’s files, images or prompts; tests check every message sent. Known secret values are replaced before anything is written to traces, reports, checkpoints or proposals. Raw traces stay local.

    what the agent sees

    • issue text
    • src/cachekit/lru.py
    • tests/test_lru.py
    • hidden tests: none

    what the evaluator holds

    • 3 fail-to-pass tests
    • 14 pass-to-pass tests
    • reference fix
    • sent back to agent: never

    a trace line, before it is written

    GEMINI_API_KEY=PATCHPILOT-CANARY-NOT-A-KEY

    05 · consent

    Only patchpilot approve creates an approval, and only when a person types the start of the patch digest. It is bound to the whole proposal, expires after 24 hours, and covers one draft PR. Try the same check here; nothing is sent.

    patch digest · sha-256 · example value

    demo

    every run ends in exactly one status

    • candidate_verified
    • unverified
    • rejected_policy
    • invalid_proposal
    • budget_exceeded
    • model_refused
    • failed_infrastructure

    Infrastructure failures are kept apart from patch failures. A GitHub write with an unknown outcome is marked uncertain and never repeated.

    evaluation · scripted

    Workflow validation, not model performance.

    scripted · offline

    The “model” is a script of hand-written answers. The real sandbox and the real hidden-test grading ran, but these results are not the success rate of any language model. The gap between modes is built into the scenarios: four of the five give the good answer second.

    Hidden-test outcome per scenario and mode. Each square is one run on one of the four synthetic tasks. Hover or focus a square for details.

    What “resolved” means

    After the run ends, grade_candidate rebuilds the tree from the pinned start, adds the hidden tests last, and requires every fail-to-pass test to pass and every pass-to-pass test to keep passing. The agent never sees these tests, and the result is never fed back.

    Limits

    • Four synthetic tasks written for this project; no real open-source bugs yet
    • Answers scripted by the same developer who wrote the tasks; one deterministic run each
    • The live comparison with a real model is pending

    Source: evals/results/scripted-benchmark.md, regenerated with uv run patchpilot benchmark --scripted.

    current status

    All five release gates passed. Still experimental.

    Verified

    • 1,061 tests pass (3 skipped, 1 expected failure), 94% coverage, Ruff and strict mypy clean
    • One live repair by local qwen3:8b: 5 tool calls, a rejected first proposal, then a fix the hidden tests confirmed
    • Hosted Linux CI passed on 469074e, Docker sandbox and grading tests included
    • One approved draft PR on a private synthetic repository: branch matched the proposal byte for byte; a second submission created nothing
    • Sandbox limits attacked in a real Docker container, locally and on Linux CI
    • Allowlist, path confinement, patch policy, hidden-test isolation
    • Spend ledger, pacing, attempt caps, no model fallback
    • Checkpoint validation and fail-closed resume

    Simulated

    • Model: scripted, hand-written answers in the benchmark, the replay and the tests
    • GitHub in tests: a local fake GitHub API (tests/fake_github.py) covers failures and reconciliation

    Pending

    • Model benchmark: one live run only; no multi-run, multi-task real-model success rate
    • Hosted models: free-tier Gemini answered all 9 requests with HTTP 503
    • Real tasks: the four tasks are synthetic
    1. AOffline engineeringpassed
    2. BReal-model validationpassed · one local run
    3. CReal GitHub draft PRpassed · approved draft PR
    4. DHosted Linux CIpassed · run 43586146851
    5. EPublic releasepassed · experimental

    Documented, not solved: tests can forge their own results in the same process, the container shares the host kernel, and the approval is an unsigned local file. See the security review and CI status.

    source and documentation

    Read the code behind every claim.

    repositoryDj1236/patchpilot-aiPython 3.12 · uv · MIT license designArchitectureTrust zones, data flow, components run itSetup and usageOffline demo, configuration evidenceEvaluationMethod, scripted results, limits securitySecurity reviewThreats, controls, residual risks agentRepair loopFeedback, checkpoints, resume writesGitHub submissionApproval binding, drafts only statusRelease gatesWhat passed, what is pending

    run the offline demonstration · no API key needed

    uv sync
colima start                              # macOS: the sandbox needs a Docker engine
uv run pytest                             # about 1,050 tests
uv run patchpilot benchmark --scripted