J-Will Builds
Build Log · Entry 001

The week my AI crew audited itself

July 2026 · 4 minute read

Every system that runs on instructions eventually drifts from them. Config files go stale, rules contradict each other, and nobody notices because nobody rereads the manual. My studio runs on instructions: context files, permissions, scheduled agents, brand rules. So I ran an experiment: I asked a fleet of AI agents to audit the very configuration they run on, and tell me what was quietly working against the mission.

Six agents fanned out across the system, one per surface, and a second wave adversarially re-verified every finding so I wasn't acting on hallucinations. What they confirmed was humbling.

What they found

  • The docs asserted a false reality. My flagship product's context file still described it as pre-launch, five weeks after launch day. Marketing reference docs promised plan features the product doesn't sell. An AI reading those files would confidently generate wrong customer-facing copy.
  • A rebrand never propagated. We had renamed a core feature months earlier. The new name lived in one scheduled agent's instructions; the old name lived in the docs, the legal pages, and half the product UI. Four sources of truth, no agreement.
  • Permissions contradicted the rules. The written policy said production deploys need confirmation. The permission layer auto-approved them. Whichever one an agent hit first won.
  • Fifty design exports were buried in the repo root, drowning the signal in every status check.

What changed

Everything above got fixed the same day: docs rewritten to match reality, the rename finished across product and legal pages, permissions aligned with policy, and two deterministic guard hooks installed so the most dangerous mistakes are now structurally impossible instead of merely discouraged. The crew proved the guards fire by attempting the forbidden operations and getting blocked.

The lesson that stuck: good intentions rot, guardrails don't. An instruction is a suggestion; a hook that blocks the command is a rule. If your AI setup matters, make the important rules structural.

The meta-lesson matters more. AI agents are spectacular at producing work and mediocre at noticing when the ground truth under that work has shifted. The fix isn't smarter agents; it's building the audit into the system: periodic self-review, adversarial verification, and a human who reads the report. That's the studio's whole operating thesis, and now it has receipts.