AI-first Software Engineering Playbook
Tips and tricks for enforcement and control against LLMs going off the hinges!
This is a living playbook: patterns extracted from running AI agents on a real production codebase. Every entry exists because an agent found a way to fail that the previous version didn't prevent. It will grow as new failure modes surface and new enforcement patterns prove out.
Stack: Claude Code (primary agent), its Agent Teams for spawning independent reviewers, OpenAI Codex CLI as second-model reviewer.
The focus is coordination and review architecture.
---
Table of Contents
- Foundations (F): Agent setup, rule enforcement, scope control
- Review Architecture (R): Multi-model review loops, convergence, escalation
- Quality Gates (Q): Holistic review, risk scaling, quality enforcement
- Enforcement and Observability (E): Completion vocabulary, audit trails, trust boundaries
- Context and Memory (M): Context loss, automation, learning loops
-----------------
Foundations
F1. Keep agent instructions ruthlessly trim
CLAUDE.md (or equivalent) is loaded every session. Bloated instructions burn context tokens and bury critical rules. Move details to linked guidance files; keep the root file under tight control.
The root file is the agent's working memory. Every line competes for attention, and buried rules get ignored.
F2. Enforce rules with fail-closed hooks, not prose
Every rule you document but don't enforce will eventually be violated. If the check can't run, the write is blocked, not allowed.
Agents find creative workarounds for every soft constraint within days. The hook is the rule; the documentation is just commentary.
F3. Define what the agent must NOT touch
Agents "helpfully" refactor surrounding code, add docstrings, rename variables. Explicit negative scope prevents invisible creep. Include "do not modify files outside [list]" in every task prompt, then verify compliance post-hoc (see E3).
Without this, you review one function and discover the agent also reorganized three imports, renamed a variable "for clarity," and "improved" a comment.
F4. Dual-loop every plan before implementation
Run the full critique loop on plans first. Use the same multi-model review process described below. A flawed plan produces code that's correct-but-wrong, and you only discover it after hours of implementation.
Catching a bad assumption in a plan costs minutes; catching it in code costs hours of throwaway work.
----------------------------
Review Architecture
R1. Never let an agent self-assess completion
Self-review is step one, not the final word. Agents are optimistic completers. They want to tell you things are done. Independent review is the only reliable signal.
Treat self-assessment like a developer saying "looks good to me" on their own PR: necessary but never sufficient.
R2. Run two review stages on different models
First stage: Claude Agent Teams (self-review + 2 independent reviewers per iteration, same model, separate context windows). Second stage: Codex critique rounds (different model).
Different training pipelines produce different systematic failures. In practice, each model consistently catches classes of issues the other misses. Cross-model review is defense-in-depth applied to code quality.
R3. Require 2 consecutive clean rounds before advancing
"Clean" = zero findings across all severity levels (Critical/Major/Minor/Questions). Exactly zero. Caps (R5) prevent infinite loops when convergence stalls.
A single clean pass can be a fluke. Two consecutive passes dramatically reduces the chance of a false negative. Same logic as requiring multiple green CI runs before merging.
R4. Fresh context each iteration
Fresh context means each review pass is independent. No accumulated assumptions from prior rounds. Tradeoff: occasional re-raised issues from lost context, but independence beats shared blind spots.
Reviewers who've seen prior iterations start pattern-matching against previous findings instead of reading the actual code.
R5. Set iteration caps
Ours: 10 for Claude, 5 for Codex. The second model's blind spots differ from the first, so residual issues get caught quickly rather than oscillating. Calibrate to your domain. Security-critical code may need more.
Without caps, noisy reviewers create infinite loops. Watch for oscillating findings where each round introduces a new minor that the next round reverts. That loop won't converge. Ask me how I know.
R6. Break-glass path
Caps hit: block and escalate to a human. Never silently accept remaining issues. The system admits failure rather than shipping silently.
Without an explicit escalation path, the temptation is to just merge after N rounds and hope for the best.
----------------
Quality Gates
Q1. Review code, tests, and docs as a single artifact
Separate reviews miss integration gaps. A test that passes but doesn't verify documented behavior is worse than no test.
Docs say one thing, tests verify another, code does a third. Holistic review catches these; sequential review of each in isolation doesn't.
Q2. Version-control your agent instructions with the same rigor as code
CLAUDE.md changes get changelogs and semver. When behavior regresses, diff the instructions to find what changed.
Undocumented instruction edits are invisible bugs. The agent starts behaving differently and nobody knows why because the change wasn't tracked.
Q3. Invest in assertion quality, not just coverage
Agents generate high-coverage test suites with meaningless assertions. Every line is hit but nothing is actually verified. 100% coverage, 0% confidence. False confidence from weak tests is harder to fix than honest gaps you know about.
Spot-check generated tests for assertion strength as part of code review.
Q4. Scale gates to risk
Major changes: full dual-model review loop. Trivial changes (≤5 lines, ≤2 files, no behavioral or security-sensitive change): skip the full loop. Always full-gate auth, feature flags, and CI config regardless of size. Enforce this with path-based hooks that detect sensitive file changes.
A one-line auth bypass is as dangerous as a 500-line refactor.
Q5. Make test passes a hard gate
"Tests were failing" never appears alongside "declared complete." An agent that can declare success without green tests will do so eventually. Usually on a Friday.
Enforce this with a hook that verifies test status before allowing any completion claim: not a policy document, an automated check.
-------------------------------------
Enforcement and Observability
E1. Ban "done" without evidence
Require "implemented, awaiting review" as the only allowed pre-review status, enforced by a hook that blocks session end without proof (green tests, clean lint, diff within scope).
Agents are eager to close tasks. Suspiciously eager. The vocabulary constraint forces them through the review gate.
E2. Log every review iteration with structured severity counts
Without an audit trail, you can't tell if quality is improving or oscillating. This data feeds into tuning caps and review allocation.
Over time, patterns emerge: certain change types consistently need more rounds. That's signal for where the architecture needs investment.
E3. Treat agent output like untrusted input
Validate paths, check scope, verify claims before allowing writes. Agents will bypass soft constraints if given the chance.
Verify that every file written was within the claimed scope, and that no writes happened to unassigned files. Trust but verify doesn't work here. Just verify. Then verify the verification.
E4. Front-load review budget on early iterations
In this setup, the first Claude iteration tends to find the structural issues. The code is least-clean before any fixes land, so there is simply more to catch. Later iterations trend toward style nits.
Track cost-per-finding per iteration and drop review intensity once the curve flattens, but account for fresh-context re-raises (R4), which can surface structural findings in later rounds. Don't spend iteration 8 budget chasing the same class of minors that iteration 3 found.
E5. Treat every enforcement gap as a design bug
Every mechanism here exists because an agent found a way around the previous version. Build for proof, not trust.
Agents take the path of least resistance. Design the system so the path of least resistance is also the correct path.
-------------------------
Context and Memory
M1. Design for context loss
Long sessions hit context limits and get compacted. The agent loses mid-task details. Auto-checkpoints, session state files, and persistent memory survive compaction. Without this, agents forget constraints and repeat mistakes. The same mistakes. With the same confidence.
Checkpoint before every compaction and maintain a structured memory file that persists across sessions. The agent reads its own notes on startup.
M2. Automate every frustration
Every time an agent violates a convention, ask: can this be a hook instead of a comment in a doc? The answer is almost always yes. Translate recurring frustrations into automated checks that run on every edit, every commit, every session start.
The compounding effect is significant. After a few months, the enforcement layer catches classes of mistakes before the review loop even starts. The agent never needs willpower; the system removes the option to fail.
M3. Build virtuous learning loops
After every enforcement gap or recurring mistake, run a structured reflection: what went wrong, what class of error does this represent, and what systemic prevention stops the entire class, not just this instance. Persist these learnings in the agent's memory files so they survive across sessions.
Keep memory files lean: a compact index loaded every session, with detailed notes in linked topic files the agent reads on demand. Periodically prune outdated entries. Learnings that have been encoded into hooks or automated checks no longer need to live in memory. Each failure makes the next session stronger; the feedback loop compounds.
---
This playbook is a snapshot. Expect it to grow.
---
I'm Alex Ragalie, a Senior Software Engineer based in Zurich, Switzerland.
I build production systems where AI agents write, review, and ship code alongside human engineers.
The patterns in this playbook come from running this architecture daily in real codebases of high performance and high complexity (500k+ lines of code, testing lines excluded).
If you're integrating AI agents into your development workflow and want to avoid the months of failure modes I've already mapped, I'm available for consulting and advisory engagements, exclusively on a per-hour basis (not interested in full time roles).
Architecture reviews, process design, hands-on implementation. Whatever gets your team from "agents sometimes help" to "agents reliably ship."
Reach out on LinkedIn or visit aragalie.com to get in touch.