The AI Coding Agent War Is No Longer About Models — It’s About the Harness

The model may write the code, but the harness decides what it sees, what it can do, how safely it acts, and whether the result is actually ready to ship. Give two AI coding products the same model, the same repository, and the same task. One agent finds the correct service, traces the execution path, edits the relevant files, runs the right tests, detects an edge case, and returns a clean, reviewable change. The other searches randomly, reads half the repository, edits the wrong abstraction, consumes its context window, and confidently announces: > “The issue has been fixed.” The model may be identical. The difference is the harness. For the first generation of AI coding tools, the competition was largely about model intelligence: stronger code completion, better reasoning, larger context windows, and higher benchmark scores. Those things still matter. But they are no longer the whole battle. The new competition is about everything surrounding the model: - How the agent gathers and prioritizes context - How it searches and understands a repository - Which tools it can use - How those tools are described and secured - Where commands are executed - Which actions require approval - How failures are represented - How context is compacted during long tasks - How work survives across sessions - How code changes are tested and verified - How developers inspect and control the run That surrounding execution system is the agent harness. OpenAI defines the harness as the system around the model that maintains context, uses tools, exposes progress, handles failure, requests approval, and returns useful results. Its Codex harness manages conversation state, streaming execution, tools, sandbox policies, approvals, and multi-turn work. DeepSeek expresses the separation in an even simpler phrase: “Agent = Model + Harness.” [OpenAI Codex platform documentation][openai-codex-platform] [DeepSeek Harness][deepseek-harness] The model supplies intelligence. The harness turns that intelligence into reliable action. --- ## Table of Contents 1. The Model Is No Longer the Whole Agent 2. The Evidence: Same Model, Different Harness, Different Result 3. What Is an Agent Harness? 4. Context Engineering 5. Tool Design 6. The Agent Loop 7. Safe Code Editing 8. Sandboxing and Permissions 9. Verification and Recovery 10. Memory and Sessions 11. Observability 12. Multi-Agent Orchestration 13. The Product Experience 14. How Leading Platforms Reveal the Strategy 15. Why Models Still Matter 16. A Realistic Example 17. Production Harness Blueprint 18. How to Evaluate Coding Agents 19. What Comes Next 20. Final Thought 21. FAQ 22. References --- ## The Model Is No Longer the Whole Agent A language model can reason about code. It can propose an implementation, generate a function, explain an error, or recommend an architectural change. But a model by itself cannot reliably: - Discover which files matter in an unfamiliar repository - Determine which documentation is current - Decide how much context should be loaded - Safely execute shell commands - Prevent an edit from overwriting concurrent work - Recover from a failed tool invocation - Track progress across a long-running task - Enforce filesystem, network, and credential boundaries - Prove that the final implementation satisfies the request Those responsibilities belong to the harness. A useful mental model is: text Agent Performance ≈ Model Capability × Context Quality × Tool Quality × Execution Reliability × Verification Strength × Safety and Recovery This is not a literal scientific formula. It is an engineering model. The important point is that agent quality behaves more like a multiplicative system than an additive one. A brilliant model with poor context can solve the wrong problem. A brilliant model with unreliable tools can corrupt files or repeat failed actions. A brilliant model without verification can generate plausible but incorrect code. A brilliant model with unrestricted system access can become a security incident. The model determines the potential ceiling. The harness determines how much of that potential becomes dependable engineering work. --- ## The Evidence: Same Model, Different Harness, Different Result The importance of the harness is no longer only an architectural opinion. Researchers are beginning to measure it directly. > Important research note: Several 2026 results discussed below are recent preprints. They are useful evidence, but they should not be treated as the final word on agent evaluation. ### 1. Same Model, Different Harness A 2026 preprint kept model weights and tasks fixed while changing how the harness handled old tool results, context pressure, and repeated or stalled work. Under a constrained 20,480-token window on a 169-task SWE-bench Verified cohort, the modified harness increased complete solutions from 43 to 72. The same frozen harness changes also improved outcomes for additional model families without model-specific tuning. [Same Model, Different Harness][same-model-different-harness] The lesson is not that one compaction strategy solves every agent problem. The lesson is that an unchanged model can produce materially different results when the surrounding execution system changes. ### 2. The Scaffold Effect Another 2026 preprint evaluated the same models across Goose, OpenCode, and OpenHands-SDK on a subset of Terminal-Bench Pro. Harness choice produced up to a 40× difference in tokens consumed per solved task. Pass-rate differences in that experiment were smaller—generally between zero and eight percentage points—but each harness exhibited distinct failure patterns, including excessive reasoning, verification failures, idle loops, and maximum-turn exhaustion. [The Scaffold Effect in Coding Agents][scaffold-effect] That distinction is commercially important. Two harnesses can achieve similar pass rates while having radically different: - Cost - Latency - Human oversight requirements - Failure behaviour - Operational predictability ### 3. SWE-agent and the Agent-Computer Interface The 2024 SWE-agent work demonstrated that interface design can substantially affect how language-model agents navigate repositories, edit code, and run tests. Its central argument was that agents benefit from purpose-built computer interfaces rather than being dropped into an unrestricted shell with weak feedback. [SWE-agent paper][swe-agent] This established an enduring principle: > How an agent interacts with a computer can be almost as important as what the model knows. ### What These Studies Do—and Do Not—Prove These studies do not prove that the harness always matters more than the model. Model capability remains fundamental, and results depend on the task, benchmark, context limit, toolset, time budget, and evaluation method. They prove something more practical: > A model name alone is no longer enough to describe the system being evaluated. The real comparison unit is: text Model + Harness + Tools + Context Policy + Execution Environment + Verification Policy When teams compare “Model A versus Model B” through different products, they may actually be comparing two complete systems—not two models. --- ## What Is an Agent Harness? The word harness is used in slightly different ways across the industry. In the broad product sense, it is the complete execution layer surrounding the model. A production coding-agent harness typically owns: 1. Prompt and context construction 2. Repository discovery and code search 3. The reasoning and tool-execution loop 4. File editing and command execution 5. Context compaction and session persistence 6. Sandboxing and permission enforcement 7. Testing and verification 8. Error recovery and loop detection 9. Tracing, observability, and replay 10. Human approval and intervention 11. Skills, plugins, hooks, and MCP integrations 12. Subagent and parallel-work orchestration A high-level architecture looks like this: mermaid flowchart TD U[User Goal] --> I[Instruction and Context Layer] I --> O[Agent Orchestrator] M[Model Provider] <--> O S[Session and Memory Store] <--> O O --> T[Tool Gateway] T --> F[File and Code Tools] T --> C[Command Runner] T --> X[MCP and External Services] C --> B[Sandbox] F --> B O --> V[Verification Engine] V --> O O --> P[Policy and Approval Engine] P --> T O --> R[Trace and Evaluation Store] O --> D[Diff, Evidence, and Final Result] The harness is not just middleware forwarding messages between a user and a model. It is a stateful software runtime. --- ## 1. Context Engineering: Showing the Model the Right Reality Prompt engineering asks: > What instructions should we give the model? Context engineering asks: > What information should the model see at this exact step—and what should be excluded? That second question becomes crucial in large repositories. A coding agent may have access to: - Thousands of source files - Years of Git history - Architecture documents - Issue descriptions - Build logs - Test failures - Database schemas - API specifications - Previous agent turns - User corrections - External documentation Dumping all of this into the model is not a strategy. It creates noise, increases cost, and can hide the most important constraint inside irrelevant material. A strong context engine must decide: - Which repository instructions apply to the current path - Which services and modules are relevant - Which files should be read fully - Which files require only selected ranges - Which symbols and references should be explored - Which previous decisions remain valid - Which tool outputs should be summarized - Which logs can be discarded - What must remain pinned during compaction - When a subagent should isolate a noisy investigation ### Larger Context Is Not the Same as Better Context A huge context window can help, but it does not remove the need for selection. A million tokens of stale documentation, generated files, duplicate logs, and irrelevant code can perform worse than a much smaller context containing: - The user’s real objective - Applicable repository instructions - The relevant call chain - The current schema - The failing tests - The constraints that define success A mature context manager should provide: - Repository-aware search - Symbol and dependency discovery - Hierarchical project instructions - Selective document retrieval - Tool-output summarization - Context deduplication - Token budgeting - Long-session compaction - Explicit preservation of goals, decisions, and unfinished work OpenAI’s documentation on long-running agents describes compaction as a way to keep stateful work moving when workflows approach context limits. [OpenAI: Shell, Skills, and Compaction][openai-compaction] ### Progressive Disclosure Beats Permanent Prompt Inflation Not every skill, policy, tool description, and project document should be loaded at startup. A better pattern is progressive disclosure: 1. Start with a compact system contract. 2. Discover relevant repository instructions. 3. Load a skill only when the task needs it. 4. Retrieve files only when evidence points toward them. 5. Summarize completed investigations before moving on. The winning harness will not be the one that reads the most code. It will be the one that consistently reads the right code. --- ## 2. Tool Design: Giving the Agent Better Hands A model cannot inspect a repository, edit a file, run tests, or query an external service through intention alone. It needs tools. Typical coding-agent tools include: - Search files and symbols - Read files or selected ranges - Find references - Apply targeted edits - Create, move, or delete files - Execute shell commands - Inspect Git status and diffs - Run tests, builds, linters, and type checks - Search documentation - Interact with issue trackers - Query databases and observability systems - Invoke external services through MCP But more tools do not automatically create a better agent. ### Tool Overload Is a Real Failure Mode If an agent is exposed to ten overlapping search tools, several generic shell tools, and dozens of poorly named MCP operations, it must first solve a tool-selection puzzle before solving the user’s problem. Weak tool design causes: - Incorrect tool selection - Invalid parameters - Excessive tool descriptions in context - Huge unstructured outputs - Repeated permanent failures - Ambiguous completion states - Unsafe retries - Permission confusion A strong tool contract should include: yaml name: run_targeted_tests purpose: Run a bounded test selection for the current change. inputs: test_paths: type: array items: string timeout_seconds: type: integer minimum: 1 maximum: 900 permissions: filesystem: read network: denied workspace_write: temporary-test-output-only behaviour: cancellable: true retry_safe: true max_output_chars: 30000 outputs: status: passed | failed | timed_out | cancelled | environment_error discovered_tests: integer passed: integer failed: integer summary: string bounded_log: string The exact schema will differ across systems, but the principles are stable: - One clear responsibility - Typed inputs - Structured outputs - Bounded response size - Explicit error categories - Cancellation and timeout support - Safe retry semantics - Permission metadata - Auditability ### Tool Errors Should Teach the Agent What to Do Next Bad error: text Command failed. Useful error: json { "status": "permission_denied", "reason": "Network access is disabled in workspace-write mode", "recoverable": true, "allowed_next_actions": [ "request_network_approval", "use_cached_dependencies", "continue_without_installation" ] } The purpose of an error is not only to report failure. It should help the orchestrator choose the correct recovery path. ### MCP Is an Integration Protocol, Not the Entire Harness The Model Context Protocol allows servers to expose discoverable tools with schemas. The current specification recommends visible tool activity and keeping a human able to deny invocations. [MCP tools specification][mcp-tools] MCP can be an important tool layer, but it does not automatically provide: - Agent planning - Context management - Sandbox isolation - Retry logic - Session persistence - Verification - Cost control - Observability - Approval UX “Supports MCP” is therefore not the same as “has a production-ready harness.” --- ## 3. The Agent Loop: Turning a Response Into a Workflow A chatbot follows a simple interaction: text Prompt → Response A coding agent needs a loop: text Observe → Understand → Plan → Act → Inspect → Verify → Continue or Stop The harness manages this loop. It sends the task and context to the model. The model requests a tool. The harness validates permissions, executes the tool, records the result, updates the task state, and asks the model what to do next. A simplified version might look like this: pseudo while run.status is active: context = context_manager.build(run) decision = model.next_action(context) if decision.type == "tool_call": policy_result = policy_engine.evaluate(decision) if policy_result == DENY: run.add_event(tool_denied(decision)) continue if policy_result == ASK: run.status = WAITING_FOR_APPROVAL surface_approval(decision) continue result = tool_gateway.execute(decision) run.add_event(result) recovery_engine.update(result) continue if decision.type == "complete": evidence = verifier.check(run) if evidence.sufficient: run.status = COMPLETED else: run.add_event(verification_failed(evidence)) run.status = RECOVERING The hard part is not the loop itself. The hard part is controlling failure. A production loop needs answers to questions such as: - How many retries are allowed? - Which failures are temporary? - Which failures require a new plan? - How do we detect repeated actions? - What counts as meaningful progress? - When should the agent stop searching? - When should it request clarification? - How can the user interrupt or steer the run? - What happens after context compaction? - What evidence is required before completion? ### Failure Classification Matters These failures should not be handled identically: | Failure Type | Appropriate Response | |---|---| | Temporary command timeout | Retry once with bounded backoff | | Invalid tool arguments | Correct the arguments using schema feedback | | Permission denial | Request approval or choose an allowed path | | Test assertion failure | Inspect the failing behavior and revise code | | Missing dependency | Check project instructions before installation | | Context exhaustion | Compact while preserving goals and decisions | | Repeated no-op edits | Stop, inspect the diff, and re-plan | | Permanent environment failure | Return partial progress and explicit blocker | A weak loop forwards every failure back to the model as plain text. A strong loop classifies failures, controls retries, protects state, and makes the next action more informed. --- ## 4. Safe Code Editing Is a Systems Problem Generating a correct code snippet is easier than modifying a living repository safely. A production editing layer must handle: - Multiple edits in the same file - Files changing between read and write - Line-ending differences - Formatting tools rewriting content - Generated files - Binary files - Symlinks and path traversal - Monorepo boundaries - Partially applied patches - Concurrent agents modifying related files - Rollback after a failed implementation - Very large files - Restricted directories ### Editing Should Be Optimistic but Conflict-Aware A safe editing tool can require a content hash or version from the original read: json { "path": "src/payments/invoice-service.ts", "expected_sha256": "9f87...", "edits": [ { "start_line": 118, "end_line": 132, "replacement": "..." } ] } If the file changed after the model read it, the tool should reject the edit rather than silently overwriting new work. ### Changes Should Be Atomic Where Possible For multi-file work, a harness can: 1. Create a checkpoint. 2. Validate all proposed paths. 3. Apply edits to a temporary workspace. 4. Run formatting and focused checks. 5. Commit the complete edit set only after validation. 6. Restore the checkpoint if the operation fails irrecoverably. Gemini CLI documents automatic checkpoints before AI-powered file modifications so users can restore earlier project state. Claude’s Agent SDK exposes file checkpointing and rewind features as part of its control layer. [Gemini CLI checkpointing][gemini-checkpointing] [Claude Agent SDK][claude-agent-sdk] A coding agent should not merely know how to write code. Its harness must know how to change a repository without damaging it. --- ## 5. Sandboxing and Permissions: Autonomy Without Recklessness Coding agents are useful because they can act. That is also what makes them dangerous. An agent with shell and network access may accidentally: - Delete files outside the repository - Read credentials - Send secrets to an external service - Execute malicious project scripts - Modify system configuration - Install unsafe dependencies - Trigger an irreversible database operation - Use an untrusted MCP tool The correct response is not to eliminate autonomy. The correct response is to define the execution boundary precisely. ### Separate the Control Plane From the Execution Plane OpenAI’s sandbox-agent documentation describes a useful split: - The harness is the control plane: model calls, orchestration, approvals, tracing, recovery, and run state. - The sandbox is the execution plane: files, commands, packages, ports, mounted storage, and snapshots. Keeping those boundaries separate allows trusted infrastructure to retain authentication, billing, audit logs, and approval state while model-directed code runs inside a narrower environment. [OpenAI sandbox-agent documentation][openai-sandbox-agents] mermaid flowchart LR H[Trusted Harness Control Plane] -->|Approved tool request| G[Execution Gateway] G --> S[Sandbox] S --> F[Workspace Files] S --> C[Commands and Tests] S --> N[Restricted Network] S -->|Structured result| G G --> H H --> A[Audit Log and Approval State] ### Permissions Should Be Capability-Based A mature system should distinguish between: - Read-only repository access - Workspace write access - External directory access - Network access - Package installation - Secret access - Destructive commands - External service writes - Production operations For example: text Read repository file → Allow Search Git history → Allow Run existing unit tests → Allow Write inside workspace → Allow in workspace-write mode Install a new package → Ask Access an external domain → Ask or allowlist Read production credentials → Deny Drop a production database → Deny Deploy to production → Separate privileged workflow Gemini CLI’s policy engine supports rules that allow, deny, or require confirmation for tool calls. Its sandbox expansion flow can request additional access for a specific command when restricted paths or network access are needed. [Gemini CLI policy engine][gemini-policy-engine] [Gemini CLI sandboxing][gemini-sandboxing] OpenAI’s Codex guidance similarly separates sandbox mode from approval mode and recommends starting with tight permissions. [Codex sandbox best practices][codex-sandbox-best-practices] ### Prompts Are Not Security Boundaries “Never read secrets” is useful guidance. It is not sufficient enforcement. Real controls should exist below the model: - Filesystem mounts - Credential brokers - Network proxies - Domain allowlists - Command policies - Resource quotas - Approval gates - Secret redaction - Process cleanup A useful design principle is: > Increase autonomy inside a trustworthy boundary instead of removing the boundary. --- ## 6. Verification Is What Separates Generation From Engineering Many weak coding agents use this completion rule: text Edit applied → Task complete A production agent should use something closer to: text Edit applied ↓ Inspect diff ↓ Run focused tests ↓ Run type check, lint, or build ↓ Analyze failures ↓ Repair or re-plan ↓ Run broader validation when justified ↓ Confirm requirements ↓ Task complete Successful tool execution is not proof of a successful implementation. A test command may exit successfully while discovering zero tests. A build may pass while the requested behavior remains missing. A unit test may pass while another service breaks. A migration may compile while being unsafe for a large production table. ### Verification Should Check the Claim, Not Just the Command A strong verifier should ask: - Does the requested behavior now exist? - Were relevant tests discovered and executed? - Did existing behavior regress? - Does the diff contain unrelated changes? - Are generated files current? - Are migrations safe and reversible? - Were debug statements or secrets introduced? - Does the implementation follow architectural constraints? - Does the evidence justify the completion claim? ### Verification Should Feed the Next Iteration The strongest pattern is a closed loop: text Implementation ↓ Verification failure ↓ Structured diagnosis ↓ Updated context ↓ Targeted correction ↓ Re-verification A useful verification result might look like: json { "status": "failed", "checks": [ { "name": "targeted_tests", "status": "failed", "discovered": 18, "passed": 17, "failed": 1, "failure_summary": "Duplicate invoice created under concurrent retry" }, { "name": "typecheck", "status": "passed" } ], "recommended_next_step": "Inspect transaction boundary and idempotency constraint" } OpenAI’s iterative-repair guidance describes agent workflows that generate an output, validate it, and use feedback for the next pass. [OpenAI iterative repair loops][openai-repair-loops] The model can claim the task is complete. The harness must demand evidence. --- ## 7. Memory, Sessions, Checkpoints, and Recovery Real engineering tasks do not always finish in one model call. An agent may need to: - Continue for hours - Pause for human approval - Resume after an application restart - Recover from a failed command - Revert an incorrect edit - Fork an alternative implementation - Preserve decisions across context compaction - Continue after the user changes direction This requires durable session state. ### Memory Should Be Layered A good system separates different kinds of memory. #### Repository Instructions Stable information such as: - Architecture boundaries - Build commands - Testing conventions - Security rules - Definition of done #### Task State Information relevant to the active run: - User goal - Current plan - Completed steps - Files inspected - Files changed - Pending approvals - Remaining work - Verification status #### Learned Preferences Carefully selected recurring information: - Review expectations - Preferred workflows - Common corrections - Output format preferences #### Ephemeral Tool Output Information that can usually be summarized or discarded: - Long logs - Search result dumps - Repeated directory listings - Intermediate diagnostics ### Memory Must Be Selective and Bounded Remembering everything is not intelligence. Unbounded memory causes: - Higher context cost - Stale assumptions - Repeated outdated decisions - Privacy risks - Difficulty identifying the real source of truth The right question is not: > How much can the agent remember? It is: > What must the agent remember to continue correctly? ### Sessions Should Be Event-Based Instead of storing only a transcript, a harness can persist structured events: json { "event_id": "evt_01842", "run_id": "run_771", "timestamp": "2026-08-29T14:26:18Z", "type": "tool_result", "tool": "run_targeted_tests", "status": "failed", "artifacts": ["test-report.json"], "context_policy": { "retain_summary": true, "retain_full_output": false } } Event-based state enables: - Resume - Replay - Forking - Auditing - Context reconstruction - Crash recovery - Cost attribution DeepSeek Harness records model-visible prompts, reasoning, tool calls, results, subagent scheduling, and context injections in an append-only session log. Resume, fork, search, and replay operate over the same event stream. [DeepSeek Harness][deepseek-harness] Anthropic’s Agent SDK exposes sessions, resumption, permissions, hooks, subagents, checkpointing, usage tracking, and OpenTelemetry support. [Claude Agent SDK][claude-agent-sdk] Long-running agents need more than chat history. They need durable execution state. --- ## 8. Observability Is the Difference Between a Demo and a Production System When a normal application fails, engineers inspect logs, metrics, traces, and database state. AI agents need the same operational discipline, plus agent-specific visibility. A production trace should help answer: - What instructions were active? - Which context sources were selected? - Which files did the agent inspect? - Which tools did it call? - What arguments were passed? - How long did each step take? - Which permission decisions occurred? - Where did retries happen? - When was context compacted? - Did a subagent contribute? - What did the run cost? - Why did the agent stop? - What evidence supported completion? A useful trace hierarchy might look like: text Run ├── User request ├── Instruction resolution ├── Context retrieval ├── Model call ├── Tool request ├── Permission decision ├── Tool execution ├── Tool result ├── Context compaction ├── Model continuation ├── File changes ├── Verification └── Final response ### Metrics That Actually Matter Useful production metrics include: | Metric | What It Reveals | |---|---| | Task success rate | Whether the requested behavior was delivered | | Accepted-change rate | Whether developers considered the output useful | | Regression rate | Whether the agent damaged existing functionality | | Tokens per solved task | Real inference efficiency | | Cost per accepted change | Business value rather than raw activity | | Wall-clock time per solved task | Developer waiting cost | | Tool failure rate | Runtime and integration reliability | | Repeated-action rate | Loop quality | | Human interventions per task | Actual autonomy level | | Permission escalation rate | Security and workflow friction | | Rollback frequency | Editing reliability | | Verification coverage | Confidence in completion | | Run-to-run variance | Behavioural consistency | | Unsupported completion rate | How often the agent claims success without proof | “Number of messages generated” is not a success metric. “Useful changes accepted without regression” is. ### Traces Should Create an Improvement Flywheel Every production failure can become: - A new evaluation case - A better tool schema - A stronger policy - A context-selection improvement - A new recovery rule - A more precise verifier - A better skill OpenAI’s agent-improvement guidance explicitly connects traces, feedback, evaluations, and harness changes into an improvement loop. [OpenAI agent improvement loop][openai-agent-improvement] Without traces, teams repeatedly fix symptoms. With traces, they can improve the system that produced the failure. --- ## 9. Multi-Agent Orchestration: Useful Tool or Expensive Theatre? Multi-agent systems sound impressive: - One agent plans - One investigates the backend - One handles the frontend - One reviews security - One writes tests - One supervises the others Sometimes this works extremely well. Sometimes it is five agents repeatedly summarizing the same repository while multiplying cost and coordination failure. ### Use Subagents When Work Is Genuinely Separable Good candidates include: - Independent architecture research - Security review - Test analysis - Documentation validation - Parallel investigation of unrelated failures - Frontend and backend work behind a stable contract Poor candidates include: - Tightly coupled edits to the same files - Tasks with unclear ownership - Work requiring continuous shared state - Small changes where delegation costs more than execution Each subagent should receive: - A narrow objective - Minimal relevant context - Explicit tools - Scoped permissions - A bounded time and token budget - A structured output contract - Clear stop conditions The parent should receive the result—not the child’s entire working history. Anthropic and Gemini both expose specialized subagents as part of their agent runtimes, while DeepSeek Harness includes subagents and workflows in its standard mode. [Claude Agent SDK][claude-agent-sdk] [Gemini CLI subagents][gemini-subagents] [DeepSeek Harness][deepseek-harness] The future is probably not the largest possible swarm. It is small, specialized, observable delegation. --- ## 10. The Product Experience Is Part of the Harness Developers do not experience a model directly. They experience: - How quickly meaningful progress appears - Whether tool activity is understandable - Whether the agent can be interrupted - Whether new instructions can steer the active run - How approvals are presented - How diffs are reviewed - Whether failed work can be resumed - Whether changes can be reverted - Whether uncertainty is communicated honestly A strong interface should distinguish between states such as: text Understanding Searching Reading Planning Editing Running tests Waiting for approval Recovering Blocked Completed It should show enough information to support trust without overwhelming users with every internal detail. ### Chat Is Not the Only Agent Interface A reusable harness can power: - IDE assistants - Issue-tracker agents - Pull-request reviewers - Operations dashboards - Database migration assistants - Security remediation systems - Internal developer portals - Domain-specific engineering workflows OpenAI’s app-server direction separates the reusable Codex agent loop from the application-owned interface, business context, tools, and approval flow. [OpenAI Codex platform documentation][openai-codex-platform] The interface is not decoration. It is where context, consent, progress, and evidence become visible to humans. --- ## How Leading Platforms Reveal the New Strategy The industry’s major coding-agent platforms increasingly expose harness capabilities as products of their own. | Platform | Harness Direction | Strategic Lesson | |---|---|---| | OpenAI Codex | Open harness, app-server, SDK, threads, streamed events, tools, sandboxes, approvals, and multi-turn work | The reusable agent loop can become infrastructure inside other products | | Claude Agent SDK / Claude Code | Tools, loop, context management, sessions, hooks, subagents, MCP, permissions, checkpointing, plugins, and observability | Coding-agent behavior can be packaged as a programmable runtime | | Gemini CLI | Tool policies, sandboxing, checkpoints, MCP, plan mode, subagents, and configurable execution controls | Reliability comes from an operational layer around model access | | DeepSeek Harness | Plugin-based models, tools, skills, sessions, sandboxes, storage, loops, scheduling, UI, and traceable session history | The harness itself can become an open, composable agent platform | Sources: [OpenAI Codex platform documentation][openai-codex-platform], [Claude Agent SDK][claude-agent-sdk], [Gemini CLI policy engine][gemini-policy-engine], [Gemini CLI checkpointing][gemini-checkpointing], and [DeepSeek Harness][deepseek-harness]. These products are not identical, and this table is not a quality ranking. The important observation is that all of them are investing heavily in capabilities that exist around inference. That is where product reliability is being built. --- ## Models Still Matter—a Lot The title of this article should not be misunderstood. Models remain fundamental. They determine capabilities such as: - Understanding ambiguous requirements - Navigating unfamiliar code - Reasoning across multiple services - Writing correct implementations - Diagnosing failures - Following instructions - Selecting appropriate tools - Maintaining coherence over long tasks - Recognizing uncertainty - Balancing quality, latency, and cost A weak model cannot be transformed into a frontier engineer merely by surrounding it with excellent middleware. The accurate statement is: > The model sets the capability ceiling. The harness determines how consistently, safely, and efficiently the system reaches that ceiling. Model and harness also interact. A tool interface that works well for one model may confuse another. A compaction strategy may preserve the cues one model needs while discarding information another relies on. Different providers may require different: - Prompt structures - Tool schemas - Reasoning controls - Streaming adapters - Retry behavior - Error handling - Context policies The future is not purely model competition or purely harness competition. It is model-harness co-design. But there is an important business difference. Model capability can often be rented through an API. A deeply integrated harness containing company-specific context, tools, policies, evaluations, workflows, and reliability improvements is much harder to copy. That is where a durable product advantage can emerge. --- ## A Realistic Example: One Bug, Two Different Agents Consider this task: > “Fix occasional duplicate invoices created when payment retries occur.” ### A Weak Harness The agent might: 1. Search for the word invoice. 2. Open the first matching controller. 3. Add an in-memory duplicate check. 4. Ignore queue retries and multiple server instances. 5. Never inspect database constraints. 6. Run one unrelated unit test. 7. Claim the bug is fixed. The code may look reasonable. The system is still broken. ### A Strong Harness The same model inside a better harness might: 1. Resolve repository and service instructions. 2. Map payment, invoice, webhook, and queue components. 3. Inspect the database schema and uniqueness rules. 4. Find every invoice-creation call site. 5. Trace retry and concurrency behavior. 6. Identify whether the operation requires an idempotency key. 7. Produce a read-only implementation plan. 8. Create a workspace checkpoint. 9. Apply a transactional or database-level guard. 10. Add concurrent retry tests. 11. Run focused tests, type checks, and migration validation. 12. Inspect the final diff for unrelated changes. 13. Report evidence, residual risks, and rollback notes. The model did not suddenly become smarter. The harness gave it: - Better context - Better tools - Better constraints - Better feedback - Better verification That is the thesis of this article in one example. --- ## Why a Stronger Model Can Still Lose | Harness Problem | Result | |---|---| | Poor context retrieval | The model solves the wrong layer of the system | | Huge unfiltered context | Important constraints disappear inside noise | | Ambiguous tool schemas | Tools are selected or called incorrectly | | Unbounded tool output | Logs consume the working context | | No progress detection | The agent repeats the same investigation | | Weak error handling | Temporary and permanent failures are treated identically | | No edit conflict detection | Newer work is silently overwritten | | No sandbox | A coding mistake becomes a system-level incident | | No verification | Plausible code is mistaken for correct code | | No checkpointing | One bad edit destroys previous progress | | No observability | Teams cannot explain or improve failures | | No cost controls | A small task consumes an unreasonable budget | | Weak stop conditions | The agent continues without increasing confidence | This is why benchmark-leading models can feel disappointing inside poorly designed products. The product may not be exposing the model’s real capability. --- ## A Production-Ready Coding Harness Blueprint A serious coding-agent platform should keep its layers explicit. ### 1. Model Adapter Layer Responsibilities: - Support one or more providers - Normalize tool-call formats - Normalize streaming events - Handle provider-specific reasoning modes - Track usage and cost - Support fallback and routing - Handle cancellation - Normalize transient and permanent errors - Preserve provider-specific strengths instead of forcing a lowest-common-denominator interface ### 2. Instruction and Context Engine Responsibilities: - Resolve repository instructions by scope - Search files, symbols, and dependencies - Retrieve relevant documentation - Deduplicate context - Compress tool results - Maintain token budgets - Compact long sessions - Preserve goals, decisions, risks, and pending work ### 3. Explicit Agent State Machine Useful runtime states include: text UNDERSTANDING PLANNING EXECUTING WAITING_FOR_APPROVAL VERIFYING RECOVERING BLOCKED COMPLETED FAILED CANCELLED Avoid hiding all behavior inside one unobservable loop. ### 4. Tool Gateway Every tool should define: - Input schema - Output schema - Permission level - Timeout - Cancellation behavior - Maximum output size - Retry safety - Error categories - Audit metadata The gateway should also provide: - Tool discovery - Health checks - Output truncation with artifact storage - Secret redaction - Concurrency limits - Per-run budgets ### 5. Policy and Approval Engine Responsibilities: - Allow, ask, or deny actions - Apply filesystem boundaries - Apply network policies - Protect secrets - Restrict external service writes - Require stronger approval for destructive operations - Prevent permission escalation - Record every decision ### 6. Sandbox and Execution Layer Responsibilities: - Isolate commands - Limit CPU, memory, processes, and time - Control writable paths - Restrict network access - Mount only necessary files and credentials - Clean up abandoned processes - Support reproducible environments - Capture output safely - Snapshot and restore workspace state ### 7. Editing and Change Management Responsibilities: - Conflict-aware edits - Atomic multi-file changes - Checkpoints - Rollback - Diff generation - Scope validation - Path traversal protection - Symlink policy - Formatting integration ### 8. Verification Engine Responsibilities: - Select relevant tests - Run builds, linters, and type checks - Validate migrations - Inspect diffs - Enforce architecture rules - Detect unrelated modifications - Check completion claims against evidence - Return structured failures to the agent ### 9. Session and Event Store Responsibilities: - Persist task state - Store plans and progress - Save tool calls and results - Support pause, resume, fork, and replay - Store checkpoints - Preserve compaction summaries - Provide crash recovery - Track cost and latency ### 10. Observability and Evaluation Responsibilities: - Trace every run - Track model and tool latency - Record approvals and denials - Measure task success - Replay failures - Compare models and harness versions - Convert production incidents into evaluations - Detect regressions before release ### 11. Product Experience Responsibilities: - Show meaningful progress - Display tool use clearly - Make approvals understandable - Support interruption and steering - Present diffs and validation evidence - Separate planning from execution - Make blockers honest and actionable --- ### Reference Architecture mermaid flowchart TB UI[CLI / IDE / Web / API] --> API[Agent Application API] API --> ORCH[Orchestrator and State Machine] ORCH <--> CTX[Context Manager] ORCH <--> SES[Session and Event Store] ORCH <--> MOD[Provider Adapters] ORCH <--> POL[Policy and Approval Engine] ORCH <--> VER[Verification Engine] ORCH --> OBS[Tracing, Metrics, and Evals] POL --> TG[Tool Gateway] ORCH --> TG TG --> FILES[File and Code Tools] TG --> SHELL[Command Tool] TG --> MCP[MCP and External Tools] TG --> GIT[Git and Review Tools] FILES --> SBX[Sandbox / Workspace] SHELL --> SBX GIT --> SBX SBX --> ART[Artifacts, Diffs, Logs, Test Reports] ART --> VER VER --> ORCH ORCH --> UI --- ### A Practical Event Model A production harness should make every important transition inspectable. typescript type AgentEvent = | { type: "run.started"; runId: string; task: string } | { type: "context.selected"; sources: ContextSource[] } | { type: "model.started"; provider: string; model: string } | { type: "model.delta"; content: string } | { type: "tool.requested"; tool: string; arguments: unknown } | { type: "approval.requested"; risk: string; rationale: string } | { type: "approval.resolved"; decision: "allow" | "deny" } | { type: "tool.completed"; status: ToolStatus; resultRef: string } | { type: "files.changed"; paths: string[]; checkpointId: string } | { type: "context.compacted"; summaryRef: string } | { type: "verification.completed"; status: "passed" | "failed" } | { type: "run.blocked"; reason: string } | { type: "run.completed"; evidenceRefs: string[] } | { type: "run.failed"; error: StructuredError } | { type: "run.cancelled"; requestedBy: "user" | "policy" | "system" }; This model supports streaming UI, replay, auditing, recovery, and analytics without depending on fragile plain-text parsing. --- ## How Teams Should Evaluate Coding Agents Now Engineering teams should stop evaluating agents only through impressive chat demonstrations. Give each candidate the same real tasks and measure the complete workflow. ### A Better Evaluation Scorecard | Category | What to Measure | |---|---| | Correctness | Did the change solve the actual problem? | | Scope control | Did the agent avoid unrelated edits? | | Context quality | Did it inspect the right code and documentation? | | Verification | Did it run appropriate tests and checks? | | Recovery | Did it handle failures without losing progress? | | Security | Did it respect filesystem, network, and credential boundaries? | | Reviewability | Was the final diff understandable? | | Transparency | Did it provide evidence for completion claims? | | Efficiency | What were the token, compute, and time costs? | | Reliability | Can it reproduce success across multiple runs? | Do not ask only: > “Which model powers this agent?” Also ask: > “What happens around the model before, during, and after every action?” That second question usually reveals more about the product. --- ### Measure Tokens Per Solved Task, Not Tokens Per Request A cheap request that fails repeatedly is expensive. A powerful agent that consumes uncontrolled tokens can also be commercially unusable. The useful unit is not the cost of one model call. It is the cost of a verified, accepted result. Track: text Cost per accepted change = Total model cost + Tool infrastructure cost + Human review cost + Recovery cost + Regression cost This is why the Scaffold Effect’s focus on tokens per solved task is valuable. Similar pass rates can hide enormous differences in efficiency and human waiting time. [The Scaffold Effect in Coding Agents][scaffold-effect] --- ### Evaluate Models and Harnesses Separately Use a two-dimensional matrix: | Test | Model | Harness | Purpose | |---|---|---|---| | A | Fixed | Changed | Measure harness improvement | | B | Changed | Fixed | Measure model improvement | | C | Changed | Changed | Measure full product performance | | D | Multiple | Multiple | Detect model–harness interaction effects | When publishing or reviewing results, disclose: - Exact model and version - Reasoning configuration - System and developer instructions - Available tools - Editing interface - Context-selection strategy - Context limit and compaction policy - Sandbox permissions - Network access - Maximum runtime and turns - Verification steps - Retry rules - Human intervention - Token and cost accounting Without these details, an agent benchmark is not fully reproducible. It is a score for an underspecified composite system. --- ### Build an Internal Evaluation Suite Public benchmarks are useful, but production teams need tasks drawn from their own environment. A strong internal suite should include: - Real bugs from the company’s repositories - Multi-file feature work - Database migrations - Security-sensitive changes - Large-log debugging - Frontend and backend integration - Hidden edge cases - Tasks designed to expose repeated loops - Tasks that require stopping instead of editing - Tasks where the correct response is to ask for approval - Tasks where a partial result is better than an unsafe guess Every escaped production failure should become a regression case. --- ## Build, Buy, or Extend? Most companies should not rebuild every low-level agent component from zero. A practical strategy is: 1. Start with an existing runtime or SDK for the basic loop. 2. Keep provider adapters modular. 3. Build company-specific context and tools. 4. Enforce business-specific security policies. 5. Add verification tied to the real definition of done. 6. Trace every run. 7. Convert failures into evaluations. Invest custom engineering where it creates real differentiation: - Domain tools - Repository understanding - Security controls - Workflow integrations - Verification pipelines - Operational reliability - User experience Do not build a custom orchestrator merely because agent loops look exciting. Build one when existing runtimes cannot satisfy a concrete product or reliability requirement. --- ## Where the Coding-Agent War Goes Next ### 1. Model Switching Will Become Normal Teams will increasingly route tasks based on: - Capability - Cost - Latency - Context size - Tool reliability - Data restrictions - Specialized strengths The harness will preserve the product experience while the underlying model changes. ### 2. Harness Benchmarks Will Become a Separate Category Evaluation will increasingly distinguish between: - Raw model capability - Standardized-harness performance - Full product-agent performance This will make comparisons more honest. ### 3. Vertical Harnesses Will Beat Generic Chat Wrappers A harness designed for database migrations, security remediation, incident response, mobile development, or frontend delivery can provide better context and safer tools than a universal chat interface. The winning product may not be the one that can theoretically do everything. It may be the one that reliably completes one valuable category of work. ### 4. Agent Observability Will Become an Engineering Discipline Teams will need dashboards for: - Agent traces - Failure categories - Tool reliability - Token economics - Context quality - Permission behavior - Verification success - Security events Agent operations will become as important as application operations. ### 5. Security Will Become a Buying Criterion Enterprises will compare agents based on: - Data boundaries - Sandboxing - Approval controls - Secret handling - MCP governance - Auditability - Policy enforcement - Deployment options The most autonomous agent will not automatically win. The most trustworthy autonomous agent may. ### 6. Multi-Agent Systems Will Become More Selective The early temptation is to assign an agent to every subtask. Mature systems will learn where parallelism helps and where it merely multiplies cost, duplicated context, and coordination failure. ### 7. Harness Improvements Will Compound Every traced failure can produce: - A better tool - A stronger policy - A new verification gate - A more useful skill - A context-selection improvement - A recovery strategy - A regression test That creates a reliability flywheel which model switching alone cannot provide. ### 8. Harnesses May Begin Adapting Themselves Recent research is exploring systems that modify or optimize the executable harness around a fixed model. Test-Time Harness Evolution, for example, treats the harness itself as the object being adapted using execution traces rather than changing model weights. This remains early research, but it points toward a future where agents dynamically configure planning, tool use, verification, and recovery for different task distributions. [Test-Time Harness Evolution][tthe] The harness may eventually become an intelligent, adaptive system of its own. --- ## Final Thought The first era of AI-assisted development was about generation. The second era is about execution. Generating code requires intelligence. Reliably changing a production system requires: - Context - Tools - Permissions - Memory - Verification - Recovery - Observability - Human control That entire surrounding system is the harness. Models will continue to improve. They will reason better, understand larger systems, and use tools more accurately. But intelligence without disciplined execution is only potential. The coding-agent winners will be the companies that convert that potential into software changes developers are willing to review, trust, and ship. > The model thinks. > The harness remembers, acts, checks, recovers, and earns trust. That is why the next AI coding-agent war will not be fought only between models. It will be fought between the systems built around them. --- ## Frequently Asked Questions ### Is an agent harness just a system prompt? No. A system prompt is one input to the model. A harness also manages context, tools, execution, permissions, memory, sessions, retries, verification, observability, and user interaction. Prompts guide behavior. Harnesses operationalize and enforce behavior. ### Is MCP an agent harness? No. MCP is a protocol for exposing tools, resources, and related capabilities. It can be an important part of a harness, but it does not automatically provide orchestration, sandboxing, context management, recovery, verification, or observability. ### Can one harness support multiple models? Yes. A provider-independent harness can normalize common concerns such as streaming, tool calls, errors, cancellation, context limits, and usage reporting while still preserving provider-specific strengths. ### Does a better harness make the model irrelevant? No. The model still determines major capabilities such as reasoning, code understanding, instruction following, and tool selection. The harness determines how reliably and safely those capabilities become real work. ### What is the most important harness capability? There is no single universal answer, but a strong verification and recovery loop is among the most important. Context and tools help an agent act. Verification tells it whether the action actually worked. Recovery determines whether failure becomes progress or repetition. ### Should every coding agent use many subagents? No. Subagents are valuable when work is genuinely separable. For small or tightly coupled changes, delegation can increase cost and coordination risk without improving quality. ### Should companies build their own coding-agent harness? Most teams should reuse an existing runtime for common orchestration and invest in the parts specific to their business: - Domain context - Tools - Security policy - Verification - Observability - Workflow integration - Internal evaluations Custom orchestration is justified when it solves a concrete requirement that existing runtimes cannot satisfy. ### How should a company choose a coding agent? Evaluate the complete model–harness pair on real internal tasks. Measure correctness, regression rate, verification, security, recovery, reviewability, tokens per solved task, cost per accepted change, and human intervention. --- ## Key Takeaways 1. A coding agent is not simply a model with shell access. 2. The harness controls context, tools, execution, permissions, verification, memory, and recovery. 3. Recent studies show that changing the harness can materially change results and efficiency while keeping the model fixed. 4. Model quality remains essential, but model-only comparisons are incomplete. 5. Verification, sandboxing, and observability distinguish a production system from a demo. 6. The durable product moat may live in domain-specific context, tools, policies, evaluations, and reliability improvements. 7. The future belongs to strong model–harness combinations, not models operating in isolation. --- ## Suggested Social Post The AI coding-agent war is no longer just GPT vs Claude vs Gemini. The model writes code—but the harness decides: - What context the model sees - Which tools it can use - What it is allowed to change - How failures are recovered - Whether the result is actually verified The model sets the capability ceiling. The harness determines how reliably the system reaches it. Read the full deep dive: The AI Coding Agent War Is No Longer About Models — It’s About the Harness --- ## References > Product capabilities and linked documentation were checked on August 29, 2026. Product behavior may change after publication. 1. OpenAI — Codex as a platform: build on the open agent harness [developers.openai.com/blog/codex-as-a-platform][openai-codex-platform] 2. OpenAI — Sandbox Agents [developers.openai.com/api/docs/guides/agents/sandboxes][openai-sandbox-agents] 3. OpenAI — Shell + Skills + Compaction [developers.openai.com/blog/skills-shell-tips][openai-compaction] 4. OpenAI — Build iterative repair loops with Codex [developers.openai.com/cookbook/examples/codex/build_iterative_repair_loops_with_codex][openai-repair-loops] 5. OpenAI — Build an Agent Improvement Loop with Traces and Evals [developers.openai.com/cookbook/examples/agents_sdk/agent_improvement_loop][openai-agent-improvement] 6. OpenAI — Codex best practices: sandboxing and approvals [developers.openai.com/codex/learn/best-practices][codex-sandbox-best-practices] 7. Anthropic — Claude Agent SDK overview [code.claude.com/docs/en/agent-sdk/overview][claude-agent-sdk] 8. Google — Gemini CLI policy engine [geminicli.com/docs/reference/policy-engine][gemini-policy-engine] 9. Google — Gemini CLI sandboxing [geminicli.com/docs/cli/sandbox][gemini-sandboxing] 10. Google — Gemini CLI checkpointing [geminicli.com/docs/cli/checkpointing][gemini-checkpointing] 11. Google — Gemini CLI subagents [geminicli.com/docs/core/subagents][gemini-subagents] 12. DeepSeek — DeepSeek Harness developer preview [deepseek.com/harness/en][deepseek-harness] 13. Model Context Protocol — Tools specification, 2026-07-28 [modelcontextprotocol.io/specification/2026-07-28/server/tools][mcp-tools] 14. Lewis, S. — Same Model, Different Harness: Different Coding-Agent Results [arXiv:2608.26218][same-model-different-harness] 15. Vats, N. and Golev, O. — The Scaffold Effect in Coding Agents [arXiv:2607.22585][scaffold-effect] 16. Yang, J. et al. — SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering [arXiv:2405.15793][swe-agent] 17. Nie, J. et al. — TTHE: Test-Time Harness Evolution [arXiv:2607.08124][tthe] --- [openai-codex-platform]: https://developers.openai.com/blog/codex-as-a-platform [openai-sandbox-agents]: https://developers.openai.com/api/docs/guides/agents/sandboxes [openai-compaction]: https://developers.openai.com/blog/skills-shell-tips [openai-repair-loops]: https://developers.openai.com/cookbook/examples/codex/build_iterative_repair_loops_with_codex [openai-agent-improvement]: https://developers.openai.com/cookbook/examples/agents_sdk/agent_improvement_loop [codex-sandbox-best-practices]: https://developers.openai.com/codex/learn/best-practices [claude-agent-sdk]: https://code.claude.com/docs/en/agent-sdk/overview [gemini-policy-engine]: https://geminicli.com/docs/reference/policy-engine/ [gemini-sandboxing]: https://geminicli.com/docs/cli/sandbox/ [gemini-checkpointing]: https://geminicli.com/docs/cli/checkpointing/ [gemini-subagents]: https://geminicli.com/docs/core/subagents/ [deepseek-harness]: https://deepseek.com/harness/en/ [mcp-tools]: https://modelcontextprotocol.io/specification/2026-07-28/server/tools [same-model-different-harness]: https://arxiv.org/abs/2608.26218 [scaffold-effect]: https://arxiv.org/abs/2607.22585 [swe-agent]: https://arxiv.org/abs/2405.15793 [tthe]: https://arxiv.org/abs/2607.08124