Why The Harness Matters More Than The Model | YC Paper Club
Y CombinatorY CombinatorKey takeaways
Harness design can substantially change what the same model weights achieve: the cited ARC-AGI result rises from about 30% to 95% with a better harness, though scores depend on the evaluation setup and the reported results are not a universal guarantee.
Effective harnesses manage context and persistent state, expose tools, computation, and agent coordination, and can improve themselves; broader capabilities matter when they enable useful work over long horizons. OpenJarvis targets private, lower-cost on-device AI by optimizing a local stack, while YC’s QM centralizes agent sessions and treats sandboxes and models as selectable resources, with human review still important for changes and permissions.
Video Flow
- 0100:00
A harness can change model performance
The same model weights can produce sharply different ARC-AGI scores under different harnesses: the opening comparison gives about 30% for a baseline and 95% with a stronger harness, with NVIDIA’s AVO reported at 100%. Harness progress is framed as moving from static wrappers toward systems that adapt their prompts or code to accumulated experience.
- 0204:12
An auto-researcher grows from a tracking interface
A project that began as a way to view an auto-research run grew into a multi-agent research workflow: agents scope ideas, run experiments, get feedback, and prepare ablations and papers. The history of harnesses moves from basic generation and in-context examples through tool use, editable memory, reusable skills, reflection, subagents, and recursive language models, then toward systems that modify prompts or harness code.
- 0318:35
Prime Agent treats context as managed compute
Prime Agent places a persistent, programmable layer between an LLM and its environment, with root sessions coordinating subagents and messaging between them. Its context-management model treats weights, active context, live computation, and files as increasingly flexible stores; the design favors broad agent-controlled capabilities, while long-horizon ARC-AGI, emulator, GPU-kernel, research, and game results vary by task and evaluation.
- 0437:27
OpenJarvis optimizes local personal AI
OpenJarvis organizes a personal AI stack around five primitives: interface, agent logic, model, inference engine, and tools or memory, alongside learning and optimization. Cloud models can optimize configurations for local models before deployment; the reported tests found lower cost and latency and, for some workloads, competitive quality, while some tasks remain beyond local models.
- 0545:49
QM centralizes company agents and their resources
QM gives YC employees customizable assistants in Slack or a web interface, with personal contexts, sandboxes, files, and scheduled jobs for tasks such as email triage, document editing, and database work. Its evolution from a general agent through coding agents and a fleet of VM-based assistants led to centralized conversations and sandboxes that agents can select, while human review remains important for database writes and permission-sensitive context.
Tools & Resources
1 references
The forked auto-research project provides the starting point for a research-agent workflow.
- 01An auto-researcher grows from a tracking interface04:14
The Minecraft work illustrates reusable skills that can be learned and reused for tasks.
- 01An auto-researcher grows from a tracking interface10:22
Code output is presented as a way to create tools or skills dynamically.
- 01An auto-researcher grows from a tracking interface10:53
ReAct appears in the overview of agent approaches that lead into reflection and self-improvement.
- 01An auto-researcher grows from a tracking interface11:14
Self-Refine is named among approaches to iteratively improving agent outputs.
- 01An auto-researcher grows from a tracking interface11:14
Reflection is named among approaches to improving agent outputs with feedback.
- 01An auto-researcher grows from a tracking interface11:14
DSPy is used as an example of optimizing system prompts against training examples.
Darwin Machines is described as evolving the harness code as well as its prompts.
- 01An auto-researcher grows from a tracking interface15:10
Continual Harness is cited for refining harness state and using history for continued improvement.
- 01An auto-researcher grows from a tracking interface16:30
The harness provides persistent agent sessions, programmable tools and state, and coordination between agents.
- 01Prime Agent treats context as managed compute18:35
Claude Code is named as a coding-agent interface for comparison with the Prime Agent experience.
Pi is named as another agent interface for comparison with Prime Agent.
- 01Prime Agent treats context as managed compute19:55
The recursive language model principle underlies programmatic tools, memory, and subagent coordination in Prime Agent.
- 01Prime Agent treats context as managed compute20:30
A community leaderboard prompt was reused for an ARC-AGI run before sandboxing issues were corrected.
- 01Prime Agent treats context as managed compute30:53
The Verifiers package is named as a way to run the reported evaluations.
- 01Prime Agent treats context as managed compute37:11
The project defines primitives for building and optimizing an on-device personal AI stack.
- 01OpenJarvis optimizes local personal AI39:40
OpenClaw is cited as a cloud-dependent personal-agent comparison.
Hermes Agent is cited as a cloud-dependent personal-agent comparison.
A local model is cited as performing roughly like an earlier frontier model in the speaker's comparison.
- 01OpenJarvis optimizes local personal AI38:48
Claude Opus is used as a reference point for the reported local-model performance gap.
- 01OpenJarvis optimizes local personal AI38:50
Ollama is listed as an inference engine option for running local models.
- 01OpenJarvis optimizes local personal AI40:19
Llama.cpp is listed as an inference engine option for local inference.
- 01OpenJarvis optimizes local personal AI40:19
VLM is listed among inference-engine options for the local stack.
- 01OpenJarvis optimizes local personal AI40:19
SGLang is listed as an inference engine option for local models.
- 01OpenJarvis optimizes local personal AI40:19
MCP is named as the standard protocol through which local tools and memory can be accessed.
- 01OpenJarvis optimizes local personal AI40:34
JAPA is listed as a prompt-based learning approach for improving the local stack.
- 01OpenJarvis optimizes local personal AI40:42
GPO is listed among weight-based learning approaches.
- 01OpenJarvis optimizes local personal AI40:46
Supervised fine-tuning is listed among weight-based learning approaches.
- 01OpenJarvis optimizes local personal AI40:46
LoRA is listed among weight-based learning approaches.
- 01OpenJarvis optimizes local personal AI40:46
ChatGPT is named as a cloud model used to optimize local configurations.
- 01OpenJarvis optimizes local personal AI43:04
Gemini is named as an alternative cloud model for optimizing local configurations.
- 01OpenJarvis optimizes local personal AI44:13
Kimi is named as an alternative model family for optimizing local configurations.
- 01OpenJarvis optimizes local personal AI44:22
GLM is named as an alternative model family for optimizing local configurations.
- 01OpenJarvis optimizes local personal AI44:22
YC's agent harness provides customizable assistants, shared work contexts, and access to company resources.
- 01QM centralizes company agents and their resources45:58
Codex is cited as a coding agent used in the earlier YC workflow.
- 01QM centralizes company agents and their resources48:20
PostgreSQL stores centralized agent conversations and context for QM.
- 01QM centralizes company agents and their resources52:42