One File, One Metric, Five Minutes
When the entire AI industry is racing to add more — more tools, more context, more autonomy — Andrej Karpathy did the opposite.
He gave his AI agent:
- One file to modify (
train.py) - One immutable scoring function (
prepare.py— locked, unreadable by the AI) - Five minutes per experiment
Result: 100 experiments in a single night. A PhD student would need weeks.
This is AutoResearch. And its most important lesson isn't technical — it's about what constraints actually do to a system's ability to improve.
The Three-File System
AutoResearch runs on three components.
prepare.py defines what "good" means: an immutable scoring function the AI cannot read or modify. It only sees the output — a number.
train.py is the only file the AI can change — algorithm parameters, preprocessing settings, loss function weights. Each change is a hypothesis.
program.md records the AI's reasoning in natural language: "I observed X, so I hypothesize Y, and next I'll test Z." It's the research journal, not code.
The loop: propose hypothesis → modify train.py → execute → score. Higher score: Git commits and advances. Lower score: auto-rollback, try again.
The mainstream AI stack keeps adding tools and autonomy. AutoResearch removes them — and produces faster, more directed improvement.
The design prevents "gaming." An immutable scoring standard means the AI can't optimize metric appearance instead of actual quality. Like GLP compliance in drug discovery, the evaluation protocol cannot flex to accommodate results.
The $15 Marketing Experiment
A marketer named Ole Hansmann applied AutoResearch logic to landing page copy.
His scoring checklist: Does the copy contain specific numbers? Avoid marketing clichés? Have one clear call-to-action? Stay under 150 words? Four binary questions.
Each round, an LLM generated variants evaluated against the checklist. Keep winners, discard losers, iterate.
4 iterations. 16 total variants. $15 in API costs.
Conversion rate: 56% → 92%.
The real insight isn't the result — it's the checklist design. Fewer than 3 criteria: too vague, the AI flails. More than 6: the AI "test-takes," optimizing checklist scores instead of actual quality. The productive range is 3–6 clear, binary questions.
The same closed-loop logic — immutable standard, variable parameters, systematic iteration — scales from ML research to marketing copy.
Why Constraints Work
Constraints do three things simultaneously.
They focus attention. Infinite options equal infinite search space — which equals finding nothing efficiently. "You can only change these parameters in this one file" transforms undirected wandering into directed search.
They prevent expansion. Systems without boundaries sprawl. Parkinson's Law applies to AI: work expands to fill available resources. Constraints force simplicity.
They shift creative mode. Unlimited resources trigger linear thinking (just add more). Limited resources trigger lateral thinking (find the clever path within the walls).
This isn't modern AI discovery. Poets constrained by rhyme scheme write humanity's most moving work. Screenwriters confined to three-act structure produce the most unexpected plot turns. The pattern is ancient: constraints collapse infinite possibility into meaningful decisions.
Apply It Without Building a System
You don't need an AutoResearch setup to use the principle.
Pick any repetitive task — social posts, meeting summaries, product descriptions, email subject lines. Ask: "Can I define 'good' with 3–6 specific, binary conditions?"
If yes, write those down. That's your constraint. Use it to evaluate AI output, or just your own drafts. The bounded evaluation makes improvement systematic rather than intuitive.
The question that should change: not "Is this idea worth experimenting on?" — but "Is this idea worth not experimenting on?"
The River Between Banks
A river without banks is a swamp.
The most effective systems aren't the freest ones. They're the most thoughtfully designed, precisely constrained ones.
Next time you want better output — from an AI or from yourself — try giving it less room to move.
One file. One metric. Five minutes.
Then watch what it creates between the walls.
References
- Andrej Karpathy, AutoResearch, GitHub, 2026
- Ole Hansmann, Landing Page Optimization Case Study, 2026
- Macarron et al., "Impact of high-throughput screening in biomedical research," Drug Discovery Today, 2011
Frequently Asked Questions
What is AutoResearch?
A framework by Andrej Karpathy where an AI agent operates under three strict constraints: one modifiable parameter file, one immutable scoring function, and a fixed time limit. Within these boundaries, it can run 100 experiments overnight.
Why do constraints improve creativity rather than limit it?
Constraints collapse infinite possibility into an actionable search space, forcing focused search instead of undirected wandering. Like a river between banks, bounded systems flow with direction.
Can this approach apply to non-technical work?
Yes. Define 3–6 binary criteria for 'good' in any repetitive task — ad copy, meeting summaries, product descriptions. That checklist becomes your constraint, making improvement systematic.
Found this useful?
Follow for new AI × biomedical research notes:
Or buy me a coffee to keep new content coming.
☕ Buy Me a Coffee