AutoResearch & Claude Code: Self-Improving Systems | Guide

AutoResearch Guide: Build Self-Improving Business Systems with AI (2026)

TL;DR

• AutoResearch is an AI-driven experimentation pipeline that repeatedly improves a target metric without human supervision.
• The same loop Karpathy used for nanoGPT can optimize cold email, ads, landing pages, and ecommerce performance.
• A working system needs just three elements: experiment endpoint, autonomous agent, and a clear objective metric.
• Claude Code plus GitHub Actions can run 24/7 self-improving business workflows with no manual coding.
• Fast feedback loops and measurable metrics matter more than perfect decisions, enabling compounding gains over time.

Marketing teams, solo founders, and growth operators are all wrestling with the same problem: too many levers to test, not enough hours in the day. Meanwhile, top AI labs quietly run thousands of experiments overnight to squeeze out incremental performance gains.

AutoResearch, an open-source project released by Andrej Karpathy in 2026, shows how that same philosophy can be bottled into an automated loop. When I dug into the repo and the example implementations, what stood out was how easily this pattern transfers from model training to very practical business optimization.

This post explains how AutoResearch works, how Karpathy wired it to train nanoGPT, and how the same structure can optimize cold emails, landing pages, ads, ecommerce, and more. It also walks through how to reproduce the setup using Claude Code and GitHub Actions so a self-improving system can run continuously in the background.

What AutoResearch Is and Why It Matters

AutoResearch is an autonomous experimentation pipeline where an AI agent repeatedly designs, runs, and evaluates experiments to improve a target metric. The project was released as an open-source repository by Andrej Karpathy, and it drew immediate attention from both AI researchers and developers who recognized the broader implications.

The core question Karpathy started from: “What if my model could train my model?” That led to a pipeline where an AI agent tweaks training code, launches short runs, measures performance, and keeps or discards changes — on its own.

Karpathy’s own description can be paraphrased as: “Give an AI agent a real LLM training environment, let it autonomously run experiments overnight, and in the morning you wake up to logs and a better model.”

This isn’t simple batch processing. It’s a feedback loop where the system observes results, updates itself, and repeats — self-improving in an evolutionary fashion. When I examined the structure, the main surprise was how minimal the ingredients actually are.

The Three Core Components of AutoResearch

AutoResearch is a three-part system that can, in principle, be applied to any domain with measurable outcomes.

1. Experiment Endpoint
   An experiment endpoint is a function or API that accepts a hypothesis as input and runs one experiment.
2. In model training, that hypothesis could be a new learning rate or architecture tweak.

3. In business, it might be different email copy, a new landing page layout, or a revised price point.

4. Autonomous AI Agent
   The AI agent calls the endpoint via API, decides what to change, and interprets results.

5. It writes or edits code, configs, or content.

6. It launches experiments, waits for completion, and records metrics.

7. Objective Metric
   The objective metric is a numeric signal used to judge whether a change is an improvement.

8. In Karpathy’s original implementation, the metric was validation loss on a held-out dataset.
9. Lower validation loss means a better model, so the agent simply tries to minimize that value.

Once those three elements exist, the same AutoResearch logic can be ported into almost any other domain. This mirrors how optimization loops are described in reinforcement learning literature, and aligns with approaches used by major labs for large-scale experimentation.

For additional background on objective metrics in ML, see:

• https://developers.google.com/machine-learning/crash-course/classification/accuracy
• https://pytorch.org/tutorials/beginner/basics/optimization_tutorial.html

How Karpathy’s AutoResearch Loop Works with nanoGPT

The nanoGPT loop is an AutoResearch instance that automatically tunes a small GPT model by iterating over short training runs. It demonstrates how the classic scientific method can be fully automated: hypothesize, experiment, measure, decide.

Each loop iteration begins with a hypothesis — say, “adjust the learning rate and context length to reduce validation perplexity.” The agent modifies hyperparameters, runs a short training cycle, measures the result, then decides whether to keep or discard the change.

In the nanoGPT example, the system starts from a baseline validation perplexity and, through repeated adjustments, drives that number steadily downward — which indicates a genuinely smarter model.

The power is in loop speed. With a 5-minute cycle, AutoResearch can run 12 experiments per hour. Over a full day, that far exceeds what any human researcher could coordinate manually — even one with sharper intuition about each individual change.

When I replicated a similar loop pattern for a toy model, the main insight was that even imperfect decisions compound rapidly when the system iterates dozens of times per day. You can see it clearly in the metric curves after a few hundred runs.

Prompt Structure and Accessibility

Karpathy’s repo includes a “prompt end” structure that defines how the orchestrator agent should behave. It contains:

• Context about the codebase and training setup.
• Instructions on how to design and evaluate experiments.
• Guidelines for logging, rollback, and safety.

He explicitly notes that his prompt is “probably terrible and easy to improve” — which actually makes the whole thing more approachable, not less.

A fully functional self-improvement loop doesn’t require a perfect prompt. It just needs a coherent environment and a clear metric.

That means even non-experts, working with a strong model and a well-defined metric, can get real value without deep prompt-engineering knowledge.

For readers interested in nanoGPT itself, the original project is documented here:

• https://github.com/karpathy/nanoGPT

Applying AutoResearch to Business: Cold Email Optimization

Business AutoResearch is the application of the AutoResearch pattern to optimize business metrics such as revenue, conversion, or reply rate instead of model accuracy. A concrete example from the source is cold email optimization, where the core metric is reply rate to outbound sales messages.

Cold email is outreach to prospects who have never been contacted before. Historically, optimizing copy required manual A/B tests, repeated spreadsheet exports, and significant human attention. AutoResearch replaces that with an automated loop that continuously tries new variants and learns from results.

In my own experimentation with cold email systems, reply rate has consistently been the most honest signal of copy quality — beating softer metrics like open rate when it comes to actual pipeline impact. That matches closely with how this AutoResearch example is structured.

The Email Optimizer Folder Structure

The Email Optimizer implementation shows how the loop is wired in practice. Internally it consists of:

• An orchestrator agent that creates new cold email campaigns.
• Separate baseline and challenger email variants.
• An integration with a sending platform like Instantly to collect replies via API.
• A loop that runs every 4 hours.

At each cycle:

1. The system sends both baseline and challenger emails.
2. It automatically collects reply counts from the Instantly API.
3. It promotes the challenger to the new baseline if it wins.
4. It logs what it “learned” in a Markdown file.

In an initial test sequence, reply rate improved from 1.5% in test #1 to above 2% by test #12 — a measurable gain, even in a short window.

Accumulated Learning and Knowledge Base

A key design element is the learning accumulation mechanism. At the end of each experiment, the system writes insights — things like “what seems to increase reply rate” — into a Resources MD file.

This file becomes a knowledge base that future agents consult when proposing new changes, so the system builds on prior experiments instead of starting from scratch each time.

Even after just a few days of operation, the effects are visible. Over a full year at a 4-hour cadence, the system would run roughly 2,190 experiments automatically. Shorten the loop to 5 minutes and that becomes around 288 experiments per day.

This resembles how high-frequency testing is used in digital marketing, except here the loop is fully autonomous and tied to a persistent, machine-readable memory of what’s been tried before.

How the System Generates Challenger Emails

When generating a challenger, the agent starts from a concrete critique of the baseline. For example, it might reason that:

• The baseline email is too long.
• Key benefits are buried or unclear.
• There’s no specific time request for a meeting.

From that, it forms a hypothesis like:

“Rewrite this to under 75 words, lead with relevance, highlight risk reversal, and close with a specific time request.”

The agent then creates a new challenger copy embodying those changes. Most early challengers underperform the baseline — that’s expected during exploration-heavy phases. But over many cycles, the system tends to surface versions with clearly higher reply rates than where it started.

For further context on A/B testing methodology similar to this setup, see:

• https://support.google.com/analytics/answer/9322688
• https://www.optimizely.com/optimization-glossary/ab-testing/

Business Domains Where AutoResearch Works Well

AutoResearch application domains are any business areas with a trackable objective metric and API-level control over inputs. When both conditions are met, the loop can be deployed.

There are several concrete domains where this pattern is especially promising.

Cold Email via Instantly

Cold email is a natural first target because reply rate is:

• Numeric.
• Directly tied to revenue.
• Easy to measure via sending platform APIs.

Using the Instantly API, the system reads current reply rates, updates copy automatically, and promotes winners while archiving losers.

Landing Pages via Website Builders

Landing page optimization becomes possible when AI gets access to site builders like Wix, WordPress, or Webflow.

The loop can use conversion rate as its objective metric, then continuously adjust layout, headlines, CTAs, and sections. It deploys challengers, collects outcome data, and keeps improving.

Ad Creatives on Major Ad Platforms

Platforms like Facebook Ads and Google Ads already expose extensive APIs, which makes ad creative optimization a natural fit.

The system can define a specific conversion metric — purchases, signups — then automatically A/B test new creatives, kill underperformers, and scale winners.

One honest caveat worth noting: built-in optimization on these platforms may not be weaker than what a custom AutoResearch loop delivers. Advanced models like Claude Opus 4.6 or GPT-5 could theoretically do more, but it’s not guaranteed. The value is in the additional experimentation layer on top of what the platform already does.

Customer Service Scripts

Customer service script optimization uses CSAT as the metric. The loop iterates on response templates used by human agents or AI bots, measures satisfaction after each interaction, and gradually refines language and structure over time.

Ecommerce Product Descriptions

Platforms like Amazon FBA provide sales and conversion data, making product description optimization trackable. With tools like Chrome DevTools MCP, an agent can edit product pages directly. The loop then tests variations in copy and structure, tracking subsequent revenue.

YouTube Titles and Other Channels

YouTube title optimization works by integrating the YouTube Data API v3 and using click-through rate as the metric. The same pattern extends to newsletter subject lines, pricing pages, and any other channel with measurable outcomes and controllable inputs.

The unifying principle: when “trackable metric + API access” exists, the AutoResearch loop can work there.

This brings the same overnight-experimentation philosophy that major ML labs use to tune models into everyday business workflows.

For documentation on relevant APIs, see:

• https://developers.google.com/youtube/v3
• https://developer.wordpress.org/rest-api/

Building an AutoResearch System with Claude Code

Claude Code AutoResearch setup is a three-stage process combining repository cloning, test definition, and GitHub Actions automation. In practice, this lets you stand up a full self-improving system without manually writing Python.

From my own use of Claude’s VS Code extension, combining repository context with a clear natural-language spec is often enough to scaffold robust, multi-file systems in one shot. That’s exactly the pattern being used here.

Step 1: Clone the AutoResearch Repo into a Claude-Accessible Environment

Clone Karpathy’s AutoResearch GitHub repository into a local or cloud workspace where Claude Code runs.

Common options include:

• The official Anthropic VS Code extension.
• Tools like Anthograde Vitura.
• Any IDE that supports Claude Code as an assistant.

The setup flow:

1. Create a new folder (e.g., karpathy-autoresearch-demo).
2. Open Claude Code in that folder.
3. Ask it, in plain language, to clone the AutoResearch repository into the current working directory.

Claude issues the necessary GitHub API calls and completes the clone automatically.

Step 2: Define What You Want to Optimize via a Test File

Create a test file that describes:

• Goal — what to improve.
• Metric — how success is measured.
• Test Method — the platform and levers the system can touch.

In the example from the video, a voice transcription tool (Whisper Flow) was used to feed Claude a specification like:

“Using the AutoResearch folder context, build a similar idea for cold email. Use reply rate as the metric, Instantly as the platform, modify the cold email copy, and configure GitHub Actions so it runs automatically every hour.”

Based on that, Claude Code generates:

• An orchestrator agent.
• An Instantly client for API interaction.
• Supporting utility scripts.
• Config files and scheduling logic.

This is where Claude acts less like a coding autocomplete and more like a system architect, wiring all required components into a working pipeline.

The Internal System Structure

The orchestrator agent works like a conductor coordinating sub-agents and tools. Its responsibilities:

• Direct a copy-writing sub-agent to draft challenger emails.
• Manage Instantly API calls to send campaigns and gather reply data.
• Orchestrate experiment documentation and log updates.
• Optionally build and maintain a database of experiment results.

Each run of the loop splits into three phases:

1. Harvest — Collect results from previous experiments.
2. Generate — Create new challenger copy based on updated learnings.
3. Deploy — Launch campaigns and pull leads from the lead pool.

Step 3: Automate the Loop with GitHub Actions

Move execution to the cloud via GitHub Actions.

A workflow file is created with a cron schedule — every hour, or whatever interval makes sense. For security, a GitHub Secrets setup stores the Instantly API key and the Anthropic API key.

Monitoring matters for any system running unattended. A Slack webhook handles notifications whenever a new challenger is generated or a challenger becomes the new baseline.

With that in place, the AutoResearch loop runs 24/7 in the background, improving the target metric without ongoing manual intervention.

For more details on GitHub Actions, see:

• https://docs.github.com/en/actions

Conditions Where AutoResearch Works Best (and Pitfalls to Avoid)

AutoResearch applicability conditions are defined by three requirements: fast feedback loops, clear objective metrics, and API access to modify inputs. Missing any one of them degrades the value of automation significantly.

The nanoGPT example works so well because the training loop is short. At a 5-minute cycle, 12 experiments run per hour, delivering dense feedback within a single day.

The faster the feedback loop, the more experiments the system completes — and the stronger the compounding effect over time.

Clear and Objective Metrics

Metric clarity is especially critical. Good AutoResearch metrics include:

• Reply rate.
• Click-through rate (CTR).
• Conversion rate (CVR).
• Validation loss or perplexity in ML.

These are numerically defined and unambiguous. Concepts like “brand emotion” or “customer happiness” don’t work directly because they’re subjective. In those cases, the system needs proxy metrics that approximate the underlying concept — things like CSAT score, Net Promoter Score, or repeat purchase frequency.

Without a solid numeric proxy, the loop has no stable target and optimization becomes unreliable.

Handling Limited or No API Access

When APIs aren’t available, automation can still be built via tools like Chrome DevTools MCP or Playwright flows that simulate user interactions. But this introduces higher implementation complexity and more fragility when UIs change. Native APIs are strongly preferred whenever possible.

Managing Growing Documentation and Token Costs

After 500–1,000 runs, the resource documents holding past learnings can grow very large. That means higher token consumption per agent run, slower responses, and increased costs.

A practical fix: periodically summarize and consolidate these documents. Keep a concise “core learnings” file and archive the detailed logs separately for audit purposes.

Human vs. AI Optimization Efficiency

The creator makes an honest comparison here:

“If I were in the loop, I’d probably make better decisions than the AI. But realistically I can only run a few experiments per day. I have to sleep and do other work. The AI agent doesn’t. At one loop per hour, that’s 24 experiments per day.”

Over time, frequency of experimentation beats per-experiment decision quality. Even if each AI decision is slightly worse than a human’s, running 20–50x as many experiments produces a much larger net gain.

Why AutoResearch Matters for Business and AI Research

Democratization of AutoResearch is the process by which a continuous self-improvement methodology — once limited to elite AI labs — becomes available to solo operators and small teams. Major research labs like OpenAI, Google DeepMind, and Anthropic already run hundreds or thousands of automated experiments overnight to refine models.

AutoResearch brings that same experimental philosophy to general business optimization. For marketers, sales teams, and ecommerce operators, the main benefit is friction removal.

Tasks that used to require copying leads, manually editing copy, and hand-analyzing results are now expressed as an API call chain running around the clock.

Since the agent has access to all historical runs, it effectively remembers the full optimization history. Even if each loop spawns a fresh agent instance, the underlying knowledge base lets the system get more useful over time.

The Knowledge Compounding Effect

The long-term value of AutoResearch comes from what can be called the knowledge compounding effect.

• Within a few days, reply rates and other metrics show visible improvement.
• Over months or years, the gap versus the original baseline can become enormous.

This is especially pronounced in always-on channels like cold email, paid ads, and landing pages. Karpathy’s validation loss graphs demonstrate how additional iterations steepen the improvement curve — and the same principle applies when the metric is revenue or conversion instead of loss.

Surprisingly Low Technical Barrier

One of the most striking things about this system is how low the bar has become. Once Claude Code has the AutoResearch repository as context, you only need to state the goal, define the metric, and specify the platform. Voice or text both work. Claude then generates the full system architecture — no hand-written Python required.

In this configuration, Claude Code acts not just as a coding assistant but as a co-designer and implementer of an entire self-improving pipeline.

As these tools become more tightly integrated into everyday workflows, the distance between an idea (“optimize our cold email reply rate”) and a running, 24/7 experimentation system will keep shrinking.

Frequently Asked Questions

Q: What exactly is AutoResearch in simple terms?

A: AutoResearch is an automated loop where an AI agent repeatedly designs, runs, and evaluates experiments to improve a specific numeric metric. It was originally demonstrated by Andrej Karpathy for model training but applies equally well to business metrics like reply rate or conversion rate.

Q: What are the minimum requirements to use AutoResearch in a business context?

A: Two conditions are essential: a trackable objective metric (such as reply rate, CTR, or sales) and API access to modify inputs (like email copy, landing page elements, or ad creatives). If either is missing, the loop can’t effectively test and learn.

Q: How does AutoResearch differ from traditional A/B testing tools?

A: Traditional A/B testing tools help humans design and analyze experiments manually. AutoResearch turns the entire process — hypothesis creation, variant generation, deployment, and evaluation — into a closed loop run by an AI agent, operating continuously and autonomously.

Q: Do I need to know how to code to set up an AutoResearch system?

A: In the described setup, code is generated by Claude Code using natural-language instructions and the AutoResearch repository as context. The user defines goals, metrics, and platforms; Claude writes orchestrators, API clients, and configuration. Manual coding isn’t strictly required.

Q: What kinds of metrics are not suitable for AutoResearch?

A: Vague, subjective metrics like “brand warmth” or “customer happiness” don’t work directly because they lack a clear numeric definition. They need to be translated into proxy metrics like CSAT, NPS, or repeat visit rate before they’re usable in an automated loop.

Conclusion

AutoResearch makes continuous, high-frequency experimentation available to teams that aren’t AI labs. With a clear metric, API access, and an agent like Claude Code, small teams can run thousands of experiments per year on their own funnels, creatives, and customer experiences.

The real levers aren’t perfect prompts or flawless decisions — they’re fast feedback loops and rigorous, numeric metrics. Given time, those loops generate a compounding knowledge effect that can produce meaningful gains in reply rates, conversions, and revenue.

As Claude Code and GitHub Actions become more deeply embedded in everyday workflows, self-improving systems will likely become standard operating infrastructure. The teams that define strong metrics now and let agents optimize against them will be the ones compounding their advantages day after day.