Same Question, Twice: a Live Before/After of AI Research

An hour after we published research-discipline, we ran the experiment you’d actually want to see: not a curated benchmark, but a single live question, asked twice, with the answers checked in public. This article is the full protocol — what we asked, what each run did, and every verification step — so you can rerun it yourself.

The setup

The question. “What are the most popular Claude Code skills right now? Name the top 5 skill repositories on GitHub by stars, with current counts. How does Anthropic’s official repo compare to community ones?” — a question with an objectively checkable answer that changes weekly, which is exactly where AI research quietly rots.

The two runs. Identical Claude Sonnet agents, identical prompt, identical tools (web search, web fetch, shell). One difference: the second agent first read SKILL.md from a copy of research-discipline installed the way any user would install it —

git clone https://github.com/Skillproofdev/research-discipline ~/.claude/skills/research-discipline

That’s the whole interventon: ~1,500 tokens of rules the model reads before working.

The referee. After both answers came back, we pulled the star count of every repository either answer mentioned — plus the candidates neither mentioned — straight from the GitHub API, the only authority on its own numbers. No LLM judging in the loop for this one; just gh api repos/<owner>/<repo>.

What came back

The baseline answer looks great. Five repositories, plausible descriptions, star counts accurate to the digit, confident synthesis at the end. If we hadn’t checked, we’d have shipped it.

It’s wrong in the way that matters. The real top-5, verified against the live API, includes three repositories the baseline never surfaced — at 228k, 190k and 164k stars. They’re not obscure: each is bigger than three of the five repos the baseline did name. Its closing claim, “Anthropic’s official repo is second only to Superpowers,” is false — the official repo is fifth. Nothing the baseline wrote was fabricated; every number it gave was real. It just answered a different question: “which repos do blog posts talk about?” — because its six tool calls were web searches, and search results are a popularity contest for coverage, not for stars.

The skill run answered the question that was asked. Its first rule — evidence is live; memory and first-page search results are leads, not proof — pushed it from googling to enumerating: it queried the GitHub API directly and swept three GitHub topic listings (agent-skills, claude-skills, claude-code-skills) to make sure nothing large was hiding outside the blogosphere. All five of its counts matched the live API when we re-checked. It also did two things nobody asked for but a careful researcher would: it separated actual skill repositories from curated link-lists (the baseline mixed them), and it attached two uncertainty flags — one noting that several top repos are only months old with unusually fast star growth ([unverified] whether that’s organic), one admitting a marketplace claim traced to a single blog ([single-source]). Both flags survived scrutiny.

The method, spelled out

Why does a text file change behavior this much? Because the failure it targets isn’t ignorance — the baseline model knows GitHub has an API — it’s the default habit of answering from whatever surfaces first. The skill converts good practice from “something the model might do” into “something the model must do before it’s allowed to assert”:

1. Rule 1 (live evidence) forced the date-stamp and the API calls instead of trusting search snippets.
2. Rule 3 (independent sources) is why the topic sweeps happened — one search path isn’t enough for a load-bearing “top 5” claim.
3. Rule 5 (unverified means saying unverified) produced the two flags instead of silent confidence.
4. Rule 7 (attack your own conclusion) is the reason it went looking for repos that would break its own ranking — and found them.

The honest costs, same as our controlled benchmark found: the disciplined run used 11 tool calls against 6, and 64,445 tokens against 49,988 — +29% on this task. Verification is not free. It’s just much cheaper than being confidently wrong in a document someone acts on.

What this doesn’t prove

One question is an anecdote, not a benchmark — the controlled six-question version with claim-level verification (13.0% → 5.4% wrong claims) is the evidence; this is the demonstration. The skill also isn’t a completeness oracle: in the controlled run it once priced one vendor tier and missed a cheaper one. And the star counts above were true on July 10, 2026 and will drift — which is, fittingly, exactly the kind of caveat the skill forces you to write.

FAQ

Did you cherry-pick the question? It was the first question we ran after publishing the skill, chosen because our own catalog data could double-check it. The controlled benchmark with six pre-registered questions is the systematic version, and it agrees.

Would a smarter model make the skill unnecessary? The baseline here was a current model with full web access. The gap wasn’t capability — it was defaults. Rules change defaults.

Why did both runs get star counts right but only one get the ranking right? Because accuracy and completeness fail independently. Every number the baseline quoted was real; the error lived in what it never looked for. That’s the most dangerous failure mode in AI research: nothing in the answer looks wrong.

What about the +29% tokens? Pair it with token-discipline, which cuts session waste by about 20% on multi-step work. Discipline on what you read, discipline on what you claim — the two are designed to run together.