The first thing I ran in January 2026 seemed reasonable. I had 50 SaaS contracts from the CUAD dataset—510 commercial contracts annotated by legal experts with inter-annotator agreement at κ = 0.82—and I needed to extract payment terms. The prompt looked like this:

You are an expert legal assistant. Analyze this SaaS contract and extract 
all payment terms including billing frequency, payment methods, late fees, 
and renewal clauses. Be thorough and comprehensive.

It missed 40% of auto-renewal clauses. It hallucinated “net 30” payment terms that weren’t in the source documents. F1 against CUAD ground truth: 64%.

“Thorough.” “Comprehensive.” Both are instructions without definitions—and a language model trained to be helpful will interpret them as: include the payment terms you expect to see in a contract like this. Not extract what’s there. Expect. That difference is where hallucinations live.

Andrej Karpathy’s June 2025 framing of “context engineering” helped me stop thinking about this wrong: the LLM is the CPU, context is the RAM, and the job is to be the OS loading working memory. Keywords matter not because they’re secret instructions, but because they constrain the probability space for the next token. A vague modifier like “thorough” expands that space. A specific constraint like “quote the exact contract language or write NOT SPECIFIED” collapses it toward ground truth.

Here’s the same instruction three ways, with F1 results from my February trials:

The grounding constraint worked because it forced the model to produce evidence before conclusions. If the exact language isn’t there to quote, the model can’t fabricate a finding—the instruction structure makes hallucination harder than compliance.

The Repetition Architecture: Where It Works and Where It Doesn’t

In December 2025, Google Research published findings showing that repeating the input prompt—transforming a single query into a sandwich structure—improves accuracy on non-reasoning tasks by enabling bidirectional attention across the query. I ran 60 trials across three task types to see how far that held.

* Not significant: 95% CI [−2.1, +5.9 pp], p = 0.35. All other results p < 0.05.

The result that mattered most was the one that didn’t work. For math word problems—pure reasoning tasks—repetition added almost nothing. The 1.9 percentage-point gain was noise: t(19) = 0.97, p = 0.35. The confidence interval crossed zero. This wasn’t a small effect. It was no effect.

That’s the actual finding, and it matters more than the entity extraction win. Repetition architecture isn’t a universal improvement. It targets a specific mechanism—bidirectional attention on retrieval tasks—and reasoning tasks don’t benefit from it because the bottleneck is inference steps, not attention coverage. Apply this indiscriminately and you’ll pay 30-40% more in tokens for nothing.

What I Tested That Failed

The three failures are worth naming specifically because they’re plausible, they feel like they should work, and they cost real money and time to discover.

Failure 1: Semantic clustering

I grouped related terms—”cancellation,” “termination,” “expiration,” “non-renewal”—hoping that dense semantic neighborhoods would improve clause retrieval. This was intuitive and wrong. No measurable improvement over single precise terms (Δ = +0.8%, p = 0.62). The model doesn’t benefit from synonym piles. It already knows what “cancellation” means in a legal context. I was adding noise, not signal.

Failure 2: Dynamic few-shot RAG

I built a retrieval system to inject similar contract examples as few-shot context. Accuracy dropped 4%—from 70% baseline down to 66%—because retrieved examples introduced format inconsistencies that confused the model’s output structure. The few-shot examples were competing with my format constraints instead of reinforcing them.

Failure 3: “Expert” persona in Claude

This is the one I’m most confident will also bite you. “You are an expert legal analyst” improved GPT-4o’s accuracy by 6%. On Claude 3.5 Sonnet, the same instruction decreased accuracy by 3%. I’ve replicated this across multiple contract types. Claude’s constitutional AI training makes it resistant to role-playing—the persona instruction adds friction rather than focus. Test personas per model. Don’t assume what works on GPT transfers to Claude.

How Over-Constraint Decay Actually Looks

This failure deserves its own section because it’s the one nobody tells you about until you’ve already shipped a broken pipeline. Here’s the exact pair of prompts. Same task. Same model. Same contracts.

The over-constrained prompt didn’t crash. It returned plausible-looking output at every step. Confidence scores appeared. Jurisdiction notes appeared. But the actual classification accuracy dropped 12 points because the model split its attention across seven requirements and the core task suffered. This is the quietly dangerous version of failure: everything looks fine, and it isn’t.

The 3-Step Framework That Produced 89% F1

This isn’t a framework I invented beforehand. It’s a description of what I was actually doing after the failures forced me toward it.

TASK: Extract payment terms from the contract below.

Entities to extract:
- Billing frequency (monthly/annual/quarterly)
- Payment methods accepted
- Late fee percentage and grace period
- Auto-renewal clause (YES/NO + notice period required)
- Cancellation refund policy

Rules:
1. Quote exact contract language for each finding
2. If an entity is not present, write: NOT SPECIFIED
3. Do not infer terms not explicitly stated
4. Use only what appears in the contract below
5. Format each entity as: ENTITY: [exact quote or NOT SPECIFIED]

CONTRACT: {contract_text}

TASK REPEAT: Extract payment terms from the contract above.

REVIEW: Check the extraction below against the contract text.
- Identify any entity where the quoted text does not appear verbatim
- Flag any entity marked NOT SPECIFIED where the term may exist
- Mark uncertain extractions as: REQUIRES REVIEW

CONTRACT: {contract_text}
EXTRACTION: {pass_1_output}

Model-Specific Differences You Need to Know

My trials weren’t consistent across models, and the inconsistencies are more useful than the averages.

The Claude/GPT-4o persona divergence is the most practically significant finding in this table. If you’re running a mixed-model pipeline, persona instructions could be simultaneously helping one model and hurting another. That’s the kind of problem that doesn’t show up in aggregate accuracy metrics.

What the ROI Actually Looks Like—and When It Doesn’t Work

That number is real. It’s also the best-case version. The two-pass validation loop costs roughly 2× the token budget—about $20/day on GPT-4o versus $8 for the baseline single-pass approach. The ROI turns negative if you’re saving less than 4.8 minutes of manual review per day. At that margin, the math only works for analysts processing high volumes of similar contracts. If you’re doing one-off contract reviews, the ROI is much harder to justify.

What Comes Next (and What I Still Don’t Know)

Summarization, translation, and multi-modal tasks are entirely untested here. The results on reasoning tasks suggest repetition architecture may not transfer beyond retrieval workloads—but I haven’t run enough trials at sufficient sample sizes to know where the boundary actually sits. The model-specific differences, particularly the Claude persona finding, need replication across a wider task range before I’d recommend routing production decisions on them.

The two things I’m most confident about from this dataset: specificity outperforms vagueness every time, and the five-constraint ceiling isn’t a guideline—it’s a performance cliff. Everything else should be tested in your domain before you build on it.

The words “thorough” and “comprehensive” felt like instructions. They weren’t. Specificity is the instruction—everything else is just hoping the model reads your mind.