Common Prompt Engineering Mistakes Beginners Make

WordPress-ready format. Paste into Posts → Add New → Code Editor. Fill SEO metadata into Yoast or RankMath before publishing.

---

The Advice Is Often Wrong — and Now There’s Research Proving It

Start here: a lot of what gets repeated as prompt engineering gospel doesn’t hold up under controlled testing. In March 2025, researchers from Wharton — Lennart Meincke, Ethan Mollick, Lilach Mollick, and Dan Shapiro — published Prompting Science Report 1: Prompt Engineering Is Complicated and Contingent. They tested each benchmark question 100 times, not once, which is how most prompting advice gets validated. Their headline finding: it is hard to know in advance whether a particular prompting approach will help or harm the LLM’s ability to answer any particular question. Being polite to the model sometimes helped performance. Sometimes it lowered it. The effect disappeared in aggregate but reappeared at the question level. The techniques that supposedly “always work” — don’t always.

That doesn’t mean prompt engineering doesn’t matter. It means the rules are more conditional than most guides admit. The real mistakes aren’t the ones you’ve read about a hundred times (“be specific,” “give examples”). They’re structural: treating prompts as one-off text rather than software, skipping measurement entirely, building for demos instead of production, and copying techniques between model families that behave differently. Those mistakes have measurable costs — in tokens, in rework, in eroded trust. Here’s what each one looks like and how to fix it.

It is hard to know in advance whether a particular prompting approach will help or harm the LLM’s ability to answer any particular question.

Meincke, Mollick, Mollick & Shapiro — Wharton School, March 2025 (arXiv:2503.04818)

Mistake 1: Building Prompts That Work in Your Demo, Not Your Production Data

This is the most expensive mistake and the least discussed one. Aakash Gupta, who reviewed over 1,500 academic papers on prompt engineering, names it first in his 2025 analysis: “Teams build prompts that work great with clean, predictable inputs but fall apart with real user data.” The problem is that when you’re writing a prompt, you’re testing it against your mental model of what users will ask — which is almost always cleaner, more specific, and more charitable than what they actually send.

Production inputs are messier. They include typos, ambiguous phrasing, mixed languages, and edge cases you didn’t imagine. A prompt that produces perfect outputs in development will encounter a real user’s input and produce something confidently wrong. The fix isn’t to make the prompt longer. It’s to test against real inputs before shipping. Collect a sample of actual user queries — even 20–30 — and run your prompt against them before you call anything production-ready. If you’re building something new with no historical data yet, construct adversarial inputs deliberately: misspellings, contradictory requests, ambiguous scope. Your prompt’s performance on those inputs is its real performance.

Mistake 2: No Measurement System

You cannot improve what you cannot measure. This sounds obvious. Almost nobody does it. Gupta’s analysis frames it bluntly: “Trying to improve prompts without systematic measurement. You can’t optimize what you can’t measure reliably.” What this looks like in practice: a developer tweaks a prompt, tests it manually against three examples, decides it’s better, and ships it. Three weeks later, a different developer tweaks it again. Neither change was tracked. Neither improvement was validated against a stable test set. The prompt’s version history lives in someone’s memory, or a Slack message, or nowhere.

The minimum viable measurement system is a test set of 20–50 representative inputs with expected outputs, a consistent scoring rubric (even a simple 1–5 human rating), and a place to store prompt versions with their scores. Tools like Agenta, Langfuse, and PromptLayer automate this — but even a shared spreadsheet with columns for prompt version, test input, expected output, and score beats nothing. The goal is to make “this change made things better” a verifiable claim rather than a feeling.

Mistake 3: Copying Prompts Across Model Families

The prompt that works on GPT-4o will not necessarily work on Claude. Not because one is better than the other — but because they were trained differently and respond to different structural cues. Gupta’s 2025 analysis names this explicitly as a top mistake: “Using the same prompt across different models. Each model family has different strengths and responds to different techniques.” Anthropic’s own documentation notes that Claude specifically responds better to XML-tagged structure, producing 15–20% better performance from that formatting change alone — because that structure is baked into how Claude processes information.

A concrete example of the divergence: the Wharton research found that chain-of-thought prompting (“think step by step”) produces different reliability profiles depending on the model and the question type. Their second report, Prompting Science Report 2: The Decreasing Value of Chain of Thought (June 2025), found that the technique’s value has decreased as models have improved their native reasoning — it helps more for some question types and less for others, and the pattern isn’t consistent across model families. If you’re copying chain-of-thought instructions from a GPT tutorial into a Claude deployment without testing, you may be paying for extra tokens while getting no quality improvement.

The fix: treat each model family as requiring its own baseline prompt. Read the model provider’s official prompting documentation before assuming transfer. Test your top-five use cases on the actual model you’re deploying, not on a different one you found examples for.

Mistake 4: Paying for Tokens You Don’t Need

Verbose prompts feel thorough. They’re usually expensive and often counterproductive. The economics here are concrete: DataStudios’ 2025 prompt ROI analysis found that optimized prompting — more compact instructions, dynamic system messages — produces 30–50% token savings in batch operations, measurable directly through log analysis. That’s not a rounding error. At scale, it’s the difference between a $3,000 monthly API bill and a $1,500–$2,100 one for the same output quality.

The specific waste patterns are well-documented. AI Cost Board’s token optimization guide identifies the two largest sources: static system prompts repeated on every call (consuming 15–25% of total tokens per request), and full conversation history included when only recent context matters (another 10–20%). Both are fixable without any change to prompt quality. Prompt caching — available on Anthropic’s Claude and OpenAI’s API — can reduce cached input token costs by up to 90% for static content like system instructions and knowledge bases that don’t change between requests. Most teams using these APIs haven’t enabled it.

A quick audit: look at your most frequent LLM calls. What percentage of input tokens is system prompt content that never changes? If that number is above 40%, caching alone will materially reduce your costs. If your system prompt contains lengthy explanations of what the AI is, long lists of rules it should follow in every edge case, and disclaimers about what it cannot do — most of that can be compressed without any quality loss. Concise, precise instructions typically outperform verbose ones because they leave less room for the model to find an interpretation you didn’t intend.

Mistake 5: Skipping the Output Format Specification

If you don’t specify an output format, you’re defaulting to free-form text — which is fine for brainstorming and almost always the wrong choice for anything you’ll use programmatically. A 2026 prompt engineering guide on Medium frames this precisely: “Format drives consistency, evaluation, and integration.” A classification result in natural language might run 50–100 tokens. The same result as JSON runs 10–20 tokens. Structured outputs aren’t just cheaper — they’re more reliable, easier to parse downstream, and easier to evaluate systematically.

The failure mode here is subtle. Free-form outputs look fine when you read them. They break your downstream code when the model decides to wrap its JSON in a markdown code block, or adds a preamble sentence before the structured content, or uses a slightly different key name than you expected. OpenAI’s JSON mode, Anthropic’s tool use / function calling syntax, and structured output features in both APIs enforce format at the response level. Use them. When you need natural language output, specify format in the prompt itself: sentence count, paragraph structure, whether to include a preamble, whether to include a conclusion. “Write a summary” produces variable results. “Write a 3-sentence summary in past tense, no preamble” produces consistent ones.

Mistake 6: Treating “Think Step by Step” as a Universal Fix

Chain-of-thought prompting — asking the model to reason through a problem before giving an answer — genuinely helps on complex multi-step reasoning tasks. It also adds tokens, slows response time, and (per the Wharton research) produces inconsistent effects depending on the question and the model. Beginners discover it works on one problem and apply it everywhere, including tasks where it adds cost with no quality improvement: simple classification, formatting, extraction from structured data, single-step lookups.

The calibration question before adding chain-of-thought: does this task require multi-step reasoning, or does it require pattern matching? If a human could answer it in three seconds without showing their work, chain-of-thought is probably overhead. If a human would need to think it through — scenario analysis, legal interpretation, complex debugging — it’s worth the tokens. The Wharton team’s finding that chain-of-thought’s value is “decreasing” as models improve their native reasoning suggests the threshold will keep shifting. Test it rather than assume it.

Mistake 7: One-Pass Generation for High-Stakes Output

The most reliable improvement for critical outputs costs no extra tokens at all. The 2026 Medium guide describes it as a two-pass approach: separate generating from reviewing. Instead of asking for a perfect answer in one shot, use a first pass to generate, then a second pass (in the same prompt or a separate one) to review against explicit criteria. “Draft → review → finalize” catches the class of errors the Wharton research identified as particularly hard to prevent: missed constraints, broken formats, unsupported claims, unnecessary complexity. These errors are common precisely because a model generating under instruction doesn’t naturally step back to check its own work against the original requirements.

For non-critical outputs this is overkill. For anything that gets shown to customers, submitted to clients, used in financial decisions, or used as training data — the review pass is cheap insurance. The review prompt doesn’t need to be elaborate: “Review the output above against these criteria: [list]. Flag any issues. If none, output ‘PASS’.” A PASS rate below 90% on your test set tells you your generation prompt needs work. A PASS rate above 98% tells you the review criteria may be too permissive.

The One Thing That Runs Under All of These Mistakes

Every mistake in this list shares a root cause: treating prompt engineering as writing rather than engineering. Writing is evaluated by a reader in the moment. Engineering is evaluated by systematic testing over time, against real inputs, with a documented feedback loop. The teams that compound their AI investments are the ones with a test set, a versioning system, and a scoring rubric — not necessarily the ones with the most sophisticated prompts. A mediocre prompt that gets measured and iterated will outperform an excellent prompt that never gets tested against production data.

The Wharton research finding — that prompting effects are inconsistent at the aggregate level but real at the question level — has a practical implication: you cannot borrow someone else’s results. What works for their task, on their model, with their data, may not transfer. The only way to know if a technique works for your case is to test it on your case. That’s not a pessimistic conclusion. It means the competitive advantage belongs to teams with good measurement infrastructure, not teams that read the most guides.

A mediocre prompt that gets measured and iterated will outperform an excellent prompt that never gets tested against production data.

Synthesis: Wharton Prompting Science Report 1 (2025), Gupta medium.com analysis, DataStudios prompt ROI research

Where to Start This Week

Pick your highest-volume LLM call — the one running hundreds or thousands of times a day. Do these four things in order.

First, measure current token usage. Most API clients return token counts in every response. Log them for one day. Calculate input tokens versus output tokens. Identify whether your system prompt is the dominant input cost — if it’s above 40% of total input tokens, you have an easy optimization before you change a single instruction.

Second, build a test set from real inputs. Pull 25–30 actual requests the system has processed. Label each with an expected output or a quality rating. This is your baseline. Every subsequent prompt change should be validated against it.

Third, check your output format specification. If you’re asking for structured data and parsing the response with regex or string manipulation, you’re doing it the hard way. Switch to JSON mode or tool/function calling syntax and let the API enforce the format.

Fourth, if you’re using chain-of-thought on every call, run an A/B test against your test set without it. If quality drops, keep it. If quality holds, you’ve found free token savings.

None of this requires new tooling. All of it is measurable within a week. The teams winning on prompt quality right now aren’t the ones with the most sophisticated techniques — they’re the ones who built a feedback loop and actually use it.

---

Sources

• Meincke, Mollick, Mollick & Shapiro: “Prompting Science Report 1: Prompt Engineering Is Complicated and Contingent” — Wharton School / arXiv, March 2025
• Meincke, Mollick, Mollick & Shapiro: “Prompting Science Report 2: The Decreasing Value of Chain of Thought” — Wharton School / SSRN, June 2025
• Aakash Gupta: “I Spent a Month Reading 1,500+ Research Papers on Prompt Engineering” — Medium, July 2025
• DataStudios: “Prompt ROI: How to Measure the Real Value of Prompt Engineering” — August 2025
• AI Cost Board: “Token Optimization Guide: Reduce AI API Costs Without Quality Loss” — February 2026
• Mario G.: “Prompt Engineering Basics (2026): A Practical Guide” — Medium, January 2026
• Stack Overflow 2025 Developer Survey — AI section (49,000+ respondents)