Mastering AI Outputs: A Professional’s Guide to Fine-Tuning with Precision

AI-driven SEO trends
AI-driven SEO trends
AI Outputs - 1
AI Outputs - 2

Tuning AI Outputs: Sampling Parameters, Structured Outputs and Fine-Tuning (What Still Works)

Last reviewed 18 October 2026 · About 12 minutes to read · By Tom Morgan

The short answer: temperature and top-p are still useful, but only on some models, and they matter far less than most guides claim. On many current reasoning models, you can’t set them at all. For reliable output, use structured outputs, a small evaluation set, and, only when those aren’t enough, fine-tuning. This guide shows which controls your model actually exposes, what the research says they do, and how to test your own settings in an afternoon.

Key takeaways

  • Check what your model allows before tuning anything. OpenAI’s reasoning models, Anthropic’s Claude and Google’s Gemini, each handle sampling differently (table in section 1).
  • Temperature 0 is not a truth setting. Published tests find little accuracy change across the 0 to 1 range, and outputs at 0 can still vary between runs.
  • Structured outputs beat parameter tweaking for extraction, classification, and anything a program has to parse.
  • Fine-tune last, not first. Prompting, examples, and retrieval are cheaper. Fine-tuning pays off for consistent format, tone, or high-volume cost reduction, and it needs an evaluation set before you start.
  • Measure, don’t copy. Every “ideal settings” table, including the one below, is a starting point. Section 7 gives you a script to find the right values for your task.

Contents

  1. Which controls does your model actually give you?
  2. What the sampling parameters do
  3. Five beliefs the research doesn’t support
  4. Starting points by task
  5. Levers that usually beat temperature
  6. Fine-tuning: when it pays off and what to use
  7. Run your own benchmark
  8. What to do this week
  9. FAQ
  10. Sources

1. Which controls does your model actually give you?

Most advice about “the perfect temperature” was written for chat models from 2023 and 2024. The API surface has changed since. Here is a snapshot as of October 2026.

Model familyTemperatureTop-pTop-kFrequency/presence penaltyWhat you tune instead
OpenAI reasoning models (GPT-5 family, o-series)Fixed, not adjustableFixedNot offeredNot supportedReasoning effort, verbosity, max output tokens
OpenAI non-reasoning models (e.g. GPT-4.1, GPT-4o)0 to 2YesNot offeredYes (−2 to 2)logit_bias, seed (best-effort determinism)
Anthropic Claude0 to 1 (default 1.0)YesYesNoThinking and effort settings. Some newer models reject temperature and top-p set together
Google Gemini0 to 2 (default 1.0)YesYesOn some modelsThinking level or budget. Google advises leaving temperature at the default on its newer reasoning models
Open-weight models (vLLM, llama.cpp, Ollama)Usually 0 to 2YesYesYes, plus repetition_penaltymin_p, grammar-constrained decoding

Two cautions. Vendors change these rules often, and the Claude and OpenAI reasoning rows age fastest, so confirm against the current API reference before you ship (links in Sources). And if you use a model through a wrapper such as a cloud marketplace or an orchestration framework, the wrapper may expose or hide parameters differently from the vendor’s own API.

Why this matters: a lot of published advice tells you to set frequency_penalty to 0.8 on “Claude.” That parameter doesn’t exist in Claude’s API. Advice that ignores the model family is the first thing to discard.

2. What the sampling parameters do

In plain English: a model writes one small piece of text (a token) at a time. Before each piece, it ranks every possible next token by how likely it is. Sampling settings decide how strictly it sticks to the top of that ranking. Strict gives predictable text, loose gives varied text, and too loose gives nonsense. Under the hood, the ranking is a list of scores called logits.

Temperature is the strictness dial. A low value lets the top-ranked token dominate. A high value gives unlikely tokens a real chance. (For the curious: the probability of token i is exp(zᵢ / T) / Σ exp(zⱼ / T), where z is the token’s score.) APIs usually treat T = 0 as greedy decoding, meaning “always take the top token.”

Top-p (nucleus sampling) keeps only the smallest set of tokens whose probabilities add up to p, then samples from that set. It was introduced by Holtzman et al. (2019) to stop the long tail of odd tokens from derailing text. Top-p 0.9 means “ignore the bottom 10% of probability mass.”

Top-k keeps only the k most likely tokens. It is blunter than top-p because it ignores how confident the model is.

Min-p is a newer option on open-weight stacks. It keeps tokens whose probability is at least a fraction of the top token’s probability, so the cutoff adapts to model confidence. The authors report better coherence at high temperatures, though the benefit is still debated.

Frequency and presence penalties subtract from a token’s logit based on how often it has already appeared. OpenAI documents the formula as: logit − (count × frequency penalty) − (1 if count > 0, else 0) × presence penalty. Frequency scales with repetition. Presence is a flat one-time penalty.

Max tokens and stop sequences are limits, not style controls, but they affect cost and structure. A stop sequence for a double newline is written "\n\n" in JSON. Many guides print it garbled, which silently breaks the setting.

Practical rule: change one of temperature or top-p, not both. They interact, and some APIs now enforce this.

3. Five beliefs the research doesn’t support

“Temperature 0 gives the most accurate answers”

Renze and Guven (2024) tested nine models on multiple-choice problem-solving tasks and found no statistically significant accuracy change when temperature moved between 0.0 and 1.0. Lower temperature can help with consistency, but it isn’t an accuracy dial. If your task fails at 0.7, it will usually still fail at 0.

“Temperature is the creativity knob”

Peeperkorn et al. (2024) had a model write short stories at different temperatures and had them rated. Temperature correlated weakly with novelty and moderately with incoherence. Raising it mostly makes text less predictable, not more inventive. Creative quality comes much more from the prompt, the examples, and the model.

“At temperature 0 you get the same output every time”

No, and on hosted APIs you should assume it won’t. Thinking Machines Lab (2025) traced much of the run-to-run variation in greedy decoding to how inference servers batch requests, so results can change with server load even when your settings don’t. A seed is best-effort at most and doesn’t remove batching effects. Design for variance instead: validate every output against a schema, cache results you need to repeat, write tests that accept a range of valid answers, and run evaluations several times so you can see the spread. Only a server you control, configured for batch-invariant inference, can get close to identical output.

“Penalties are a free fix for repetition”

Penalties apply to every token, including the ones you want repeated: JSON keys, code identifiers, product names, a legal term that must appear in each clause. In structured or technical output, penalties above about 0.5 often do more harm than good. If repetition is the problem, fix the prompt or shorten the generation first.

“The same settings work on every model”

A temperature of 1.0 means different things on different models because each model has its own logit scale and training history. Settings that suit one model are a guess for another. This is why section 7 exists.

4. Starting points by task

These are conventional starting values, not benchmarked results. They assume a model that exposes the parameters. For OpenAI reasoning models, skip this table and use the effort setting instead.

TaskTemperatureTop-pPenaltiesNotes
Data extraction, classification01.0NoneUse a JSON schema. See section 5
Code generation, SQL0 to 0.31.0NonePenalties can break repeated identifiers
Legal or compliance drafting0.2 to 0.41.0NoneHuman review is non-negotiable, whatever the setting
Customer support replies0.3 to 0.70.9 to 1.00 to 0.3Brand tone usually comes from examples, not sampling
Long-form drafting0.7 to 1.00.9 to 1.00 to 0.5Vary the outline, not just the dice
Brainstorming (many options)1.0 to 1.30.950 to 0.5On Claude, cap at 1.0 and generate more samples instead

Where I’d be cautious: any table that gives penalties of 1.0 or more as a default. At those levels, many models start avoiding common words, and the output reads as strange rather than varied.

5. Levers that usually beat temperature

Structured outputs. For anything a program will read, constrain the output with a JSON schema or grammar. OpenAI, Anthropic, and Google all offer some form of schema-constrained output, and open-weight servers support grammar-constrained decoding. This removes a whole class of failures that no temperature setting can reach.

Better instructions and examples. Two or three well-chosen examples of the exact output you want do more for tone and format than any sampling change. If the model misunderstands the task, parameters can’t fix it.

Reasoning effort. For hard problems on reasoning models, the effort or thinking setting is the main quality-versus-cost control.

Sampling several times and choosing. Generate a handful of candidates and pick one with a rule, a checker, or a second model call. This is the dependable way to get variety and quality together, and it works on models where temperature is locked.

Retrieval. If the model gets facts wrong because it lacks your data, give it the data at query time. Fine-tuning is a poor way to add facts.

An evaluation set. Without 50 to 200 representative inputs and a way to score the outputs, every change is a guess.

6. Fine-tuning: when it pays off and what to use

Try these first, in this order

  1. A clearer prompt with examples.
  2. Structured outputs.
  3. Retrieval for missing knowledge.
  4. A stronger or reasoning-capable model.
  5. Fine-tuning.

Fine-tuning earns its cost when you need a consistent house format or voice across thousands of outputs, when a task is narrow and high-volume so a smaller tuned model can replace a larger one, or when the domain vocabulary is unusual enough that prompting keeps missing it.

A common mistake is tuning for style or format because the first prompt didn’t produce it. Before you tune, build the strongest baseline you can: a detailed prompt, three to five examples, and a schema-constrained output. Run it on your evaluation set. If it lands within a few points of your target, tuning mostly adds cost: data curation, an evaluation harness, retraining when the base model is retired, and a model you now have to maintain. Tune only when the best prompt baseline still loses on your own numbers.

The main methods

  • Supervised fine-tuning (SFT): train on input-output pairs. The default starting point.
  • Preference tuning (for example, DPO): train on pairs of better and worse answers, useful for tone and judgment calls.
  • Reinforcement-style fine-tuning: train against a grader that scores outputs, useful when you can check correctness automatically.
  • LoRA and QLoRA: train small adapter weights instead of the whole model. LoRA (Hu et al., 2021) and QLoRA (Dettmers et al., 2023) are why a single GPU can handle many fine-tuning jobs on open models.

Tools worth knowing in 2026

ToolTypeGood forCheck before you commit
AxolotlOpen source, YAML-configuredLoRA/QLoRA and full fine-tunes on your own GPUsHardware and your team’s comfort with training configs
UnslothOpen sourceMemory-efficient LoRA/QLoRA on limited GPUsWhich model architectures are supported
Hugging Face TRLOpen source librarySFT, DPO, and RL-style training in codeNeeds ML engineering skills
Vendor fine-tuning APIs (OpenAI and others)HostedTuning without managing infrastructureWhich models are currently tunable, and whether tuned models cost more to run
Hosted tuning platforms for open-weight modelsHostedTuning without running GPUsPricing, data handling, and how long your base model stays supported

Treat this table as a map, not a recommendation. Products in this space change quickly, so confirm current features and pricing on each vendor’s site.

Forgetting and how to limit it

Fine-tuning can reduce a model’s general ability while it learns your task. Biderman et al. (2024) compared LoRA with full fine-tuning and found LoRA usually learns less on the new task but also forgets less of the base model’s abilities. Practical guards:

  • Keep a small general-ability test set and run it before and after tuning.
  • Use LoRA or a low learning rate rather than full fine-tuning when general ability matters.
  • Mix some general examples into the training data.
  • Compare against the untuned model with the best prompt you can write. If the gap is small, don’t tune.

A simple cost break-even

Fine-tuning has upfront costs (data preparation, training, evaluation, engineering time) and ongoing ones (hosting, retraining). The saving comes from a cheaper model per call.

Months to break even = upfront cost ÷ (monthly calls × cost saved per call)

Hypothetical example, for illustration only: 2,000,000 calls a month, a cost of $0.004 per call on the large model and $0.0008 on the tuned small one, and $15,000 upfront. Savings are $6,400 a month, so break-even is about 2.3 months. At 100,000 calls a month, the same project takes about 47 months and probably isn’t worth it. Use your own numbers, and add the costs people forget: curating training data, rebuilding when the base model is deprecated, and the engineer time to keep the evaluation current.

7. Run your own benchmark

Rules of thumb only go so far, so here is a way to test settings on your own task in an afternoon. The script below sweeps temperature and top-p on a task with a checkable answer and reports accuracy with a 95% confidence interval, so you can see whether a setting makes a real difference or just noise.

Method

  1. Build two files of numeric-answer questions, dev.jsonl and test.jsonl50 to 200 each, one per line: {"q": "…", "a": 42}. Choose settings on dev, then report numbers from test once, so you aren’t just picking the luckiest setting.
  2. Point the client at an OpenAI-compatible endpoint (a local vLLM or Ollama server works, or a vendor that supports the same interface).
  3. Run each setting several times per question, because a single sample hides variance.
  4. Compare intervals, not point estimates. If two settings’ intervals overlap heavily, you have no evidence they differ. The interval resamples questions, not repeats, because repeated runs of one question aren’t independent.

python

"""sweep.py - measure how temperature and top_p change accuracy on YOUR task.
Usage: python sweep.py dev.jsonl"""
import csv, json, random, re, sys
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")  # any OpenAI-compatible server
MODEL = "your-model-name"
TEMPS = [0.0, 0.3, 0.7, 1.0, 1.3]
TOP_PS = [1.0, 0.9]
RUNS = 5  # repeats per question per setting

def bootstrap_ci(per_question, reps=2000, seed=0):
    """95% interval from resampling QUESTIONS (repeats of one question are not independent)."""
    rng = random.Random(seed)
    n = len(per_question)
    means = sorted(sum(rng.choices(per_question, k=n)) / n for _ in range(reps))
    return means[int(0.025 * reps)], means[int(0.975 * reps)]

def extract(text):
    hits = re.findall(r"ANSWER:\s*(-?[\d,]*\.?\d+)", text)
    return float(hits[-1].replace(",", "")) if hits else None

tasks = [json.loads(line) for line in open(sys.argv[1])]  # {"q": "...", "a": 42}

rows = []
for t in TEMPS:
    for p in TOP_PS:
        per_question = []
        for task in tasks:
            hits = 0
            for _ in range(RUNS):
                out = client.chat.completions.create(
                    model=MODEL, temperature=t, top_p=p, max_tokens=800,
                    messages=[{"role": "user",
                               "content": task["q"] + "\nFinish with 'ANSWER: <number>'."}],
                ).choices[0].message.content
                hits += extract(out) == float(task["a"])
            per_question.append(hits / RUNS)
        lo, hi = bootstrap_ci(per_question)
        acc = sum(per_question) / len(per_question)
        rows.append([t, p, len(tasks), RUNS, round(acc, 3), round(lo, 3), round(hi, 3)])
        print(rows[-1])

with open("results.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["temperature", "top_p", "questions", "runs", "accuracy", "ci_low", "ci_high"])
    w.writerows(rows)

Notes on using it. The script scores exact numeric answers, which suits math and extraction. For text tasks, don’t rely on regex matching: check against a JSON schema or a rubric, add a reviewed model-as-judge if you need one, and spot-check by hand. This is a teaching script, not a production harness; for real decisions, use a larger held-out set and log every raw output. If a vendor rejects both parameters together, vary one at a time. For creative tasks, replace accuracy with a diversity measure such as the share of distinct two-word sequences across outputs (distinct-2), and have people rate quality, since automatic scores miss most of what matters.

Your results. Publish your own sweep alongside the script: model and version, number of questions, run date, and the results table. Numbers measured on your task are worth more than any rule of thumb in this article. <!– RESULTS TABLE: add after running sweep.py (model, n, date, temperature, top_p, accuracy, 95% interval) –>

8. What to do this week

  • List every place your product calls a model and note which parameters each call sets. Delete any that the model ignores or rejects.
  • For extraction and classification, switch to schema-constrained output.
  • Build an evaluation set of at least 50 real inputs with known good outputs.
  • Run the section 7 sweep on your most important task and keep the winning settings.
  • Before any fine-tuning project, write down the number you’re trying to move and the untuned baseline you must beat.
  • Put a review date on any parameter guidance you publish or share internally. It expires.

9. FAQ

What temperature should I use? It depends on the model and task. Start at 0 for extraction and code, 0.3 to 0.7 for support and drafting, and higher only for brainstorming, then measure. On models that lock temperature, you can’t set it.

Does temperature 0 stop hallucinations? No. It makes output more consistent, not more truthful. Retrieval, structured outputs, and verification reduce unsupported claims far more.

Can I use top-p and temperature together? Many people do, but changing both makes results hard to interpret, and some newer APIs reject the combination. Pick one to tune.

Does Claude support frequency or presence penalty? No. Anthropic’s API offers temperature, top-p, and top-k, not those penalties. Check the current API reference for your model.

Do reasoning models let me set temperature? Usually not. OpenAI’s reasoning models fix it, and you control effort instead. Rules differ by provider, so check the documentation.

When is fine-tuning worth it? When prompting, examples, retrieval, and a stronger model haven’t closed the gap, and you have an evaluation set and enough call volume for the savings to cover the upfront cost.

Will fine-tuning teach the model new facts? It’s unreliable for that. Use retrieval for facts and fine-tuning for style, format, and task behaviour.

10. Sources

Papers and technical write-ups

Vendor documentation (confirm current parameter support here before publishing or shipping)

Open-source tools

Parameter ranges and model behaviour were checked against general knowledge current to mid-2026 and are not a substitute for the live vendor documentation above. If you find something out of date, tell us, and we’ll correct it and log the change here.

Change log: 18 October 2026, full rewrite. Replaced the May 2025 version; removed unsourced case studies and quotes, corrected provider-specific parameter advice, and removed the off-topic SEO section.

Tags

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *