When Fine-Tuning Is the Right Tool

Why this matters

A team spends six weeks fine-tuning an open model to answer questions about their internal policies. The first evaluation looks excellent. A month later the leave policy changes, and the model keeps confidently quoting the old one. Re-training takes another week. Meanwhile, a different team with the same problem built a retrieval pipeline in four days and updates it by replacing a document.

Fine-tuning is one of the most requested skills in AI engineering, and one of the most misapplied. It is powerful for the right problem: teaching a model a consistent format, a narrow classification task, a house style, or how to behave like a much larger model on one job at a fraction of the cost. It is the wrong tool for keeping facts current, and an expensive way to fix a problem a better prompt would have solved.

This course teaches the full workflow on open-weight models: choosing the approach, preparing data, training efficiently with LoRA and QLoRA, evaluating honestly, and serving the result. This first lesson is about the decision, because the cheapest fine-tuning project is the one you correctly decide not to run.

The idea

Start from the failure, not the technique. Write down, in one sentence, what the current system does wrong and how you would measure it. Then match the failure to the tool.

Prompting changes the instructions and examples the model sees. Try it first, always. It is fast to change, needs no training infrastructure, and modern models follow detailed instructions well. If a careful prompt with a few examples meets your quality bar, stop.

Retrieval-augmented generation (RAG) supplies relevant documents at question time. It is the right tool when the model lacks knowledge, especially knowledge that changes, must be cited, or depends on who is asking. Updating a document updates the answer.

Fine-tuning changes the model's weights using examples of inputs and desired outputs. It is the right tool when the model has the knowledge but not the behaviour: it needs a strict output format every time, a classification scheme it keeps confusing, a tone that prompts cannot hold, or domain phrasing it handles poorly. Fine-tuning also lets a small model learn one task well enough to replace a large, expensive model on that task, which cuts latency and cost at volume.

The options combine. A common production pattern is a fine-tuned small model that follows the output format reliably, fed by retrieval for the facts.

Then price it. Fine-tuning has costs that prompting does not: collecting and labelling data, training runs, evaluation, hosting, and re-training when the base model or the task changes. It pays off when volume is high, the task is stable, and the quality gain or per-request saving is large. It rarely pays off for a low-volume feature whose requirements are still moving.

Finally, check the licence. Open-weight models come with licences that differ on commercial use, attribution and acceptable use. Read the licence of the base model before you invest in training on it.

Diagram

Worked example

A support platform routes 40,000 tickets a day. The current system calls a large hosted model with a 3,200-token prompt, most of it few-shot examples, and gets 150 tokens back: a category, a priority and a short reason in JSON. Quality is good. The bill is the problem.

You price the options with illustrative figures. At 3 US dollars per million input tokens and 15 per million output tokens, the hosted approach costs about 474 dollars a day. A fine-tuned small open model on two rented GPUs, at 2.50 dollars per GPU-hour, plus an estimated 1,500 dollars a month of upkeep for re-training, monitoring and engineering time, costs about 170 dollars a day. Add a one-off 4,000 dollars for labelling and training, and the fine-tuned option breaks even after about 13 days.

Before committing, you check the two conditions that make the numbers real. First, is the task stable? The category list has changed twice in two years, so yes. Second, can a small model reach the quality bar? You run a two-day pilot: 2,000 labelled tickets, a quick LoRA run, and an evaluation set built as the LLMOps course describes. The pilot model matches the large model's accuracy within one point and produces valid JSON on every case.

Notice how sensitive the decision is to volume. At 10,000 tickets a day the hosted option costs about 118 dollars a day, still below the fine-tuned option's fixed 170. At 2,000 tickets a day it costs about 24 dollars. For a smaller company the right answer would be to keep prompting and optimise the prompt with caching.

Code

This standalone script reproduces the worked example. It uses only the Python standard library; every figure is illustrative and should be replaced with your own prices and quotes.

# Illustrative figures: replace with your own prices, volumes and quotes.
REQUESTS_PER_DAY = 40_000

# Option A: a large hosted model with a long few-shot prompt.
A_INPUT_TOKENS, A_OUTPUT_TOKENS = 3_200, 150
A_PRICE_IN, A_PRICE_OUT = 3.00, 15.00          # USD per million tokens

# Option B: a fine-tuned small open model on rented GPUs.
B_GPU_PER_HOUR, B_GPUS = 2.50, 2               # two GPUs for capacity and redundancy
B_ONE_OFF = 4_000                               # data labelling, training runs, evaluation
B_MONTHLY_UPKEEP = 1_500                        # re-training, monitoring, engineer time

a_daily = REQUESTS_PER_DAY * (A_INPUT_TOKENS * A_PRICE_IN + A_OUTPUT_TOKENS * A_PRICE_OUT) / 1_000_000
b_daily = B_GPU_PER_HOUR * 24 * B_GPUS + B_MONTHLY_UPKEEP / 30
print(f"A: ${a_daily:,.0f}/day   B: ${b_daily:,.0f}/day plus ${B_ONE_OFF:,} one-off")
saving = a_daily - b_daily
print(f"break-even after {B_ONE_OFF / saving:.0f} days" if saving > 0 else "B never breaks even")
for volume in (2_000, 10_000):
    a = volume * (A_INPUT_TOKENS * A_PRICE_IN + A_OUTPUT_TOKENS * A_PRICE_OUT) / 1_000_000
    print(f"at {volume:,} requests/day: A ${a:,.0f}/day vs B ${b_daily:,.0f}/day")

Expected output: A at 474 dollars a day, B at 170 dollars a day plus 4,000 one-off, break-even after 13 days, then A at 24 and 118 dollars a day for the two smaller volumes against B's 170.

Common mistakes

"Fine-tuning teaches the model our documents." It is tempting because training on documents seems like the obvious way to add knowledge. Models absorb facts from fine-tuning unreliably, cannot cite them, and keep stale versions after the source changes. Use retrieval for knowledge.

"Fine-tune first, because it will beat any prompt." It sounds rigorous. A well-structured prompt on a capable model is often good enough, and it is the baseline any fine-tuned model must beat. Without that baseline you cannot tell whether training helped.

"Training is the main cost." A single LoRA run on a small model can cost little. Data preparation, evaluation, hosting and re-training usually cost far more over a year. Price the whole lifecycle.

"Any open model is free to use commercially." Open weights are not the same as an unrestricted licence. Read the base model's licence before training.

Your turn

Pick one model-powered feature you know. Write the failure in one sentence and how you would measure it. Classify it as a knowledge, behaviour or cost problem, and name the first approach you would try.

Then adapt the script with your own volume and prices, including a realistic upkeep figure.

Success criterion: you can state which approach you would choose, the volume at which your answer would change, and the evidence a two-day pilot would need to produce before you commit.

Recap

• Choose the technique from the failure: knowledge problems point to retrieval, behaviour and format problems to fine-tuning, and everything starts with prompting.
• Fine-tuning changes weights, so it is slow to update and poorly suited to facts that change.
• A fine-tuned small model can replace a large model on one stable, high-volume task at much lower cost.
• The economics depend heavily on volume and must include data, evaluation, hosting and re-training.
• Open-weight model licences differ and must be checked before any training investment.

Sources

• Ouyang et al. (2022), Training language models to follow instructions with human feedback. Shows supervised fine-tuning and preference training changing model behaviour more than model size alone.
• Zhou et al. (2023), LIMA: Less Is More for Alignment. Evidence that a small number of high-quality examples can shape output style and format.
• Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. The original RAG formulation for supplying external knowledge at inference time.