Small Models, Narrow Jobs: What Fine-Tuning SLMs for a Talk-to-Data Agent Taught Us

We replaced GPT in an airline's analytics agent with two small fine-tuned models that run on our own hardware. The models were the easy part. Data, measurement, and knowing when to say “I don't know” were not.

The short version

1. Decompose until each model's job is small. A 360M model writes good SQL once it knows which question it is answering. Choosing that question is the hard part, so that is where the bigger model went.
2. Row counts lie. Our early datasets had thousands of rows but only 57–249 genuinely different questions behind them.
3. A wrong answer is worse than “I don't know.” Score correct, safe, and wrong separately, and never buy correct with wrong.
4. Check the ruler. Our most expensive mistakes were in measurement, not modelling.
5. Be honest about the frontier model. GPT-5 was more accurate. Small models win on privacy, cost, and control, and GPT-5 earned its keep as an auditor of our labels.
6. Judge a new model on every test. Each retraining improved what it targeted and moved something else.

Why small models at all?

The agent answers business questions in plain English: “What was the load factor on our busiest route last week?”, “Which agencies sold the most tickets in September?”, “Show me cancellations by weekday.” It turns them into queries over the airline's reporting database and explains the result.

A frontier model does this well, but every question is a paid API call, the schema and the question leave the building, and latency depends on someone else. The domain is narrow and the database fixed, which is exactly where small fine-tuned models should shine. We used SmolLM2 (360M and 1.7B), LoRA fine-tuning, and llama.cpp for serving. That stack worked from day one. Everything else took iteration.

The architecture we ended up with

The planner turns the question into a small JSON plan: a capability from a fixed catalogue (“load factor by route for a date range”) plus slot values such as route and dates, or it asks to clarify or says the data can't answer. A validator checks the plan, then the question goes to the SQL model, to a reviewed governed query, or to a deterministic tool for metrics with fixed definitions. Dates (“last week”) and staff shorthand (“LF”, “pax”, “RBD”) are rewritten by code before any model sees them: Python is perfect at calendar arithmetic, and small models are not.

Two small models sit inside deterministic code. The planner never writes SQL; the SQL model never chooses what to answer.

Lesson 1: Decompose until each model's job is small

We first ran 210 held-out questions through the whole agent with two 360M models. When we traced the wrong answers, almost all came from the planner choosing the wrong capability or filling a slot wrongly; only about two were genuine SQL mistakes.

Given the right question, a 360M model writes the right SQL almost every time. Choosing the right question from 330 similar-sounding ones is a different, harder job.

So we gave the planner 1.7B parameters and kept the 360M SQL model. On the same 210 questions, through the same harness, correct answers rose from 153 to 181 and wrong answers fell from 22 to 6. Put capacity where the difficulty is, not everywhere.

Lesson 2: Row counts lie; count independent questions

Our first dataset versions grew to nearly 8,000 rows while accuracy on new phrasings barely moved. An audit explained why:

Most rows were the same question with different wrappers (“please”, a typo), and every question of a type got the identical SQL. Asking about a different route or month could not change the answer, so the model learned to pick one of 54 memorised strings. The rebuild starts from families: a question type with slots (route, dates, metric), many independently written phrasings, and an answer generated from the slot values.

Rule of thumb: Before celebrating a dataset size, count distinct (question meaning, answer) pairs. If the answer never varies with the question's details, the model is learning a lookup table.

Lesson 3: A wrong answer is worse than “I don't know”

In analytics, a confident wrong number ends up in someone's slide deck. So every evaluation scores three outcomes: correct, safe (the agent asked or declined), and wrong (it answered a different question).

That framing stopped a tempting optimisation. Small models sometimes invent capability names; the validator rejects them and the agent says it couldn't interpret the request, which is safe. Grammar-constrained decoding forces only valid names, so we measured it on a blind set of 157 questions:

Correct went up by two; wrong went up by eighteen. Unsure, the model used to produce an invalid name the validator caught. Constrained, it picked the nearest valid capability and answered the wrong question confidently. The grammar stays in the code, switched off.

Constraining a model's output makes it look more reliable while hiding the uncertainty you were relying on to stay safe.

Lesson 4: Evaluate blind, end to end, and check the ruler

Training loss on our 1.7B planner fell below 0.01. That says nothing about whether it will plan “cancellations by weekday” correctly. What made our numbers trustworthy:

• A sealed test set written blind, by a separate agent that saw the catalogue but never the training data. Once a set has been used to choose a model or shape data, it is demoted to development evidence and a new one is written.
• Scoring through the whole agent, because rewrites, validator, and guard all change the final answer.
• Choosing checkpoints by that test, not by loss. In our last run the lowest-loss checkpoint made 20 wrong plans; another made 17.
• Reading every wrong answer. It found scorer bugs and an agent bug that compared two periods backwards, which no score had shown.

Even so, in one day three comparisons were silently wrong: a model file exported under the wrong name, so no server started and 157 connection errors were scored as “safe”; a run against the old catalogue; a run against the old SQL contract. Later we found that timeouts were reported to users as “the model is unavailable” and scored as safe refusals. Every one produced plausible numbers. Our evaluation script now refuses to run unless every input matches a pinned hash and each server serves the intended file, and any failed request marks the whole run as not evidence.

A result far from the previous model's is a setup problem until proven otherwise.

Lesson 5: Measure the frontier model on equal terms

Before the small models, the planner was GPT-5. Our planner's instructions are one sentence, because it learned the catalogue in training, so to be fair GPT-5 got the whole catalogue in its prompt (about 18,000 tokens) and, in one variant, 40 worked examples. Same blind questions, same scorer:

With examples, GPT-5 made half the wrong plans of our best model, and at about half a cent per question it isn't expensive. Reading its mistakes also exposed our scorer: half its “wrong” plans followed equivalent conventions our models had simply learned from their labels.

For this job, the case for small models is privacy, predictable cost, and control, not accuracy. Say so plainly.

Lesson 6: Use the frontier model as an auditor, not a labeller

Letting GPT-5 write our labels would have imported its mistakes unexamined, so we used it to check them. It re-planned 5,081 training and validation records (never a test item) for USD 27.50, and every disagreement was judged against the catalogue's rules:

Raw agreement would have overstated our label errors fourfold. The real ones clustered in 40 of 438 question families, mostly one sorting convention applied two ways, because one template decision is copied into every phrasing. We fixed 62 families and had GPT-5 draft 19 new “contrast” families for decisions our models kept confusing, each reviewed before use.

Lesson 7: Judge a new model on every test

We retrained the 1.7B planner twice: first on data aimed at its known errors, then on the audited data. Each time the blind set improved, and each time something else moved:

The first retraining fixed what it targeted but pushed neighbouring decisions the wrong way, so we didn't ship it. The second answered 39 more of the 500 business questions, a clear gain, but refused more of the SQL test and made four new mistakes there. Its checkpoints traded the two against each other: the one that recovered the SQL test fell back to 293 on the bank. No checkpoint passed every gate we had set, and the owner chose the one that answers the most real business questions, with the previous model kept for rollback.

Decide before you look. We wrote promotion gates for all three suites before training. When the result fell short, the trade-off was explicit and the decision was the owner's, not a number we talked ourselves into.

Where we ended up

On a bank of 500 realistic business questions, labelled with what the data can and can't answer, the served agent now behaves as expected far more often than the agent we started with:

Both models run locally, with no per-question cost and no data leaving the organisation. We haven't reached our own standard of zero wrong answers: the remaining errors are mostly choices between neighbouring capabilities. That is where the next round of data, and a fresh blind test set, will go.

Lessons for the next project

• Write the capability catalogue and the “can't answer” list before any training example.
• Give each model one narrow job; push dates, vocabulary, and definitions into code.
• Build data from families with slots; report independent questions, not rows.
• Score correct, safe, and wrong separately, through the whole agent, on a blind sealed set.
• Test the scorer like production code, read every wrong answer, and count a failed request as an error.
• Choose checkpoints by the test, never by loss.
• Benchmark a frontier model with the same information, and use it to audit your labels, not write them.
• Set promotion gates on every suite before training, and promote only what passes them, or decide the trade-off explicitly.

The full measurements, including significance tests and the operational details, are in our research paper.