Fine-tuning is still romantic. Vendors sell it as the path to “your model, your moat.” For most product features, a strong base model + good prompts + retrieval + tools ships faster, cheaper, and fails more gracefully when the world changes.
Fine-tune when you have stable labelled data, a narrow domain style, and a clear offline metric — not when the base model “doesn't quite get our vibe” in a playground session.
We fine-tune when
- Output format or tone must be rock-solid across thousands of calls — and prompt engineering has hit a wall you can measure, not feel.
- You have clean data and ownership of training rights. Scraping customer tickets without consent is not a dataset.
- Latency or cost of huge prompts is the real bottleneck. A smaller fine-tuned model beating a frontier model on a narrow task is a legitimate win.
That is a short list on purpose. If you are not in all three neighbourhoods, you are probably not ready.
What wins first: prompts, RAG, tools
Most Forge builds never need a custom weight. The stack that ships:
- Prompt + schema. Structured outputs, few-shot examples, versioned markdown. Fixes 80% of “format drift” without touching weights.
- RAG. Private knowledge belongs in retrieval, not in memorised weights that go stale the day policy changes.
- Tools. If the answer requires live data — inventory, account status, calendar — a tool call beats a fine-tuned guess every time.
We run evals on this stack before anyone mentions fine-tuning. If quality is still short after prompts, retrieval, and tools are tuned, then we talk about data collection — not before.
The data problem nobody mentions
Fine-tuning needs hundreds to thousands of high-quality examples. “We have years of tickets” usually means years of inconsistent formatting, PII you cannot use, and labels that disagree depending on who closed the shift.
Cleaning that data often takes longer than the fine-tune itself. And every time product changes, you re-label or accept drift. RAG updates when you re-index a doc. A fine-tuned model updates when you retrain — with all the pipeline cost that implies.
Forge sequencing we recommend
Ship the agent, measure, then decide. Teams that start with fine-tuning usually restart with RAG three weeks later — after discovering their “proprietary knowledge” was actually a moving target in Confluence.
When fine-tuning is the right call, we treat it as a Forge phase-two optimisation: same eval harness, same golden set, compare base model + RAG against fine-tuned variant on cost, latency, and quality. The winner is whoever passes evals at the price point the product can carry — not whoever sounds best in a side-by-side demo.