Should you use prompting, RAG, or fine-tuning to customize an AI model?

Fine-Tuning vs. RAG vs. Prompting: How to Choose

Start at the cheapest rung and only climb when you must. Prompting shapes behavior with no infrastructure; RAG grounds the model in your data so answers stay current and citable; fine-tuning changes the model's weights to bake in a style, format, or skill. Most teams need prompting plus RAG, and reach for fine-tuning last — for how the model should behave, not what it should know.

These three get pitched as rivals, but they solve different problems and stack cleanly. The useful framing — echoed by Databricks’ Denny Lee on the show — is a ladder: try the cheap thing first, measure, and only climb a rung when the result demands it.

1. Prompting — start here, always

Prompt engineering shapes the model’s behavior through the instruction alone: the wording, the examples, the output format. No infrastructure, instant to iterate, and on modern models it gets you further than people expect. If a better prompt solves it, you’re done — don’t build anything.

2. RAG — when the answer depends on your data

Retrieval-augmented generation fetches relevant documents at answer time and gives them to the model, so responses are grounded in your sources rather than the model’s frozen training data. Weaviate’s Bob van Luijt frames this as the durable pattern for enterprise AI: it keeps answers current, lets you cite where they came from, and is the main lever against hallucination — all without touching the model. If the problem is what the model knows, RAG is usually the answer.

3. Fine-tuning — when behavior must change

Fine-tuning adjusts the model’s weights on your examples. It’s the right tool when you need a consistent style or format, a specialized skill, or lower latency on a narrow task — things prompting can’t reliably pin down. It’s also the most expensive and brittle option: you need quality training data and a pipeline to retrain as things change. Denny Lee’s ladder puts it last for a reason — reach for it for how the model behaves, rarely for what it knows.

The decision in one line

Prompting changes the instruction; RAG changes the context; fine-tuning changes the model. Knowledge that updates → RAG. Behavior that must be consistent → fine-tuning. Everything else → prompting first.

Why it matters

The common failure is starting at the top of the ladder — fine-tuning a model to teach it facts that change weekly, when RAG would have been cheaper and stayed current. Match the technique to the problem, climb only when you’ve measured a real gap, and remember the rungs combine: a fine-tuned model fed by RAG and steered by a good prompt is a normal production setup.