Reading AI Engineering out of order
Chip Huyen's AI Engineering is a doorstop worth lifting. I'd read it out of order, evaluation first, and save finetuning for a rainy day.
Some technical books end up living on a desk as a kind of moral reminder. They’re heavy, they’re good, and you’ve read the first forty pages four times. AI Engineering by Chip Huyen is at risk of becoming that book for a lot of product engineers, which would be a pity, because it’s the most complete thing I’ve found on building applications on top of models you didn’t train yourself.
It came out in December, runs past 500 pages, and covers the whole stack in ten chapters. There’s an opening on what AI engineering even is, a chapter on how foundation models get trained and sampled, two on evaluation, then prompting, retrieval and agents, finetuning, dataset engineering, inference optimization, and a closing chapter on architecture and user feedback. Running through all of it is a sort of patience. Start with a prompt. Add retrieval when the model doesn’t know enough. Reach for finetuning only once the cheap tricks stop helping. Measure every step, or you won’t know whether the step did anything.
If I were reading it with a team and trying to keep everyone from stalling around chapter 7 (everyone stalls around chapter 7), I’d shuffle the deck.
Evaluation would go first, chapters 3 and 4. They sit in the middle of the book physically and, I’d argue, spiritually too. Chapter 3 is about why judging open ended output is so slippery and what you can do anyway. It walks through the language modeling metrics like perplexity, then exact checks where they’re possible, such as whether generated code actually runs and passes its tests, then similarity against reference answers, then a model acting as judge, then plain side by side comparison. The section on AI as a judge is the one I’d read twice. It’s honest that the judge is itself a model with its own moods: it can be inconsistent, it costs money, it has biases toward certain answers, and changing the judge’s prompt quietly changes your scores. It’s still worth using, the way a kitchen scale is worth using once you’ve checked it against a bag of sugar.
Chapter 4 turns that into something you could actually build on a Tuesday. It covers the criteria you might care about (domain capability, generation quality, how well a model follows instructions, cost and latency), how to pick a model, the build versus buy question, and how much to trust public benchmarks, which is to say cautiously. Then it lays out an evaluation pipeline in steps: evaluate each component of the system, not just the final answer, write down a guideline for what good looks like, and then decide on methods and data. I suspect most teams skip these chapters to get to the prompting, which is a bit like skipping the chapter on thermometers because you’re eager to cook.
Picture a hypothetical team building a feature that summarizes long customer support threads for whoever picks the ticket up next. The tempting path is to write a prompt, read five outputs, nod, and ship. The book’s path is slower and more boring. First you write down what a good summary is for this feature: it names the customer’s actual problem, it mentions anything already tried, it doesn’t invent a resolution, and it fits on a phone screen. That’s the guideline, and writing it usually starts an argument, because three people had three different summaries in their heads. Then you collect a few dozen real threads, including the weird ones (the thread that’s mostly angry capital letters, the one where the customer solves it themselves halfway through). Some criteria you can check with code, like length. Some need a judge model with a rubric. A few need a human, at least at first, to see whether the judge agrees with people. Then every change to the prompt, the model, or the retrieval gets run against the same set. It’s tedious. It also means the next argument about whether the new prompt is better gets settled with a table instead of vibes.
Then chapters 5 and 6, since prompting and retrieval are what most people are doing all day.
Chapter 5 covers the basics you’d expect (zero-shot and few-shot, system prompts versus user prompts, how much context a model can really use) and then some practical habits I didn’t expect a book to bother with, like organizing and versioning prompts as though they were code. They are code, more or less, just code that a compiler won’t check. The section on defensive prompting deserves a slow read from anyone putting a model in front of the public. It goes through people trying to extract your system prompt, jailbreaks, prompt injection, and getting a model to leak information it saw in training or in context, and then what you can do about each at the prompt, model, and system levels. The honest message is that you can reduce these risks a lot and remove them never.
Chapter 6 is half retrieval and half agents. The retrieval half compares term-based search with embedding-based search, and I appreciated that it doesn’t treat the older approach as a fossil. Keyword search is cheap, fast, and often good enough, and plenty of systems combine the two. Then come the unglamorous details that decide whether retrieval works at all, like how you chunk documents, whether you rerank results, whether you rewrite the user’s question before searching. There’s a good argument buried here that a lot of “the model is hallucinating” complaints are really “the retriever handed it the wrong paragraph” complaints, and you can only tell which by evaluating retrieval on its own.
The agents half is a good conceptual base, if a brief one. It covers tools, planning, how agents fail, and how you might evaluate them, plus a short section on memory. The failure modes discussion is the part I’d send people back to, because agent failures compound in a way single calls don’t. A model that picks the wrong tool, or the right tool with the wrong arguments, at step two of ten has quietly poisoned the remaining eight.
And then in chapter 10, you see the application from one model invocation to a mini-city. I really liked the sequence of adding capabilities: more context, then guardrails (in and out), then a router/gateway over multiple models, then caches for latency reduction, then agent patterns. All of that with monitoring and orchestration. And the section about user feedback is one of the cooler bits of the book. In a conversational product, users are constantly giving you feedback on how well you’re doing. And it’s often not even intentional. They rephrase their question, they pause mid-stream, they say “no, the other one” they copy an answer. They give up. You’re getting a wealth of evaluation data from your real users, whether or not you bother with it.
Chapters 7 through 9 can wait on the shelf until a specific problem sends someone looking. Though the finetuning chapter’s discussion of when to finetune is worth reading early, mostly because it talks you out of it so persuasively. It lays out the reasons for and the reasons against, and how finetuning relates to retrieval. My rough summary is that finetuning tends to help with how a model responds and retrieval with what it knows, though she’s more careful about it than that. The rest gets into memory math and parameter-efficient techniques, which most product teams will never need. Chapter 8, on datasets, covers curation, synthetic data, and the humble work of deduplicating and cleaning. Chapter 9 is about making inference faster and cheaper, with a clear explanation of the latency metrics people actually feel, like how long until the first token appears and how fast the rest stream in. I’d hand that chapter to whoever gets asked why the bill doubled.
Chapters 1 and 2 are good background for anyone new to the field. Everyone else can skim them and feel only slightly guilty.
My reservations are mild. A book about a field that changes monthly is going to be a snapshot, and some of the specific model comparisons will age like milk while the principles age more like cheese. Agents get part of one chapter, which was a fair allocation when it was written. Six months on, agents that call tools and run code over many steps are where a lot of the attention has gone, and anyone wanting depth there will need to look elsewhere. (The evaluation chapters apply to agents perfectly well, which may be the more useful gift.) It’s also a book about building AI applications, and a team adopting coding agents to write its software will find less for itself here. Call it scope. It’s still nice to know before you start.
Some sections read like a survey, too. That happens when someone tries to map a whole territory in one go, and I’d rather have the map than not.
What if the only thing that “sank in” for a team was doing one tiny annoying thing: document the idea of what good output should be for one feature. Collect twenty examples. And then try the current version against that before anyone even thinks about looking at a prompt. A lot of the arguing can be turned into comparison. I guess... maybe arguing is the fun part? I don’t know. Or maybe I do. I’m not sure.