Putting It Together

Fifteen chapters, fifteen mechanisms. Not one of them tells you what to build.

What they give you is a price list. Attention tells you what a long prompt costs. The KV cache tells you how many users fit on a box. The agent loop tells you what a twentieth step does to your success rate. Every chapter converts a design choice into a number you can compute before you write any code.

This chapter spends them, in order, on one system.

The system

An enterprise support assistant. It answers questions from private documentation, looks up the current state of a customer’s account, opens tickets, escalates what it cannot resolve, and cites its sources. Several tenants share one deployment. Three-second p95.

You have met its bill twice. Chapter 1 costed its simplest ancestor at $1,200 a month. Chapter 5 cut that to $660 with prompt caching. Chapter 15 sized the full agent at $4,850, then at $2,850. This chapter is the reasoning that produces the thing Chapter 15 priced.

Eight questions. The order matters more than any single answer, because each one narrows what the next is allowed to be.

1. Does this need a model at all?

Ask first, because it is free to ask and expensive to skip.

Sort the request surface before designing for it. “Reset my password” is a routing problem — a classifier and a link. “What is my balance” is a database query with a sentence wrapped round it. “Why was I charged twice in March, and does your refund policy cover it” is the one that genuinely needs a model: it requires reading a policy, reading an account, and reasoning about both together.

Chapter 10 put this on a ladder from L0 to L4, with the warning that most problems solved at L2 should have been L0 or L1. That applies to the product, not only to agents. A fixed path is cheaper, faster, testable, and debuggable. You trade all of it for flexibility, and you should only pay when the path truly cannot be known in advance.

Perhaps a third of this system’s traffic never needs a model. Routing it away is the largest single cost reduction available here, and it happens before the other seven questions have been asked.

2. Name what you are optimising, and what you will trade

Latency, throughput, availability, scalability, security, maintainability — all still apply, unchanged. AI systems add attributes conventional architecture has no vocabulary for, and each already has a chapter behind it.

The list is not the point. Ranking it is, because several of these oppose each other directly. Faithfulness trades against helpfulness: a system that refuses whenever it is less than certain is perfectly faithful and largely useless. Cost per task trades against success rate: more steps, more retries, and a bigger model all buy accuracy.

For a support assistant the ranking writes itself. Faithfulness first — a confidently invented refund policy is worse than no answer at all. Refusal quality second. Cost per task third. Raw helpfulness last, because escalating to a human is a perfectly good outcome and the cheapest safety net available.

Write that order down. Most of what follows is resolved by it.

3. Private knowledge: retrieval, or just context?

Chapter 6 gave the regime rule, and it deserves an honest answer before anyone builds a pipeline. Under roughly 50,000 tokens of stable content, put it in a cached prompt: no index, no chunking decision, no embedding migration, and Chapter 5’s caching makes it cheap.

This corpus is gigabytes and changes daily, so we are in the other regime. But notice what the question did. It made build RAG a conclusion rather than an assumption, and it named the condition under which the conclusion would flip.

4. Which retrieval shape, and what does it cost in latency?

Chapter 7’s ladder, climbed slowly. Naive RAG is a demo. Advanced RAG — hybrid search and reranking — is the sensible production default. Agentic and graph retrieval are answers to measured failures you have not had yet.

Advanced, then. Which settles two decisions already made for you.

Chunking (Chapter 8) is the one nothing downstream can undo. This corpus is prose with tables in it, so: recursive splitting into roughly 300-token children grouped under 1,500-token parents, index the children and return the parents, never split a table from its header, and contextual prefixes — because a chunk reading “this does not apply to enterprise customers” is worse than useless alone.

Two-stage retrieval (Chapter 9) closes gaps 1 and 2 together. Hybrid dense and sparse is not optional here: error codes and product identifiers are precisely what dense retrieval cannot see. Fuse by RRF, then rerank the top 100 with a cross-encoder.

Now spend the latency budget, because three seconds is a constraint rather than a hope.

input guardrail       80 ms
classify and route   150 ms
hybrid retrieval     120 ms   dense + sparse
rerank 100           400 ms   ~4 ms each (Ch 9)
prefill ~5,300 tok   300 ms
                   ---------
TTFT               ~1,050 ms

5. Workflow or agent?

Chapter 10’s arithmetic decides this and it is unforgiving. At 95% per step, four steps succeed 81% of the time and twenty steps succeed 36%.

So: as little agent as the task will admit.

Answering a documentation question is a pipeline, not a loop — retrieve, rerank, generate, check the citations. Looking up an account is one tool call. Creating a ticket is a fixed sequence: validate, create, confirm. All L0 and L1, all testable, all debuggable.

The loop earns its place in exactly one branch: the question that needs several lookups whose shape depends on what the earlier ones returned. Bound it at four steps — what Chapter 15 priced, and what the arithmetic tolerates — and make hitting the cap an escalation rather than an apology.

6. Where are the boundaries and the gates?

Three questions, three different answers, all from Part III and Chapter 14.

What the system can do, and who decides. Tools sit behind the capability boundary of Chapter 11. The tenant comes from the authenticated session and never from an argument the model filled in. The refund tool has a ceiling, and the ceiling lives in the tool rather than in the prompt, because a prompt instruction is a suggestion and a credential is a guarantee.

What needs a human. Chapter 10’s test is reversibility, not importance. Reading documentation, reading an account, searching: let them run. Issuing a refund, closing an account, mailing a customer: gate them. Put the gate on the tool so every path that ever reaches it inherits the gate.

What happens when the content is hostile. Support tickets are attacker-writable text and this system reads them. Chapter 14 is blunt that no prompt fixes this. Separate the trust domains: whatever reads ticket content does not hold the credential that can send mail. Guardrail the input, the actions, and the output, because each catches something the others do not.

7. How will you know it works?

Chapter 13: the eval is the product, and it has to exist before the system does.

Score retrieval and generation separately — they fail independently and look identical from outside. Context recall and precision say whether the right chunks arrived; faithfulness and answer relevancy say what the model did with them. High recall with low faithfulness is a generation problem. The reverse is a retrieval problem. One number for “quality” tells you neither.

Put correctly refused and correctly escalated in the eval set as passes. Step 2 ranked refusal quality second, and a metric that punishes declining will quietly train the system out of the behaviour you asked for.

Then trace everything, sampled, retaining every failure. When faithfulness drops next quarter, the trace is the difference between ten minutes and a week.

8. What does it cost?

Last, because only now is there something to cost.

Chapter 15 did this arithmetic in full: a 1,500-token cached prefix, 3,000 tokens of retrieved chunks, 800 tokens of history, 400 out, four steps, 10% retries, 50,000 tasks a month. $4,850, falling to $2,850 once the prefix is cached and retrieval is cut from ten chunks to five.

Two things are worth noticing about where that number came from.

It is the same number, reached from the opposite direction. Chapter 15 started from a proposal and priced it. This chapter started from a requirement and derived it. That they meet is the point of the whole book: the mechanisms are constraints, and constraints do not care which end you enter from.

And the largest saving is not in the list. Question 1 routed roughly a third of traffic away from the model entirely, before any of this arithmetic applied. The cheapest token is still the one you never send.

The system those eight questions produced

Two things in that picture are worth more than the boxes.

The line that goes around the model. Question 1 sent roughly a third of traffic down the deterministic path, where it never becomes a token. That branch is the largest cost decision in the diagram, and it was settled before any of the other seven questions were asked.

The trace underneath. It is not a stage. It is the record every stage writes to, and Chapter 13’s evals, Chapter 15’s cost attribution, and Chapter 14’s incident review all read the same rows. Systems that add it last never quite get it, because you cannot trace requests that have already been served.

What is absent is just as decided. There is no agent loop on the interactive path — the latency napkin ruled it out, because a loop streams nothing until it stops. There is no memory tier beyond the session, because nothing in the requirements asked the assistant to remember a customer between conversations, and Chapter 12 is blunt that a durable store you did not need is a privacy surface you now own.

The sequence, and the trades inside it

The shapes, and where each one is taught

Eight shapes cover most of what you will build. None is new — seeing them together is how you notice which one you are reaching for, and what it charges.

Most production systems are two or three of these composed rather than one. The support assistant is classify-and-route wrapping a RAG pipeline, with a bounded agent on a single branch and an approval gate on a single tool.

Every step above is a trade this book has already priced. Collected in one place, so you can argue about them before you build rather than after:

There is no column that is always right. There is only the ranking you wrote down in question 2, and what it says about each row.

What to carry forward

The sequence is the deliverable, not the architecture it produced. Ours suits a support assistant with those requirements and that ranking; change the ranking and the same eight questions produce a different system.

Ask them cheapest first. Most designs are decided by questions 1 and 2, and teams reach them last — after the framework is chosen, the pipeline is built, and the constraint that would have ruled it out has become expensive to discover.

That is the whole method. The fifteen mechanisms give you the prices. This chapter gives you the order.

The end, and the beginning

Fifteen ideas: tokens, embeddings, attention, the generation loop, the KV cache, the context budget, the retrieval gap, chunking, two-stage retrieval, the agent loop, tools, memory, evaluation, failure, and cost. Then one method for spending them.

They are the load-bearing ones — the concepts that will still be true when the model names on today’s benchmark have been replaced twice over. Everything else in this field is a specific, dated instance of one of them.

The guide this book was distilled from has the specifics: the vendors, the frameworks, the current numbers, the case studies. Go there when you need today’s answer. Come back here when you need to know why it is the answer.