What is different
A classic web system moves rows and bytes. An AI system moves tokens through a model that runs on a GPU. Three things follow from that.
- Each request is expensive. A database query costs a fraction of a cent. A model call costs tokens, and tokens cost money on every request.
- The answer streams. The model produces one token at a time. The user sees the first word long before the last one.
- The answer can be wrong. The model does not know your data, and it can make things up. The system around it has to supply the facts and check the output.
One request, step by step
- Prompt. The user's message, plus the system instructions, plus the chat history. All of it is tokens, and all of it is paid for.
- Retrieve. The system looks up documents the model should see: the user's orders, the product manual, yesterday's chat. This is the RAG step, covered below.
- Prefill. The GPU reads every token of the prompt in one pass. A long prompt makes this step slow.
- Decode. The GPU writes the answer one token at a time. Each token takes about the same time as the last.
- Stream. Each token goes to the user as it is written, over a long-lived HTTP response or a WebSocket.
The three numbers
Interviewers grade the numbers. Say these three out loud, with a value for each, before you draw anything else.
1. Time to first token
Time to first token (TTFT) is how long the user waits before the first word appears. It is retrieval plus prefill plus the queue in front of the GPU. Time between tokens is how fast the rest of the answer arrives. Keep the two apart. A chat feels fast when the first token comes in under half a second, even if the full answer takes ten seconds.
- Cache the retrieval result for a repeated question
- Trim the prompt: fewer history turns, shorter instructions
- Put a small model in front for easy requests
- Keep a GPU warm; a cold start costs seconds
- Batch requests on the GPU, but a bigger batch slows each one
- Use a smaller or quantized model
- Cap the answer length
2. GPU capacity for each tenant
A GPU serves a batch of requests at once. More requests in the batch means more throughput and more wait for each request. The design question is how many GPUs you need, and how you stop one customer from taking all of them.
- Capacity. Requests per second, times tokens per request, divided by tokens per second per GPU. State your assumptions and do the division out loud.
- A queue in front. When the GPUs are full, requests wait in a queue with a timeout, instead of failing at once.
- A limit per tenant. Each customer gets a share of the batch, by tokens per minute, not by request count. One tenant with huge prompts cannot starve the others.
- Priority. Paid users ahead of free users, interactive chat ahead of batch jobs.
3. Token cost per month
Cost is users, times requests for each user, times tokens for each request, times the price for each token. Write the multiplication on the board. Then name the levers that make it smaller.
- A cache. Many prompts repeat. Answer a repeated prompt from a cache and skip the model.
- A small model first. Route easy requests to a cheap model. Send only the hard ones to the big one.
- Shorter prompts. Summarize old chat history instead of sending all of it.
- A limit for free users. A daily token budget for each free account.
RAG: give the model the facts
Retrieval-augmented generation (RAG) means the system finds the right documents and puts them in the prompt. The model then answers from those documents instead of from memory.
- Chunks. Documents are cut into pieces of a few hundred tokens. Too small and a chunk loses its meaning. Too big and the prompt fills with text the model does not need.
- Embeddings. Each chunk becomes a vector. Similar text gives similar vectors, so a question finds chunks about the same thing.
- The index. A vector database finds the nearest chunks fast. It is a cache of meaning, and like any cache it goes stale. Say how documents get re-indexed when they change.
- Rerank. A second, slower model orders the top candidates. This is where quality comes from.
Agents: the model takes actions
An agent is a loop. The model reads the task, picks a tool, the system runs the tool, and the result goes back into the prompt. The loop ends when the model says it is done, or when the system stops it.
- A step limit and a timeout. The loop must end. Name the maximum number of turns.
- Tools that are safe to repeat. The model can call a tool twice by mistake. "Look up an order" is safe to repeat. "Refund an order" is not, so it needs an idempotency key.
- Permissions. The agent gets the user's rights, and no more. A tool that writes needs a confirmation step.
- A record of each turn. When an agent does something odd, you need the log to see why.
Evals: how you know it works
A classic system has tests that pass or fail. A model's answers are not that simple. An eval is a set of questions with known good answers, and a score for how close the system gets. Job postings for AI engineers name evals more often than RAG. Interviewers ask about them.
- Offline. A fixed set of questions, run before each change. A score that drops blocks the release.
- Online. Sample real answers and grade them, with a model as the grader or with people. Watch the score over time.
- Guardrails. Checks on the input (is this request allowed?) and the output (did the model leak a secret, or give medical advice?). These run on every request and must be fast.
- A canary. A new model or prompt goes to a small share of traffic first. Compare the scores, then roll it out.
What to say in the interview
- State the scale: users, requests a day, tokens for each request.
- Give the latency budget: time to first token, time between tokens.
- Draw the request path: prompt, retrieval, prefill, decode, stream.
- Do the GPU division. Add the queue and the limit for each tenant.
- Do the cost multiplication. Name two levers.
- Say how you would know the answers are good: the eval set and the canary.