Interview Q&A

They solve different problems.

• RAG gives the model knowledge at question time: it retrieves relevant documents and puts them in the prompt. Use it when facts change often, when answers must cite sources, or when data is private and per-user.
• Fine-tuning changes the model's behaviour: tone, output format, a domain's vocabulary, or a narrow task it should do reliably. Use it when prompting alone can't get consistent behaviour.

A good rule of thumb: RAG for what the model should know, fine-tuning for how it should act. They also combine well, for example a model fine-tuned to always answer in a strict JSON format, fed with retrieved context.

Both control how the next token is sampled.

• Temperature rescales the probabilities before sampling. Low values (around 0–0.3) make the most likely tokens even more likely, giving focused, repeatable output. High values flatten the distribution, giving more varied and creative output.
• Top-p (nucleus sampling) keeps only the smallest set of tokens whose probabilities add up to p, then samples from those. With top_p = 0.9, very unlikely tokens are never picked.

For extraction, classification or code, use low temperature. Usually you tune one of the two, not both.

Evaluate retrieval and generation separately, because they fail differently.

Retrieval

• Context recall: did we retrieve everything needed to answer?
• Context precision: how much of what we retrieved was actually relevant?

Generation

• Faithfulness / groundedness: is every claim in the answer supported by the retrieved context?
• Answer relevance: does the answer actually address the question?

In practice: build a small golden set of real questions with expected answers and source pages, score it automatically (frameworks like Ragas compute these metrics, often with an LLM as judge), and spot-check by hand. Track the scores on every change to chunking, prompts or models.

When a transformer generates text one token at a time, each new token attends to every previous token. Without caching, the keys and values for the whole prefix would be recomputed at every step.

The KV cache stores the key and value tensors for tokens already processed, so each new step only computes them for the newest token. This makes generation much faster.

The cost is memory: the cache grows with sequence length × layers × heads × batch size. That's why long context windows are expensive to serve, and why techniques like grouped-query attention, paged attention and KV-cache quantization exist. It's also the idea behind prompt caching: reusing the cached prefix across requests that share the same system prompt.

Attention computes softmax(QKᵀ / √d_k) V.

The dot product of two vectors with d_k dimensions, whose components have zero mean and unit variance, has a variance of about d_k. For large d_k the scores become large in magnitude, so the softmax saturates: one token gets almost all the weight and the gradients become tiny.

Dividing by √d_k brings the variance back to about 1, which keeps the softmax in a range where it still learns well.

LoRA (Low-Rank Adaptation) fine-tunes a model without updating its original weights. For selected weight matrices W, it learns a small update ΔW = B·A, where A and B are thin matrices of rank r (often 8–64).

Why it's popular:

• Far fewer trainable parameters, often well under 1% of the model, so it fits on modest GPUs.
• Small adapters: you can keep one base model and swap adapters per task or customer.
• No extra latency once the adapter is merged back into the weights.

QLoRA goes further by keeping the frozen base model in 4-bit precision while training the LoRA adapters.

A model generates the most plausible continuation, not a verified fact. Hallucinations rise when the answer isn't in its training data or context, when the prompt pressures it to answer anyway, or when retrieval returns the wrong context.

Ways to reduce them:

1. Ground it: retrieve the right context, and instruct the model to answer only from it.
2. Allow "I don't know": say explicitly that refusing is better than guessing.
3. Require citations: ask for source references, then check them in code.
4. Constrain the output: structured output (JSON schemas) for anything a program will consume.
5. Verify: a second pass or rules that check numbers and claims against the source.
6. Measure: track faithfulness on a test set so you notice regressions.

Quantization stores weights (and sometimes activations) in fewer bits, for example 8-bit or 4-bit integers instead of 16-bit floats.

Gains: a much smaller memory footprint (4-bit weights are about a quarter of FP16), so bigger models fit on smaller GPUs or devices, and faster inference because generation is usually limited by memory bandwidth.

Costs: some loss of accuracy, which grows as precision drops. Methods like GPTQ and AWQ calibrate the quantization on sample data to keep the most important weights accurate. Always re-run your own evals after quantizing, especially for maths, code and long-context tasks.

Think in layers, before and after the model:

1. Input guardrails: detect prompt injection and jailbreaks, block out-of-scope topics, and redact PII before it reaches a third-party model.
2. Context controls: retrieve only documents the signed-in user is allowed to see, and treat retrieved text and tool output as untrusted.
3. Action controls: the model proposes actions; deterministic code checks permissions, limits and confirmations before anything executes. Never let the model move money directly.
4. Output guardrails: check for leaked PII, unsupported financial advice and unsupported claims; enforce the response format.
5. Observability: log prompts, retrieved context and decisions; run red-team tests and evals before every release.

Tools like NeMo Guardrails or an LLM gateway help, but the key idea is that safety lives in code around the model, not only in the prompt.

There's no universal number; it depends on the documents and the questions.

• Too small: each chunk lacks context, and answers get split across chunks.
• Too large: chunks mix several topics, similarity gets diluted, and you waste context window.

A practical approach: start around a few hundred tokens with some overlap, split on the document's natural structure (headings, paragraphs, table boundaries) rather than a fixed character count, and attach metadata like the section title and page number to every chunk. Then evaluate several settings on a golden question set and keep the one with the best retrieval recall. For long structured documents, consider a hierarchical or tree-based index instead of flat chunks.