LLM (Large Language Model)
A large language model is a neural network trained on vast text corpora to predict tokens, capable of general-purpose language tasks via prompting. Examples include the Claude, GPT, Gemini, and Llama families. In LLM Ops the model is usually consumed through an API rather than hosted in-house.
RAG (Retrieval-Augmented Generation)
RAG is the pattern of fetching relevant external passages and inserting them into the prompt so the model answers from authoritative data rather than memory. It is the dominant approach for grounded question answering over private or current knowledge.
ANN (Approximate Nearest Neighbour)
ANN search finds vectors close to a query vector without scanning the entire index, trading a small accuracy loss for large speed gains. Algorithms such as HNSW and IVF make vector search scale to millions of embeddings.
HNSW (Hierarchical Navigable Small World)
HNSW is a graph-based ANN index that organises vectors into navigable layers, giving fast, high-recall similarity search. It is the default vector index in pgvector, Qdrant, Weaviate, and Redis vector search.
BM25 (Best Matching 25)
BM25 is a classic lexical ranking function that scores documents by term frequency and inverse document frequency with length normalisation. It is the keyword half of hybrid search and excels at exact-term and rare-token matches that embeddings miss.
TTFT (Time To First Token)
TTFT is the latency from sending a request to receiving the first generated token. It dominates perceived responsiveness in streaming UIs and is reduced by prompt caching, shorter prompts, and smaller or faster models.
TPM / RPM (Tokens / Requests Per Minute)
TPM and RPM are the rate limits providers enforce on an API key, capping tokens and requests per minute respectively. Production systems must handle 429 responses with backoff and may need request queuing or higher tiers to stay within limits.
MCP (Model Context Protocol)
MCP is an open standard from Anthropic that lets applications expose tools, resources, and prompts to LLM clients through one interface. It replaces bespoke per-integration glue with a common discovery-and-invocation protocol.
SFT (Supervised Fine-Tuning)
SFT trains a base model on labelled input-output pairs to specialise it for a task or format. It is the most common fine-tuning method and is typically followed by preference optimisation for alignment.
RLHF (Reinforcement Learning from Human Feedback)
RLHF aligns a model with human preferences by training a reward model on ranked outputs and optimising the policy against it. It is how base models are turned into helpful, harmless assistants.
DPO (Direct Preference Optimization)
DPO aligns a model directly from preference pairs without training a separate reward model or running reinforcement learning, making alignment simpler and more stable. It has become a popular alternative to RLHF for preference tuning.
LoRA (Low-Rank Adaptation)
LoRA is a parameter-efficient fine-tuning method that freezes the base weights and trains small low-rank adapter matrices, drastically cutting compute and storage. Many task-specific adapters can be swapped over one shared base model.
PEFT (Parameter-Efficient Fine-Tuning)
PEFT is the umbrella term for techniques such as LoRA, QLoRA, and prefix tuning that adapt a model by training a tiny fraction of parameters. It makes fine-tuning feasible on modest hardware and budgets.
RAGAS (Retrieval-Augmented Generation Assessment)
RAGAS is an open-source framework for evaluating RAG pipelines with reference-free metrics such as faithfulness, answer relevancy, and context recall. It is widely used to gate RAG quality in CI.
LLM-as-Judge
LLM-as-judge denotes using a model to grade or compare other models' outputs against a rubric, scaling evaluation beyond manual labelling. Bias controls include a different judge model and randomised option order.
CoT (Chain-of-Thought)
CoT is prompting that elicits explicit step-by-step reasoning before the final answer, improving multi-step and arithmetic tasks. It raises token cost and should be hidden from users when reasoning is sensitive.
ReAct (Reason + Act)
ReAct is an agent pattern that interleaves reasoning traces with tool actions and observations in a loop until the task is solved. It underpins many tool-using agents by letting the model plan, act, and revise.
NER (Named Entity Recognition)
NER is the task of identifying and classifying spans of text such as people, organisations, dates, and amounts. LLMs perform it via structured extraction prompts, and labelled NER data is common in fine-tuning sets.
JSON Mode
JSON mode is a model feature that constrains output to valid JSON, often against a supplied schema, so responses can be parsed reliably by downstream code. It is the basis of structured output and should still be validated programmatically.
p50 / p95 / p99 Latency
These are percentile latencies: p50 is the median request time, while p95 and p99 capture the tail that worst-affected users experience. SLOs are usually set on p95 or p99 because averages hide slow outliers.
SLO / SLA (Service Level Objective / Agreement)
An SLO is an internal target for a reliability metric (e.g. 99% of requests under 3s); an SLA is the externally promised version with consequences. LLM services set SLOs on latency, availability, and sometimes quality metrics.
TCO (Total Cost of Ownership)
TCO is the full cost of running an LLM feature: input and output tokens, embeddings, vector store, re-ranking, infrastructure, and human evaluation. Token cost alone understates it, so model routing and caching target the whole picture.
RRF (Reciprocal Rank Fusion)
RRF combines several ranked lists by summing reciprocal-rank scores per item, producing one fused ranking without score calibration. It is the standard, robust way to merge lexical and vector results in hybrid search.
kNN (k-Nearest Neighbours)
kNN retrieval returns the k vectors closest to a query under a distance metric, exactly (brute force) for small corpora or approximately (ANN) at scale. The k value sets how many candidate passages enter the context.
OOD (Out Of Distribution)
OOD inputs differ markedly from what a model or eval set represents, where behaviour is least predictable. Detecting and routing OOD queries to a fallback or human is a guardrail against confident wrong answers.
PII (Personally Identifiable Information)
PII is data that can identify an individual, such as names, emails, or national IDs. LLM pipelines must detect and redact PII before logging, sending to third-party APIs, or storing in vector indexes to meet privacy obligations.
DSPy
DSPy is a framework for programming (rather than hand-writing) LLM pipelines, where prompts and few-shot examples are optimised automatically against a metric. It treats prompting as a compilable, tunable program.
Eval Set / Golden Set
An eval set (golden set) is the fixed collection of inputs and reference answers used to measure system quality release over release. Treating it as immutable - not editing answers to pass - is what keeps the measurement honest.
Context Stuffing
Context stuffing is the anti-pattern of cramming as many retrieved passages as possible into the prompt, which raises cost and latency and buries the relevant evidence among noise. Re-ranking and tighter top-k beat stuffing for both quality and price.