What is the difference between an AI agent and a simple LLM call?
A single LLM call takes a prompt and returns one response with no control over what happens next. An agent runs a loop: it decides on an action (usually a tool call), an executor performs it, the result is fed back, and the model decides again, repeating until the goal is met or a stop condition fires. The key distinction, per Anthropic's guidance, is that the model dynamically directs its own process and uses tools, whereas a simple call is one-shot. Agents are more capable but harder to test, slower, and more expensive, so you adopt them only when the task genuinely needs multi-step, model-directed control.
When should I build an agent versus a fixed workflow?
Prefer the simplest thing that works. If you can express the task as a fixed sequence of LLM and code steps (a "workflow"), do that, because predictable paths are cheaper, faster, and far easier to test and debug. Reach for an agentic loop only when the task requires the model to decide the number and order of steps at runtime, for example open-ended research or multi-step debugging where the path cannot be known in advance. Many production systems are mostly deterministic workflows with one or two agentic steps, and that is a feature, not a compromise.
How do I reduce hallucinations in a RAG system?
Ground every factual claim in retrieved evidence and make the model refuse when evidence is missing. Concretely: improve retrieval so the right passages actually reach the context (the generator cannot use what it never sees), instruct the model to answer only from the supplied context, require citations to specific passages, and add a faithfulness eval to catch regressions. Lowering temperature helps marginally, but the dominant levers are retrieval quality and a prompt that explicitly permits "the context does not cover this" rather than forcing an answer.
What is the right chunk size for RAG?
There is no universal number, but a common, effective range is a few hundred tokens (roughly 300-600 characters per chunk in practice), split on natural structure like paragraphs or headings, sometimes with a small overlap. Chunks that are too large dilute relevance and waste context budget; chunks that are too small lose the surrounding meaning a passage needs to be self-contained. The reliable approach is to start in that range, measure retrieval recall and answer faithfulness on your own eval set, and adjust empirically rather than guessing.
Do I need a vector database, or is pgvector enough?
For most applications, pgvector (the PostgreSQL extension) is enough and is often the better choice because it keeps your embeddings beside your relational data, in one system you already operate. Dedicated vector databases like Pinecone, Weaviate, or Qdrant earn their keep at very large scale, with specialised index tuning, or when you need features Postgres lacks. Start with pgvector and an HNSW index; migrate to a dedicated store only when you have a measured reason, not by default.
How should I evaluate an agent or RAG system?
Build a fixed evaluation set of representative questions with known good answers and grounding contexts, then score every change against it as a regression gate in CI. For RAG, measure faithfulness (is the answer entailed by the context), answer relevancy, and context recall, using a framework such as RAGAS, promptfoo, or DeepEval. For agents, also measure tool-call correctness and end-to-end task completion. The discipline of running evals on each change is what turns prompt tinkering into engineering you can trust.
What is prompt injection and how do I defend against it?
Prompt injection is an attack where adversarial instructions hidden in user input or in content the agent retrieves (a web page, a document, an email) hijack the agent into ignoring its real instructions or misusing its tools. The most dangerous form is indirect injection, where the payload lives in fetched data the model treats as trusted. Defence is layered: isolate and clearly label untrusted content, give tools the least privilege they need, require human approval before irreversible actions, and never let retrieved text silently override your system instructions. There is no single fix, so defend in depth.
How many tools should I give an agent?
As few as possible, each clearly distinct and well documented. Large, overlapping tool sets confuse the model about which to call and increase the chance of wrong or malformed calls. Write precise tool descriptions and tight input schemas (enums, required fields, sensible types), because the schema is the contract the model plans against. If you have many capabilities, group or route them rather than exposing dozens of flat options, and test tool-call accuracy directly with evals.
Should I use a multi-agent system?
Usually not as a first move. Multiple specialised agents can help on genuinely decomposable problems, but they multiply cost, latency, and coordination complexity, and they introduce new failure modes around hand-offs and shared state. The standard guidance is to first push a single agent with good tools and good context as far as it will go; adopt multi-agent only when you can show that the single-agent design provably falls short. When you do, keep the orchestration explicit and observable.
How do I control the cost of an agentic application?
Cost compounds across loop iterations and tool calls, so the biggest lever is reducing the number of steps. Use prompt caching for the large static prefix you resend each turn, route easy sub-tasks to smaller cheaper models, cap retrieved context to what is needed, set a maximum iteration count, and track token spend per run. Measuring cost per task on your eval set, alongside quality, lets you trade them deliberately instead of being surprised by the bill.
What is the Model Context Protocol (MCP) and do I need it?
MCP is an open standard from Anthropic for connecting LLM applications to tools, data sources, and prompts through a uniform client-server interface, so you write an integration once and reuse it across MCP-compatible hosts. You do not strictly need it, you can wire tools directly through your model's function-calling API, but MCP reduces bespoke per-tool glue and makes integrations portable. It is most valuable when you have many integrations or want to share tool servers across applications.
How do I give an agent memory across sessions?
Distinguish short-term from long-term memory. Short-term memory is just the conversation history kept in the context window, trimmed or summarised as it grows. Long-term memory requires an external store: persist facts, preferences, or past results (often in a database or vector store) and retrieve the relevant ones into context on each new request. The agent itself is stateless between calls, so any persistence is something you design and own, including what to write, when to retrieve it, and how to keep it from bloating the context.
Why does my agent loop run forever or rack up huge costs?
Almost always a missing or weak stop condition. Every agent loop must be bounded: a maximum number of iterations, a token or cost ceiling, an error threshold, and a clear signal for "final answer reached". Without these, a model that keeps choosing to call tools, or oscillates between two actions, will loop until it exhausts your budget. Always set a hard iteration cap and log each step so you can see where it gets stuck.
Should I fine-tune a model or use RAG and prompting?
Try prompting and RAG first; fine-tune for behaviour, not for facts. RAG is the right tool for injecting knowledge, because it keeps information fresh, citable, and editable without retraining. Fine-tuning shines when you need consistent style, format adherence, or to bake in a narrow skill, but it is costlier, slower to iterate on, and goes stale as your data changes. The common progression is prompt, then RAG, then fine-tune only if a measured gap remains.
What temperature should I use for an agent?
For agents and any task needing consistency, structured output, or reliable tool calls, use a low temperature (often near zero) so behaviour is reproducible and easier to test. Higher temperatures add useful variety for brainstorming or creative writing but make agents flaky and harder to evaluate. Treat temperature as a tunable you set deliberately per task, and remember it interacts with top-p, so changing both at once muddies your results.
How do I make agent tool calls safe to retry?
Make the underlying operations idempotent, so executing a tool call twice has the same effect as once. Because agents loop and retries are common, a non-idempotent action (charging a card, sending an email) can fire multiple times. Pass an idempotency key the executor deduplicates on, design write operations to be safe under replay, and gate truly irreversible actions behind a human approval step. Safe retries are what let an agent recover from transient failures without causing harm.
What is reranking and when is it worth adding?
Reranking is a second retrieval stage that reorders an initial candidate set with a more accurate but more expensive model (typically a cross-encoder), pushing the best passages to the top before they reach the LLM. It is worth adding when first-stage retrieval returns roughly relevant results but the very best ones are not consistently at the top, which hurts answer quality and wastes context budget. The pattern is fast vector retrieval of many candidates, then rerank down to a small, high-precision final set.