Compound AI Systems
A compound AI system performs an AI task through multiple interacting components rather than one model call alone. Components can include language or specialist models, retrievers, databases, rules, rankers, verifiers, code executors, and external tools. The defining property is composition around an end-to-end task. The control flow may be fixed, learned, or agent-directed, so a compound system is not automatically an autonomous agent.
Origin and context
The Berkeley AI Research post in February 2024 named a shift from optimizing a single model to designing systems of interacting components. An IBM Research paper later proposed an enterprise blueprint with planners, registries, data sources, agents, and production constraints. Systems researchers subsequently focused on the resource consequences of these workflows, arguing that orchestration and cluster scheduling need to be coordinated rather than optimized independently.
Why it matters
Composition lets a product add current or private data, deterministic checks, specialized tools, and different cost-quality paths without retraining one monolithic model for every change. It also moves reliability to the system level. A strong component can be undermined by poor retrieval, routing, permissions, state handling, or verification, while local metrics may miss failures caused by component interactions.
Example
A support assistant may classify a request, retrieve authorized account and policy data, call a language model, validate the proposed action, and either answer or hand the case to a person. The team evaluates the entire trace, including retrieval and tool failures, instead of reporting only the language model's benchmark score. It also budgets latency and cost across components and tests what happens when one dependency is unavailable.
How it differs
Agentic Workflows
An agentic workflow gives a model or policy some control over selecting steps or tools. A compound AI system is broader: its interactions can be a deterministic pipeline with no autonomous planning. Agentic workflows are one possible control pattern inside a compound system.
Retrieval-Augmented Generation
RAG combines retrieval with generation to supply external evidence. It is a common compound-system pattern, but compound systems can use many other combinations, and a RAG pipeline can itself contain routing, reranking, verification, and tool calls.
Maturity and evidence
Maturity is rated 3. The term has a clear primary definition and independent architectural and systems research. The design space is active rather than standardized: shared methods for end-to-end optimization, tracing, resource allocation, and safety evaluation are still developing.
Limits and open questions
Adding components can improve control but also expands latency, cost, security boundaries, and failure combinations. Components may be optimized against incompatible metrics, and a verifier can share blind spots with the generator it checks. Dynamic routing makes two apparently identical requests follow different paths. Teams need versioned configurations, trace-level evaluation, access controls, dependency fallbacks, and end-to-end tests. The label should describe a real system architecture, not decorate any application that makes two API calls.
Related terms
References
- The Shift from Models to Compound AI SystemsBerkeley AI Research · 2024-02-18 · class B
- A Blueprint Architecture of Compound AI Systems for EnterpriseIBM Research / arXiv · 2024-06-02 · class A
- Towards Resource-Efficient Compound AI SystemsMicrosoft Research, Brown and MIT / arXiv · 2025-01-28 · class A
Last updated: 2026-09-03
This term is also covered in the Skills Atlas as distributed systems skill.
This term is also covered in the Skills Atlas as retrieval augmented generation skill.