# Localhost Latency at Scale: The Spice Cluster-Sidecar Architecture

**TL;DR:** Any application, service, or AI agent that needs high-performance, low-latency access to large-scale operational data faces the same three challenges: low-latency retrieval (SQL, full-text, vector), a safe blast radius so a misbehaving workload can't take down a database or access data it shouldn't see, and enough compute behind it to answer the hard questions. The Spice **cluster-sidecar (hybrid) architecture** does: a lightweight Spice sidecar runs inside each application pod and serves query, search, and LLM inference on `localhost` from a scoped working set while a central Spice cluster handles ingestion, Cayenne acceleration, Ballista-powered distributed execution, hybrid search indexing, and refresh. The application sees one endpoint on `localhost`.

## The problem: applications need fast, safe, distributed access to data  
Any application, service, or AI agent that queries operational data at scale puts pressure on three things at once:

1. **Latency on the retrieval path**. Applications want answers in single-digit milliseconds.
2. **Blast radius**. Giving any workload direct credentials to production tools can exhaust connection pools or touch rows it shouldn't.
3. **Occasional heavy queries**. Most reads are narrow, but heavy queries can stress the system.

## The solution: an application-local data, search, and inference plane  
The Spice cluster-sidecar architecture gives each application a complete data plane on `localhost` and keeps the data systems behind a single, centrally managed tier:
- Application-local sidecars serve the hot path, answering queries from a scoped working set.
- A centralized Spice cluster is the only tier that talks to your data systems.
- Transparent delegation when a query exceeds local working set.

## Architecture at a glance  
Applications only ever talk to their sidecar. The cluster is the only tier with actual data source credentials.

### The sidecar as a sandbox  
The sidecar is the only data-plane surface the application touches, carefully scoped to limit access and control resource usage. 
- Scoped working set, not the whole warehouse.
- No origin credentials in the application.
- Narrow network surface via loopback interface.

## Why split the tiers  
This architectural division is aimed at achieving scalability without overloading production databases. The cluster ingests each dataset once, producing one authoritative materialization that every sidecar gets a consistent view of.

### Sidecars stay lightweight  
A sidecar:
- Starts in seconds.
- Runs on minimal memory.
- Scales 1:1 with application pods without oversaturating the database.

## The cluster tier: Ballista + Spice Cayenne  
The cluster mode uses [Apache Ballista](https://datafusion.apache.org/ballista/) for distributed query execution and [Cayenne](https://spiceai.org/docs/components/data-accelerators/cayenne) for acceleration beyond 1 TB.

## Results caching: the third latency tier  
Spice employs result caching to minimize redundant processing:
1. **Sidecar results cache.**
2. **Sidecar working set.**
3. **Cluster delegation.**

## Engineering decisions  
### 1. Declarative sidecar configuration  
Every sidecar is configured through a `spicepod.yaml`, eliminating drift and special cases.

### 2. Cache coherency  
Sidecars pull data from the cluster on a configurable interval, making the system predictable.

### 3. Resilience through local state  
Sidecars continue serving data if the cluster is unreachable, allowing for graceful handling of connectivity issues.

## When to use it  
- AI agents or data-intensive services needing rapid retrieval.
- Clear boundary between application code and data systems.
- Workloads requiring real-time retrieval and large-scale analytics.

## A concrete example: a multi-tenant agent platform  
A multi-tenant SaaS platform employs AI agents each with a dedicated sidecar to optimize query access and performance while securing sensitive data.

## Getting started  
The hybrid architecture documentation is available at [spiceai.org/docs/deployment/architectures/hybrid](https://spiceai.org/docs/deployment/architectures/hybrid). Utilize [Spice.ai open source](https://github.com/spiceai/spiceai) or the managed [Spice Cloud](https://spiceai.org/docs/deployment/architectures/hosted) for enhanced utility.