Apache Iceberg at Spice AI: Query, Accelerate, and Write to Open Table Formats | Spice AI

Apache Iceberg at Spice AI: How we Query, Accelerate, and Write to Open Table Formats

Phillip LeBlanc
Co-Founder and CTO of Spice AI
February 25, 2026

TL;DR: Spice integrates Apache Iceberg as a first-class data source -- connect to any Iceberg catalog, query tables with full SQL semantics, selectively accelerate hot datasets for sub-millisecond reads, and write back with ACID guarantees. This post covers catalog integration, query acceleration, write support, DataFusion internals, and production lessons from running Iceberg at scale.

Apache Iceberg has become the default open table format for production data lakes, bringing ACID transactions, schema evolution, and time travel to data stored in object storage. But Iceberg alone doesn't solve the performance problem: querying data in object storage like S3 carries inherent latency that's too high for many application workloads.

At Spice AI, we integrate Iceberg as a first-class data source - customers can connect to Iceberg catalogs and query tables with full SQL semantics, then selectively accelerate hot datasets for sub-millisecond reads. This post explains how that integration works, from connecting your first catalog to the query engine internals.

What this post covers:

What is Apache Iceberg?

Apache Iceberg is an open table format - a specification for how to organize metadata and data files so that data stored in object storage (S3, GCS, HDFS) can support database-level properties like ACID transactions, schema evolution, and time travel. It's not a storage system or a query engine, but a layer that sits between the two.

Unlike the older Hive approach, which treats directories as tables and relies on naming conventions for partitioning, Iceberg maintains explicit metadata files that track:

These capabilities are organized into three layers:

Iceberg Catalog (REST, Glue, Hive, Hadoop) Table Metadata (Schema, Partitions, Snapshots) Data Files (Parquet in object storage, e.g. S3)

This metadata-driven design is what gives Iceberg its core strengths. We'll return to specific features - ACID transactions, hidden partitioning, schema evolution, time travel, and file-level pruning - later in this post, in the context of how Spice uses them.

The latency gap: Why Iceberg alone isn't enough for applications

Iceberg solves the thorny data lake consistency and reliability challenges, but there's still an inherent performance gap for latency-sensitive workloads. Object storage like S3 carries 50-200ms per-request latency by design, and resolving an Iceberg query requires multiple metadata round-trips (manifest list, manifests, then data files) before touching a single row. Parquet itself is optimized for bulk analytical scans, not the high-concurrency, low-cardinality reads that application and increasingly AI agent workloads demand.

Consider a concrete example: a 10TB Iceberg table of user events, partitioned by date, stored in S3. Your data team runs ad-hoc queries over years of data - Iceberg's partition pruning makes this efficient. But your application serves a dashboard that queries the last 7 days of data, thousands of times per second.

The common workaround is to copy hot data into a faster system (ClickHouse, Redis, etc.) and introduce an ETL pipeline to keep it in sync with the lake. This works, but it introduces drift between systems, duplicates storage costs, and adds another operational surface to maintain.

This is the gap Spice is designed to fill.

How Spice.ai bridges the gap

Spice.ai is a data compute engine that lets you query data across sources using standard SQL. It connects to databases, data lakes, and warehouses - including Iceberg catalogs - and presents them through a single query interface. The key capability for Iceberg workloads is acceleration: Spice can materialize a hot working set into a faster local engine (DuckDB, SQLite, or Spice Cayenne), then keep it automatically synchronized with the source. Iceberg stays the authoritative source of truth; Spice handles the caching and refresh logic so you don't have to build a separate pipeline.

Spice is built on Apache DataFusion for query planning and execution, and on Apache Arrow as the in-memory columnar format. We'll cover the DataFusion internals later in this post; for now, the important thing is that Spice treats Iceberg as a first-class catalog - connect once, query across data sources, then choose which datasets to accelerate.

Connecting Iceberg to Spice

Configuration in Spice uses the spicepod - a YAML file that declares which data sources to connect and how to access them. To connect an Iceberg catalog, add a catalogs entry pointing to the catalog's REST endpoint (or Glue, or Hadoop):

catalogs:
  - from: iceberg:http://localhost:8181/v1/namespaces
    name: ice
    params:
      iceberg_s3_endpoint: http://localhost:9000
      iceberg_s3_access_key_id: admin
      iceberg_s3_secret_access_key: password

On startup, Spice connects to the catalog, discovers all namespaces (databases) and tables, and registers them for SQL access:

spice run
# 2025-01-27T19:08:37Z  INFO Registered catalog 'ice' with 1 schema and 8 tables

Every table is now queryable - no additional configuration needed:

sql> SHOW TABLES;

+--------+-------+------+
| table_catalog | table_schema | table_name |
+--------+-------+------+
| ice           | tpch_sf1     | lineitem   |
| ice           | tpch_sf1     | nation     |
| ice           | tpch_sf1     | orders     |
| ice           | tpch_sf1     | customer   |
+--------+-------+------+

sql> SELECT COUNT(*) FROM ice.tpch_sf1.lineitem;

+------+
| count(*)  |
+------+
| 6001215   |
+------+
Time: 0.186233833 seconds

Tables registered this way go directly to the Iceberg source. This is the simplest setup, but it means query latency depends on the source (S3, network, file format, etc.). For low-latency use cases, the next step is acceleration.

Accelerating Iceberg tables

Tables registered through a catalog are queryable but not accelerated. To accelerate a table, register it as a dataset in the spicepod. This gives you control over the acceleration engine, refresh schedule, and which subset of data to cache:

datasets:
  - from: ice.analytics.events
    name: events
    acceleration:
      enabled: true
      engine: cayenne
      refresh_sql: |
        SELECT * FROM events
        WHERE event_date > NOW() - INTERVAL '7 days'
      refresh_check_interval: 10m

The from: field references the catalog table (ice.analytics.events), so the Iceberg catalog still serves as the source of truth. The dataset definition adds the acceleration layer on top.

What happens behind the scenes:

  1. On startup, Spice executes the refresh_sql against the Iceberg table
  2. Results are loaded into a local acceleration engine (Spice supports DuckDB, SQLite, Arrow, and Spice Cayenne for queries with low memory and DuckDB-like scale)
  3. Queries against events hit the acceleration engine (local NVMe storage)
  4. Every 10 minutes, Spice re-executes the refresh SQL to pull new data

You don't have to choose between Iceberg and fast queries. Iceberg stays your source of truth. Acceleration adds a transparent cache for predictable, low-latency reads.

Writing back to Iceberg

Spice supports INSERT operations with full ACID guarantees via Iceberg's transaction protocol (check out the Iceberg cookbook for a full example). To enable writes, set access: read_write on the catalog:

catalogs:
  - from: iceberg:http://localhost:8181/v1/namespaces
    access: read_write  #required to enable INSERT operations
    name: ice
    params:
      iceberg_s3_endpoint: http://localhost:9000
      iceberg_s3_access_key_id: admin
      iceberg_s3_secret_access_key: password
      iceberg_s3_region: us-east-1

- Insert new rows
INSERT INTO ice.tpch_sf1.region (r_regionkey, r_name, r_comment)
VALUES (5, 'ANTARCTICA', 'A cold and remote region');

- Verify
SELECT * FROM ice.tpch_sf1.region WHERE r_regionkey = 5;
SELECT * FROM ice.tpch_sf1.nation WHERE n_nationkey = 25;

How this works under the hood:

  1. Spice writes new Parquet files to S3
  2. Creates a new manifest file listing the new files
  3. Creates a new snapshot referencing the updated manifest
  4. Atomically updates the metadata pointer in the catalog

If another writer commits between steps 1-3, Spice's commit fails and retries. This is Iceberg's optimistic concurrency in action - every write creates a new snapshot, and the catalog uses atomic compare-and-swap operations to ensure only one writer wins. The loser retries from the latest snapshot.

Under the hood: How Spice queries Iceberg

Now that we've covered the user-facing configuration, let's look at how the query engine processes an Iceberg query. This section introduces the internal components - if you're primarily interested in using Spice with Iceberg, the sections above have you covered.

Apache DataFusion: Spice's query engine

Spice is built on Apache DataFusion, an extensible SQL query engine written in Rust. DataFusion handles SQL parsing, query planning, and execution. (We covered DataFusion in depth in the first post in this series.)

DataFusion organizes data sources into a three-level hierarchy: Catalog -> Schema -> Table. This maps directly to Iceberg's own structure:

Iceberg namespaces can technically be nested (catalog.a.b.c.table), but most catalog implementations use a single level.

When you add an Iceberg catalog to the spicepod, Spice creates three types of DataFusion components to handle it:

How a query flows through the system

When you run SELECT * FROM ice.db.events, here's how Spice resolves it:

User Query SELECT * FROM iceberg.db.tableName
Resolution / Table Lookup iceberg.db.table → IcebergTableProvider
IcebergTableProvider (iceberg-datafusion crate) Reads metadata, plans which Parquet files to read for the scan
Parquet Files (S3, GCS, HDFS, Local FS)

DataFusion parses the three-part table name (ice.db.table), resolves ice to the Iceberg CatalogProvider, db to the SchemaProvider for that namespace, and table to an IcebergTableProvider. The TableProvider then uses Iceberg's metadata to plan the most efficient read from S3.

Schema discovery: Loading tables from a namespace

Each Iceberg namespace becomes a DataFusion SchemaProvider. On initialization, the provider:

  1. Lists all tables in the namespace from the catalog
  2. Filters them against glob patterns if an inclusion list is configured
  3. Loads each table concurrently

Performance note: Loading table metadata is the slow path - each table requires fetching metadata from S3 (metadata.json, manifest lists). For catalogs with hundreds of tables, this can take 10-30 seconds on startup. Use glob filtering to scope the inclusion list to only the tables a given deployment needs. This keeps startup time predictable regardless of catalog size.

Query optimization through Iceberg metadata

Spice uses the community iceberg-datafusion crate to connect Iceberg tables to DataFusion. When a query hits an Iceberg table, the TableProvider uses Iceberg's metadata to minimize the data read from S3:

This means a query like SELECT customer_id FROM events WHERE event_date = '2025-01-15' might read 50MB instead of 10TB - partition pruning eliminates irrelevant files, and column projection skips every column except customer_id.

Key Iceberg features Spice leverages

The query optimization described above depends on several foundational Iceberg features. Here's a closer look at each.

ACID transactions via optimistic concurrency

Every write creates a new metadata file pointing to a new snapshot. The catalog uses atomic compare-and-swap operations to commit changes. If two writers conflict, one fails and retries:

Read snapshot S1 Read snapshot S1 Add files, commit snapshot S2 Commit successful Add files, try to commit snapshot S3 Rejected (S2 already committed) Retry from S2 Writer A Catalog Writer B

This comes with a tradeoff - too many concurrent changes by multiple writers will result in a lot of wasted work as rejected writes retry. But the benefit is that writers can work independently, with the catalog serving as the coordination point.

Hidden partitioning

Iceberg maintains a separate partition spec that maps logical columns to physical partition values via transforms. Unlike Hive-style partitioning, users never query partition columns directly - partitioning is always defined as a transform on existing columns.

For example, consider a table:

events (
    event_id   BIGINT,
    user_id    BIGINT,
    event_time DATE
)

With the partition spec:

PARTITION SPEC
    day(event_time)

Iceberg physically stores a derived partition value such as event_time_day = 2024-01-15, but this field is not part of the table schema. When a user writes WHERE event_time = '2024-01-15', Iceberg rewrites the predicate automatically using the partition transform and prunes data files accordingly.

Changing the partitioning strategy only requires updating the spec. Old data remains partitioned under the old spec; new data uses the new one. Iceberg's metadata tracks which data files belong to which spec, and during scan planning it evaluates predicates across all active specs.

Schema evolution without rewrites

Iceberg tracks schema evolution in metadata; each data file records which schema version it was written with. Readers handle differences automatically:

Time travel

Every commit creates an immutable snapshot. You can query as of any snapshot ID or timestamp. The tradeoff is that many small changes create many snapshots, increasing query planning time.

File-level pruning via metadata statistics

Each manifest file stores min/max values for every column in every data file. Before scanning any Parquet, Iceberg can eliminate files that cannot contain matching data:

WHERE event_date > '2024-01-01' AND user_id = 12345

event_date max < 2024-01-01 user_id out of range Predicate matches Read Manifests Check Statistics File matches predicate? Skip file Scan file

Parquet files from modern writers also include these statistics in the Parquet footer - but Iceberg's metadata-level statistics enable pruning without opening the files at all.

Write predicates that Iceberg can optimize:

- Good: Iceberg can prune partitions
SELECT * FROM events WHERE event_date > '2024-01-01'

- Less optimal: Function prevents pushdown
SELECT * FROM events WHERE YEAR(event_date) = 2024

Error handling: Making failures actionable

Iceberg operations can fail in many ways - network issues, permission errors, corrupted metadata, or missing files. Generic error messages make these hard to debug. Spice maps Iceberg errors to specific messages that tell you what went wrong and what to do about it.

For example, if you have a typo in a namespace name:

Generic Iceberg error:

Unexpected => Failed to execute http request, NoSuchNamespaceException

Spice error:

The namespace 'analytics_prod' does not exist in the Iceberg catalog, verify the namespace name and try again.

Or if the catalog URL is wrong:

Generic Iceberg error:

Unexpected => Failed to execute http request, source: error sending request for url (http://localhoster:8181/v1/config)

Spice error:

Failed to connect to the Iceberg catalog or object store at (http://localhoster:8181/v1/config). Verify the Iceberg catalog is accessible and try again.

The same approach applies across TLS certificate errors, unsupported features, and invalid table metadata - each error includes context about the cause and a suggested next step.

Production lessons

After running Iceberg in production, here are some lessons learned:

1. Catalog discovery is a startup bottleneck

Listing all namespaces and tables requires many metadata requests. Each table load fetches metadata from S3, and for catalogs with hundreds of tables, startup can take 10-30 seconds. We address this with:

2. S3 request signing can expire on large tables

AWS SigV4 signatures have a fixed validity window. When loading a large Iceberg table, Spice may queue hundreds of S3 requests for metadata files, manifests, and Parquet files. If requests wait too long in the queue, AWS rejects them with RequestTimeTooSkewed. This looks like a permissions error, but it's actually a timing issue - the signature expired before the request was sent. Concurrency limiting (point 1) helps here by keeping the request queue from growing too large.

3. Environment credentials can leak into explicit configs

The iceberg-rust library uses OpenDAL for S3 access. By default, OpenDAL loads credentials from multiple sources - explicit properties, environment variables, and ~/.aws/config. Even when you provide credentials explicitly in the spicepod, OpenDAL can pick up AWS_SESSION_TOKEN from the environment and mix it with your explicit access key. This causes authentication failures that are hard to trace because the credentials you configured are correct - it's the extra session token that's wrong. Spice now sets s3.disable-config-load=true to prevent this. If you hit unexpected auth errors with Iceberg, check whether AWS credentials are set in your environment.

4. Not every table in a catalog is an Iceberg table

When connecting to AWS Glue, the catalog may contain Hive, Parquet, or CSV tables alongside Iceberg tables. Loading a non-Iceberg table as Iceberg fails. Spice handles this by skipping non-Iceberg tables with a warning instead of failing the entire catalog. If you see "Failed to load table" warnings on startup, this is likely why - it's not an error, it's Spice filtering out tables it can't read as Iceberg.

Our contributions to iceberg-rust

Spice depends on iceberg-rust for catalog access, metadata handling, and Parquet scanning. We maintain a fork for changes that haven't been upstreamed yet, and contribute fixes back to the upstream project as they stabilize. Merged upstream to apache/iceberg-rust:

Current limitations and future work

What works today:

Current limitations:

What's next:

Apache Iceberg + Spice AI: Summary

Apache Iceberg provides ACID transactions, schema evolution, and open file formats for data lakes. Spice adds federated SQL access with local acceleration on top. Together, they give you:

Run Iceberg locally in under 5 minutes using the Spice .ai Iceberg Catalog Connector.

Frequently Asked Questions

What is Apache Iceberg?

Apache Iceberg is an open table format that brings ACID transactions, schema evolution, hidden partitioning, and time travel to data stored in object storage like S3, GCS, or HDFS. It organizes metadata and Parquet data files so that data lakes can support database-level reliability without requiring a specific storage system or query engine.

How does Spice AI integrate with Apache Iceberg?

Spice connects to Iceberg catalogs (REST, AWS Glue, or Hadoop) and registers every table for SQL access automatically. Once connected, you can query Iceberg tables alongside other data sources like PostgreSQL or Snowflake through a single SQL interface.

Can Spice AI accelerate Iceberg tables for low-latency queries?

Yes. Spice can materialize frequently queried subsets of Iceberg tables into a local acceleration engine (DuckDB, SQLite, or Spice Cayenne) with configurable refresh intervals. This delivers sub-millisecond read performance while Iceberg remains the authoritative source of truth.

Does Spice AI support writes to Iceberg tables?

Yes. Spice supports INSERT operations with full ACID guarantees via Iceberg's optimistic concurrency protocol. New Parquet files are written to object storage, and the catalog metadata pointer is updated atomically. See Writing to Apache Iceberg Tables with SQL for a detailed walkthrough.

What Iceberg catalog types does Spice support?

Spice supports REST catalogs, AWS Glue, and Hadoop catalogs. Connect by adding a catalogs entry to your spicepod configuration file with the catalog endpoint and credentials.

Get Started with Apache Iceberg + Spice AI

Experience federated SQL, acceleration, and ACID-compliant writes with Apache Iceberg and Spice AI. Follow the Iceberg Catalog Connector guide to run locally in minutes.