Apache Iceberg with Snowflake: Open Table Formats Without the Hype

Why Everyone Is Suddenly Talking About Table Formats

For most of the last decade, putting data into Snowflake meant handing it over: Snowflake stored it in its own format on its own storage, and you read it through a Snowflake warehouse. That was a good deal for us when I was CTO of a subscription e-commerce business: one platform, one bill, very little to operate.

Open table formats change that deal. With Apache Iceberg, the data lives as Parquet files in a bucket you own, described by metadata that any compatible engine can understand. Snowflake becomes one reader and writer among several, alongside Spark, Trino, Athena, DuckDB and whatever comes next.

In recent consulting work, "should we move to Iceberg?" is one of the most common data questions I get. The honest answer is usually "not yet, but design so that you could".

What an Open Table Format Actually Is

A data lake of Parquet files on S3 is just files: no idea which files make up the current table, no transactions, and no safe way for two writers to change it at once.

A table format adds a metadata layer on top of the files that turns them into a proper table. Iceberg's design is a tree:

Catalog  →  points to the current metadata file for each table
  metadata.json      schema, partition spec, snapshot list, properties
    manifest list    one per snapshot, lists the manifests in it
      manifests      list data files, with per-column stats and partition values
        data files   Parquet (usually), plus delete files for row-level changes

A few properties fall out of that structure, and they are the reason the format matters.

Snapshots and atomic commits

Every write produces a new snapshot and a new metadata file. The commit is a single atomic swap of the catalog pointer from the old metadata file to the new one. Readers either see the old table or the new table, never half of a write. Concurrent writers use optimistic concurrency: if someone else committed first, you retry against the new state.

Time travel

Because old snapshots are kept until you expire them, you can query the table as it was at a point in time or at a specific snapshot ID. This is the same idea as Snowflake's native Time Travel, just implemented in open metadata rather than inside Snowflake.

Schema evolution

Iceberg tracks columns by ID, not by name or position. Renaming a column, adding one, or reordering them is a metadata-only change that does not rewrite data files and does not silently mis-map old files.

Partition evolution and hidden partitioning

You define partitions as transforms on columns, such as days(order_ts) or bucket(16, customer_id), and queries filter on the original column. Iceberg works out which files to skip. You can change the partition spec later (from daily to hourly, say) without rewriting history, because each data file records which spec it was written with.

File-level statistics

Manifests carry min/max values and null counts per column per file, so engines can prune files before opening them. This is what makes Iceberg queries fast enough to be practical on object storage.

Iceberg, Delta Lake and Hudi

There are three serious open table formats. They solve the same core problem and differ in heritage and ecosystem.

If Snowflake is your main engine, Iceberg is the natural choice because it is the format Snowflake supports natively. The format wars have also cooled: Delta's UniForm writes Iceberg-compatible metadata and Apache XTable translates between formats. Pick Iceberg for a Snowflake estate and do not lose sleep over it.

How Snowflake Supports Iceberg

Snowflake's Iceberg tables store data and metadata in your own cloud storage, not Snowflake's. The key decision is who owns the catalog, because the catalog is the thing that says which metadata file is current, and therefore who is allowed to write.

External volumes: storage you own

Whichever catalog you use, you start with an external volume: a Snowflake object that points at an S3 bucket (or GCS or Azure container) and an IAM role Snowflake can assume. Roughly:

CREATE EXTERNAL VOLUME lake_vol
  STORAGE_LOCATIONS = (
    (
      NAME = 'eu-west-2-lake'
      STORAGE_PROVIDER = 'S3'
      STORAGE_BASE_URL = 's3://acme-data-lake/iceberg/'
      STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/snowflake-iceberg'
    )
  );

You then describe the volume to get Snowflake's IAM user and external ID for the role's trust policy, as with a storage integration.

Snowflake-managed Iceberg tables

Here Snowflake is the catalog. It writes the Parquet and the Iceberg metadata to your bucket, handles commits, and you get nearly the full Snowflake experience: DML, clustering, Time Travel, governance policies, streams and tasks, dynamic tables in many cases.

CREATE ICEBERG TABLE analytics.orders (
  order_id     NUMBER,
  customer_id  NUMBER,
  order_ts     TIMESTAMP_NTZ,
  total_gbp    NUMBER(10,2)
)
  CATALOG = 'SNOWFLAKE'
  EXTERNAL_VOLUME = 'lake_vol'
  BASE_LOCATION = 'analytics/orders/';

Other engines read these tables through Snowflake's Iceberg REST catalog endpoint (Snowflake has been exposing its catalog this way via Horizon) or, more crudely, by pointing at the metadata files directly. Writes from outside Snowflake are the thing to check carefully against current documentation: the safe assumption is that Snowflake is the writer and everything else reads.

Externally managed Iceberg tables

Here another catalog owns the table: AWS Glue Data Catalog, or an Iceberg REST catalog such as Apache Polaris (Snowflake Open Catalog is the managed version of Polaris), Unity Catalog, or similar. Snowflake connects through a catalog integration:

-- Illustrative: a Glue catalog integration
CREATE CATALOG INTEGRATION glue_cat
  CATALOG_SOURCE = GLUE
  TABLE_FORMAT = ICEBERG
  GLUE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/snowflake-glue'
  GLUE_CATALOG_ID = '123456789012'
  CATALOG_NAMESPACE = 'events'
  ENABLED = TRUE;
 
CREATE ICEBERG TABLE raw.clickstream
  EXTERNAL_VOLUME = 'lake_vol'
  CATALOG = 'glue_cat'
  CATALOG_TABLE_NAME = 'clickstream';

For a REST catalog the integration uses CATALOG_SOURCE = ICEBERG_REST with an endpoint and credentials instead. Snowflake has also added catalog-linked databases, which mirror a whole namespace from a remote catalog rather than registering tables one by one, and write support to externally managed tables has been arriving in stages. The exact feature matrix moves every quarter, so treat any blog post (this one included) as a map, not a contract, and check the docs for the specific operations you need.

The practical summary:

What You Gain

Engine choice. The same table can serve Snowflake for BI, Spark for heavy transformation, Athena or Trino for ad hoc queries, and DuckDB on a data scientist's laptop.

One copy of the data. Without Iceberg, using two engines usually means two copies and a pipeline keeping them in sync, with all the reconciliation fun that implies. With Iceberg, the copy is the table.

Less lock-in. If your data is Parquet plus open metadata in your bucket, leaving Snowflake is a compute migration, not a data migration. Even if you never leave, that changes the tone of renewal negotiations.

Storage on your terms. Your own lifecycle rules, encryption keys and access logs. For some regulated workloads that alone settles the question.

What You Give Up

This is the part vendor slides skip.

Some performance and features. Snowflake has spent years tuning its native format. Iceberg tables in Snowflake are close, and getting closer, but you should expect gaps: certain optimisations, some table types and features, and behaviour around very small or very frequent writes. For externally managed tables, the gaps are wider and you also pay latency for metadata refreshes from the remote catalog. Benchmark your own queries before you promise anyone parity.

More operations. Native Snowflake tables have no file layout for you to worry about. Iceberg tables do. Streaming or frequent small writes produce thousands of tiny files and manifests, and query performance degrades until something compacts them. You need a plan for:

• compaction (rewriting small files into larger ones)
• expiring old snapshots so storage does not grow without bound
• removing orphan files left by failed writes
• rewriting manifests when they fragment

On Snowflake-managed tables, Snowflake handles most of this. On externally managed tables, it is your Spark job, your Glue optimiser configuration, or your catalog vendor's feature, and someone has to own it. A rough example of what that looks like in Spark:

-- Spark SQL with Iceberg procedures, run on a schedule
CALL lake.system.rewrite_data_files(table => 'events.clickstream');
CALL lake.system.expire_snapshots(table => 'events.clickstream',
                                  older_than => TIMESTAMP '2026-09-01 00:00:00');
CALL lake.system.remove_orphan_files(table => 'events.clickstream');

Catalog governance. The catalog becomes critical infrastructure. If two catalogs both think they own a table, or someone writes files around the catalog, you get corruption that is genuinely hard to unpick. Access control also splits: Snowflake roles and masking policies apply inside Snowflake, but an engineer with S3 read on the bucket can read the Parquet directly and bypass all of it. Your governance model has to cover the bucket, the catalog and every engine, not just Snowflake.

Split cost visibility. Native Snowflake storage is one line on one invoice. With Iceberg, storage and request costs land on your AWS bill, compute lands on Snowflake's bill and maybe EMR's, and cross-region or cross-account access can add data transfer charges. None of it is necessarily more expensive, but it is harder to see, and costs you cannot see tend to grow. I covered the tagging discipline this needs in AWS cost optimisation for a mid-size business.

When Iceberg Is Worth It

For a mid-size business with one data team, one warehouse and a dbt project, the answer today is often not yet. If Snowflake is your only engine, Iceberg mainly adds moving parts to protect you from a lock-in you are not currently feeling. Snowflake's native tables are excellent at exactly that shape of workload, and your data team's time is better spent on modelling, quality and data contracts than on compaction schedules.

It becomes worth it when one or more of these is true:

• You genuinely run, or will soon run, a second engine on the same data (Spark for ML feature pipelines, Trino for a product team, Athena for engineers).
• You have large volumes of raw event or log data that is mostly cold, where paying warehouse-grade storage and loading costs makes little sense.
• Regulatory or contractual requirements say the data must live in storage you control.
• You are already copying data between platforms and paying for the pipelines and reconciliation.
• You want a credible exit option ahead of a major contract renewal.

A short decision checklist

Answer these honestly before committing:

1. Can you name a second engine and a real workload that will read these tables in the next twelve months?
2. Who owns compaction, snapshot expiry and orphan file clean-up, by name?
3. Which catalog is the single source of truth for each table, and is that written down?
4. How will access control work for someone with direct S3 access?
5. Have you benchmarked your ten most expensive queries on Iceberg against native tables?
6. Can finance see the combined Snowflake and S3 cost of a table, per domain?
7. Is there a rollback plan if performance or reliability disappoints?

If you cannot answer the first question, stop there. If you cannot answer two to four, you are not ready.

A Pragmatic Migration Path

When the answer is yes, I would not do a big bang. The path that works is incremental and starts where the risk is lowest.

Step 1: Start with raw, append-heavy data

Raw event streams and CDC change logs are the best first candidates: large, mostly appended, and nobody's board pack depends on them directly. If you already land CDC into S3 (as in several of the patterns in RDS Postgres to Snowflake), writing that landing layer as Iceberg is a small change with a clear payoff.

Step 2: Choose Snowflake-managed first unless you have a reason not to

If Snowflake is your main engine, Snowflake-managed Iceberg tables give you open storage with the least operational burden. You still get other engines reading through the catalog. Move to an external catalog such as Polaris or Glue only when a non-Snowflake engine needs to write, because that is the moment catalog ownership really matters.

Step 3: Keep curated models native for now

dbt marts, the tables BI tools hit hundreds of times a day, can stay as native Snowflake tables until you have evidence Iceberg performs well enough for them. A mixed estate is fine. In dbt, the table format is a model configuration, so moving a model later is a change of config plus a rebuild, not a rewrite.

Step 4: Run in parallel and compare

Run native and Iceberg versions side by side for a few weeks. Compare row counts, key metrics, query times and combined cost across both bills. Switch consumers only when they match.

Step 5: Automate the housekeeping before you scale

Before the tenth table, not after it, put compaction, snapshot expiry and monitoring (file counts per partition, average file size, snapshot age) on a schedule with alerts. Small-file problems creep in slowly and then all at once.

Step 6: Add engines one at a time

Bring in the second engine for one read-only workload through the catalog and learn what breaks. Only then consider letting it write.

How This Fits With Ingestion

Very frequent micro-batches are where Iceberg's small-file problem bites hardest. If you ingest in seconds (see Snowpipe Streaming), check how compaction keeps up: a table that is fast to write and slow to read is not a win.

The Takeaway

Apache Iceberg is the right long-term direction for analytical storage: open files, open metadata, any engine. Snowflake's support for it is now genuinely good, especially for Snowflake-managed tables in your own bucket. But open storage is not free storage. You take on catalog governance, file maintenance and a split cost picture in exchange for flexibility. If you only have one engine, keep your native tables, keep your raw data in open formats on S3 where it is cheap to do so, and make sure nothing in your design would stop you switching later. Adopt Iceberg table by table when a second engine or a real storage requirement turns up, not because it is the year's favourite conference topic.