docs / destinations / filesystem_iceberg

Apache Iceberg

Iceberg tables in your own REST catalog; data files in your S3, GCS or Azure bucket. The URI scheme picks the cloud. The only lake option that can merge.

Every destination receives the same layout; see the tables Ingest creates.

Setup

Iceberg tables in your catalog, with the data files in your bucket. This is the only lake option that can merge, which is what makes it the right choice for most connectors: plain Parquet files cannot upsert, so a resource with a merge disposition cannot be deployed to them at all.

Two things have to exist: access to the bucket, granted exactly as for the matching plain-Parquet destination, and an Iceberg REST catalog.

Grant access with a role: no redeploy on either side

The grant works exactly as for Amazon S3: create a role named IngestAccess-… in your account, trust your workspace's Ingest role with your workspace slug as the external id, and paste the ARN. Full trust-policy and permissions JSON is in that guide; the only difference here is what the permissions policy contains.

For an S3-backed warehouse that is the whole S3 half. A GCS- or Azure-backed warehouse authorises the way the matching Parquet destination does (see those guides), with the credential going into the same fields.

The REST catalog

Ingest speaks the Iceberg REST catalog protocol: Polaris, Nessie, Lakekeeper, Tabular, or Glue through its REST endpoint. It does not speak the Hive metastore or the Glue SDK directly; if Glue is your catalog, use the Athena · Glue + Iceberg destination instead, which does.

You need from the catalog:

  • Catalog URI: the REST base URL, e.g. https://catalog.example/api/catalog
  • Catalog warehouse: the warehouse identifier the catalog knows your bucket by, e.g. s3://my-lake/

and a principal in the catalog that may create namespaces and tables. How that principal is created is catalog-specific, and it is separate from the S3 grant above: the catalog authorises metadata, the bucket policy authorises data files, and a setup with one and not the other fails halfway through the first write.

What to paste

Base URI, Iceberg REST catalog URI, Catalog warehouse.

Then test it

Test connection creates a scratch schema and a table, writes a few rows through the same load path a real run uses, reads them back to check the count is exactly right, round-trips pipeline state, and drops the lot. Because it really writes, it exercises BOTH halves (the catalog credential and the bucket policy), so a setup with one and not the other is caught here.

Staging

needs_staging: none. Ingest writes Parquet data files straight into your bucket and commits them to your catalog; there is no intermediate hop and no Ingest-owned store at any point.

Writes

Same adapter as plain object storage with table_format: iceberg, and the difference is not cosmetic: Iceberg is the only lake option here that can merge. A resource with a merge disposition reaching a plain-file destination raises MergeUnsupportedError ("plain files never merge"), so for most connectors this is the lake option that actually works.

Conformance

Every adapter passes the same suite before it ships: create a table with every canonical type, append 100k rows, atomic replace, merge (latest dedup_sort wins; child tables delete-then-insert by root id), schema evolution, NULL-key rejection, package replay without duplicates, state and schema round-trip, row-count read-back, identifier edge cases, double-underscored table names (__quarantine, child tables), and rollback on a mid-load failure.