Iceberg tables

Ingest lands every endpoint as an Iceberg table in Snowflake (or in your own S3 bucket). Iceberg is an open table format, your data lives in your storage, in a format any modern query engine can read.

Why Iceberg

  • No vendor lock-in. Your data is in open Parquet files with an open metadata layer. If you ever leave Ingest, the data stays where it is, no export, no proprietary format.
  • Snowflake-native. Snowflake reads Iceberg directly via an external volume. You query Ingest tables the same way you query any Snowflake table.
  • Time travel and schema evolution. Iceberg supports snapshots so you can query historical state, and it handles schema changes without rewriting the whole table.

What an Ingest Iceberg table looks like

One table per endpoint. Each row carries the fields the source API returned, plus a small set of metadata columns added by the dbt staging layer:

  • request_id: identifier for the API request that produced this row.
  • ingested_at: timestamp recorded when Ingest fetched the data.
  • _source_file: the S3 file the row was loaded from.
  • _loaded_at: timestamp when the row landed in Snowflake.

Storage location

Iceberg files live in S3. You can use an Ingest-managed bucket or bring your own bucket, the same Iceberg files land in your AWS account, and Snowflake reads them via an external volume.

Querying

Once data is loaded, Iceberg tables are just Snowflake tables, joins, views, dbt models, and BI tools all work the way they would against any other Snowflake table.

Related