Ingest documentation
From any API to tables you can query
Ingest authors the connector, runs the pipeline and lands typed, deduplicated tables in a destination you own. These pages cover what it connects, what it writes, and how to set each destination up.
Start here
Data sources
Every source: the connectors certified against a real API today, and the vendors Ingest builds on request.
The tables Ingest creates
What lands in your warehouse (data tables, child tables, quarantine, the three _ingest_ control tables) and the semantics of each.
Machine-readable index
The same contents as plain text, for AI crawlers and agents.
Connectors
asanarest · productivity · beta
austintexas_311rest · public_data · beta
batonrouge_311rest · public_data · beta
berkeley_311rest · public_data · beta
bloomingtonin_311rest · public_data · beta
boston_311rest · civic · beta
buffalony_311rest · public_data · beta
cambridge_311rest · public_data · beta
cincinnati_311rest · public_data · beta
cityofchicago_311rest · public_data · stable
cityofnewyork_311rest · public_data · stable
dallas_311rest · public_data · beta
encinitas_311rest · public_data · beta
honolulu_311rest · public_data · beta
kcmo_311rest · public_data · beta
lacity_311rest · public_data · beta
montgomerycountymd_311rest · public_data · beta
nola_311rest · public_data · beta
norfolk_311rest · public_data · beta
oakland_311rest · public_data · beta
openbrewerydbrest · food_and_drink · stable
rickandmortyapirest · entertainment · stable
sanfrancisco_311rest · public_data · beta
sanmateocounty_311rest · public_data · beta
seattle_311rest · public_data · beta
somervillema_311rest · public_data · stable
26 certified, and 979 more vendors Ingest builds on request. Browse them all.
API kinds
File
Buckets and local paths via fsspec: `csv`, `jsonl`, `json`, `parquet`, `xlsx`.
GraphQL
One document per resource, POSTed as `{query, operationName, variables}`.
gRPC
A channel per resource, TLS by default, with auth carried in metadata.
REST
Ingest's original engine. Only `{method,url,headers,params,data,json,timeout,allow_redirects,ssl,proxy}` are forwarded to the transport.
SOAP
Envelope templates rendered with XML-escaped parameters, parsed with `lxml`.
SQL
Databases read through DB-API or ADBC with chunked fetch.
WebSocket
A bounded batch window over a stream, not real-time delivery.
Destinations
Amazon AthenaPlain Parquet on S3 with Glue external tables: append and replace only. Opt into Iceberg tables through Glue to enable merge.
BigQuery
Load jobs straight from the worker. Native `MERGE`. No bucket to provision.
ClickHouse
Native insert. Merge uses `ReplacingMergeTree` keyed by `dedup_sort`.
Databricks
`COPY` from cloud storage, then `MERGE`.
DuckDB
The local oracle. Native merge, transactional, no staging.
ABAzure Blob Storage
Parquet files in your Azure container, authorised by a connection string.
Google Cloud Storage
Parquet files in your own GCS bucket. Ingest holds no credential: your runs authenticate with the AWS identity your workspace already has.
Apache Iceberg
Iceberg tables in your own REST catalog; data files in your S3, GCS or Azure bucket. The URI scheme picks the cloud. The only lake option that can merge.
Amazon S3
Parquet files in your S3 bucket, written under a role you own and can revoke.
Generic SQL
Any DB-API or SQLAlchemy dialect: MySQL, SQL Server, SQLite. Batched INSERT with emulated merge.
MotherDuckDuckDB, hosted. Same adapter, same semantics.
SQL ServerBULK load, then `MERGE`.
Postgres
The conformance oracle. `ON CONFLICT` merge, transactional.
Redshift`COPY` from S3. Merge is emulated.
Snowflake
Tables in your own Snowflake database, loaded through a stage the dataset owns. Ingest holds no credential: your runs authenticate with the AWS identity your workspace already has.