# Ingest

> Managed pipelines from any API (REST, GraphQL, SOAP, gRPC, WebSocket, files and SQL) into your
> warehouse or lake. WebSocket sources are bounded batch windows, not real-time delivery.

## Guides

- [The tables Ingest creates](https://ingestdata.ai/docs/guides/tables): What lands in your warehouse (data tables, child tables, quarantine, the three _ingest_ control tables) and the semantics of each.

## API kinds

- [File](https://ingestdata.ai/docs/api-kinds/file): Buckets and local paths via fsspec: `csv`, `jsonl`, `json`, `parquet`, `xlsx`.
- [GraphQL](https://ingestdata.ai/docs/api-kinds/graphql): One document per resource, POSTed as `{query, operationName, variables}`.
- [gRPC](https://ingestdata.ai/docs/api-kinds/grpc): A channel per resource, TLS by default, with auth carried in metadata.
- [REST](https://ingestdata.ai/docs/api-kinds/rest): Ingest's original engine. Only `{method,url,headers,params,data,json,timeout,allow_redirects,ssl,proxy}` are forwarded to the transport.
- [SOAP](https://ingestdata.ai/docs/api-kinds/soap): Envelope templates rendered with XML-escaped parameters, parsed with `lxml`.
- [SQL](https://ingestdata.ai/docs/api-kinds/sql): Databases read through DB-API or ADBC with chunked fetch.
- [WebSocket](https://ingestdata.ai/docs/api-kinds/websocket): A bounded batch window over a stream, not real-time delivery.

## Destinations

- [Amazon Athena](https://ingestdata.ai/docs/destinations/athena): Plain Parquet on S3 with Glue external tables: append and replace only. Opt into Iceberg tables through Glue to enable merge.
- [BigQuery](https://ingestdata.ai/docs/destinations/bigquery): Load jobs straight from the worker. Native `MERGE`. No bucket to provision.
- [ClickHouse](https://ingestdata.ai/docs/destinations/clickhouse): Native insert. Merge uses `ReplacingMergeTree` keyed by `dedup_sort`.
- [Databricks](https://ingestdata.ai/docs/destinations/databricks): `COPY` from cloud storage, then `MERGE`.
- [DuckDB](https://ingestdata.ai/docs/destinations/duckdb): The local oracle. Native merge, transactional, no staging.
- [Azure Blob Storage](https://ingestdata.ai/docs/destinations/filesystem_azure): Parquet files in your Azure container, authorised by a connection string.
- [Google Cloud Storage](https://ingestdata.ai/docs/destinations/filesystem_gcs): Parquet files in your own GCS bucket. Ingest holds no credential: your runs authenticate with the AWS identity your workspace already has.
- [Apache Iceberg](https://ingestdata.ai/docs/destinations/filesystem_iceberg): Iceberg tables in your own REST catalog; data files in your S3, GCS or Azure bucket. The URI scheme picks the cloud. The only lake option that can merge.
- [Amazon S3](https://ingestdata.ai/docs/destinations/filesystem_s3): Parquet files in your S3 bucket, written under a role you own and can revoke.
- [Generic SQL](https://ingestdata.ai/docs/destinations/generic_sql): Any DB-API or SQLAlchemy dialect: MySQL, SQL Server, SQLite. Batched INSERT with emulated merge.
- [MotherDuck](https://ingestdata.ai/docs/destinations/motherduck): DuckDB, hosted. Same adapter, same semantics.
- [SQL Server](https://ingestdata.ai/docs/destinations/mssql): BULK load, then `MERGE`.
- [Postgres](https://ingestdata.ai/docs/destinations/postgres): The conformance oracle. `ON CONFLICT` merge, transactional.
- [Redshift](https://ingestdata.ai/docs/destinations/redshift): `COPY` from S3. Merge is emulated.
- [Snowflake](https://ingestdata.ai/docs/destinations/snowflake): Tables in your own Snowflake database, loaded through a stage the dataset owns. Ingest holds no credential: your runs authenticate with the AWS identity your workspace already has.

## Connectors

- [asana](https://ingestdata.ai/docs/connectors/asana): rest · productivity · beta
- [austintexas_311](https://ingestdata.ai/docs/connectors/austintexas_311): rest · public_data · beta
- [batonrouge_311](https://ingestdata.ai/docs/connectors/batonrouge_311): rest · public_data · beta
- [berkeley_311](https://ingestdata.ai/docs/connectors/berkeley_311): rest · public_data · beta
- [bloomingtonin_311](https://ingestdata.ai/docs/connectors/bloomingtonin_311): rest · public_data · beta
- [boston_311](https://ingestdata.ai/docs/connectors/boston_311): rest · civic · beta
- [buffalony_311](https://ingestdata.ai/docs/connectors/buffalony_311): rest · public_data · beta
- [cambridge_311](https://ingestdata.ai/docs/connectors/cambridge_311): rest · public_data · beta
- [cincinnati_311](https://ingestdata.ai/docs/connectors/cincinnati_311): rest · public_data · beta
- [cityofchicago_311](https://ingestdata.ai/docs/connectors/cityofchicago_311): rest · public_data · stable
- [cityofnewyork_311](https://ingestdata.ai/docs/connectors/cityofnewyork_311): rest · public_data · stable
- [dallas_311](https://ingestdata.ai/docs/connectors/dallas_311): rest · public_data · beta
- [encinitas_311](https://ingestdata.ai/docs/connectors/encinitas_311): rest · public_data · beta
- [honolulu_311](https://ingestdata.ai/docs/connectors/honolulu_311): rest · public_data · beta
- [kcmo_311](https://ingestdata.ai/docs/connectors/kcmo_311): rest · public_data · beta
- [lacity_311](https://ingestdata.ai/docs/connectors/lacity_311): rest · public_data · beta
- [montgomerycountymd_311](https://ingestdata.ai/docs/connectors/montgomerycountymd_311): rest · public_data · beta
- [nola_311](https://ingestdata.ai/docs/connectors/nola_311): rest · public_data · beta
- [norfolk_311](https://ingestdata.ai/docs/connectors/norfolk_311): rest · public_data · beta
- [oakland_311](https://ingestdata.ai/docs/connectors/oakland_311): rest · public_data · beta
- [openbrewerydb](https://ingestdata.ai/docs/connectors/openbrewerydb): rest · food_and_drink · stable
- [rickandmortyapi](https://ingestdata.ai/docs/connectors/rickandmortyapi): rest · entertainment · stable
- [sanfrancisco_311](https://ingestdata.ai/docs/connectors/sanfrancisco_311): rest · public_data · beta
- [sanmateocounty_311](https://ingestdata.ai/docs/connectors/sanmateocounty_311): rest · public_data · beta
- [seattle_311](https://ingestdata.ai/docs/connectors/seattle_311): rest · public_data · beta
- [somervillema_311](https://ingestdata.ai/docs/connectors/somervillema_311): rest · public_data · stable