Connectors

A connector is the piece of Ingest that knows how to talk to a specific external API. You bring credentials and pick endpoints, the connector handles authentication, pagination, retries, and rate limiting for you.

What a connector handles for you

  • Authentication. API key, Bearer token, Basic, query-param, and OAuth 2.0 (client_credentials, authorization_code with refresh tokens, and JWT-bearer grants). Ingest stores the credential in AWS Secrets Manager and resolves it on every request.
  • Pagination. Each connector declares the pagination shape its provider returns, and the runtime walks the full result set without you writing a loop.
  • Rate limiting. Each connector ships with the provider's published limits. The runtime throttles requests and backs off on 429s.
  • Retries. Transient failures retry with exponential backoff. Fatal auth errors stop the run.
  • Storage. Each endpoint lands in its own Iceberg table, with metadata columns added by the dbt staging layer (request_id, ingested_at, _source_file, _loaded_at).

Endpoints inside a connector

A connector is a bundle of endpoints. Each endpoint syncs into its own table, and you can enable or disable any subset at any time. For APIs with hierarchical data (e.g. projects → tasks), the parent-child relationship is expressed in the dbt request models, which build the child URLs from parent rows.

Available connectors

Don't see what you need?

Tell us what API you need.

Related