docs / destinations / databricks

Databricks

`COPY` from cloud storage, then `MERGE`.

Every destination receives the same layout; see the tables Ingest creates.

Setup

About ten minutes, all in the Databricks UI. Hover any ? for what a thing is and why Ingest needs it.

Warehouse connection details

  • Sidebar → SQL → SQL Warehouses → open the warehouse you want Ingest to use. Ingest runs every statement through one SQL warehouse. Any size works; serverless is fine: the first statement after a cold start just takes a little longer.
  • Connection details tab → copy Server hostname and HTTP path; paste below as Workspace host and HTTP path. The two connection values Ingest's form asks for. The HTTP path is per-warehouse, so copy it from this warehouse, not from an example.

Catalog name

  • Sidebar → Catalog → note the name of the catalog your data should land in (usually main or workspace); paste below as Unity catalog. Databricks addresses every table as catalog.schema.table. You supply only the catalog: Ingest creates a schema per pipeline (named after the pipeline's Dataset field, which defaults to the connector name: a Shopify pipeline into main lands in main.shopify) and a small staging volume beside it.

Service principal

  • Your username → Settings → Identity and access → next to Service principals, Manage → Add service principal → name it ingest. A machine identity, so the pipeline survives any person leaving. Ingest exchanges its secret for short-lived tokens automatically: nothing to rotate on a calendar. Creating one needs a workspace admin.
  • Tick the Databricks SQL access entitlement. The workspace-level permission to use SQL warehouses at all. It is no longer always on by default, and without it every statement is refused no matter what else you grant.
  • On the service principal's page, open the Secrets tab → Generate secret; copy the two values it shows; paste below as Service principal client ID and Service principal OAuth secret. The two values you paste into Ingest. The secret is shown once; if you lose it, generate a new one and delete the old. The client ID is the principal's UUID, also shown as Application ID.

Permissions

  • SQL Warehouses → â‹® next to your warehouse → Permissions → add ingest with Can use. Can use is the lowest level that may run queries; Can view and Can monitor cannot. Every Unity Catalog grant in the world does nothing on a warehouse the principal may not use.
  • In the SQL Editor, run, with your catalog name and the client ID you copied: The whole grant. Ingest creates each pipeline's schema and staging volume itself and, as their owner, holds every privilege inside them: nothing else to enumerate. SQL grants address a service principal by UUID, never by name; to grant by picking ingest from a list instead, use Catalog → your catalog → Permissions. If you own the catalog yourself you can skip this step. Prefer not to hand out CREATE SCHEMA? Pre-create the pipeline's schema and grant USE SCHEMA, CREATE TABLE, MODIFY, SELECT, CREATE VOLUME, READ VOLUME and WRITE VOLUME on it instead.
GRANT USE CATALOG, CREATE SCHEMA ON CATALOG your_catalog TO `your-client-id`;
  • Generating a new secret for this principal later is always safe. Replacing the principal is not: a deleted and re-created service principal is a new identity, and schemas the old one created stay owned by the old one. Runs then fail with does not have USE SCHEMA until you grant each existing pipeline schema to the new principal or drop the schema so Ingest recreates it. The run failure in Ingest names the schema and carries the exact GRANT statement to paste, so nothing here needs memorising. Ownership never transfers to a replacement principal, even one with the same name; only the UUID identifies it.

In Ingest

  • With all five fields filled (Workspace host, HTTP path, Unity catalog, Service principal client ID, Service principal OAuth secret), press Test connection. A worker connects as your new principal, creates a scratch schema, writes rows through the same load path a real run uses, reads the exact count back, round-trips pipeline state, and drops everything. That proves the grants, not just the credential.

Staging

needs_staging: any. Staging always happens in the destination's own cloud and region; data does not take a detour through ours.

Conformance

Every adapter passes the same suite before it ships: create a table with every canonical type, append 100k rows, atomic replace, merge (latest dedup_sort wins; child tables delete-then-insert by root id), schema evolution, NULL-key rejection, package replay without duplicates, state and schema round-trip, row-count read-back, identifier edge cases, double-underscored table names (__quarantine, child tables), and rollback on a mid-load failure.