Redshift

`COPY` from S3. Merge is emulated.

Every destination receives the same layout; see the tables Ingest creates.

Setup

Redshift is the one destination here that needs object storage as well as a database user. Its COPY cannot read a local file, so Ingest writes Parquet to a staging prefix in your S3 bucket and Redshift loads from there. That is Redshift's requirement, not a house style; every other warehouse here takes one credential and nothing else.

1. A database user

CREATE USER ingest PASSWORD 'a-long-random-password';
GRANT CREATE ON DATABASE dev TO ingest;

2. A staging prefix

Any prefix in a bucket in the same region as the workgroup, for example s3://your-bucket/ingest-staging. Ingest writes parts there and cleans up after each load.

3. An IAM role Redshift assumes to read it

Redshift reads the staged files as itself, not as Ingest, so the permission lives on a role attached to your cluster or workgroup. AWS's own requirement for COPY from S3 is permission to LIST the bucket and GET the objects being loaded:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": "arn:aws:s3:::your-bucket" },
    { "Effect": "Allow", "Action": ["s3:GetObject"], "Resource": "arn:aws:s3:::your-bucket/ingest-staging/*" }
  ]
}

Trust redshift.amazonaws.com to assume it, attach it to the workgroup or cluster, and, if your user is not a superuser, GRANT ASSUMEROLE ON 'arn:aws:iam::…:role/…' TO ingest FOR COPY;.

S3ServiceException: Access Denied on a load is this role, not the database user.

What to paste

Endpoint, Database, Username, Password, S3 staging prefix, and the IAM role ARN from step 3.

Then test it

Test connection creates a scratch schema and a table, writes a few rows through the same load path a real run uses, reads them back to check the count is exactly right, round-trips pipeline state, and drops the lot. Because it stages to S3 and runs a real COPY, it exercises step 3 as well: an Access Denied here is the IAM role, not the database user.

Staging

needs_staging: s3. Staging always happens in the destination's own cloud and region; data does not take a detour through ours.

Conformance

Every adapter passes the same suite before it ships: create a table with every canonical type, append 100k rows, atomic replace, merge (latest dedup_sort wins; child tables delete-then-insert by root id), schema evolution, NULL-key rejection, package replay without duplicates, state and schema round-trip, row-count read-back, identifier edge cases, double-underscored table names (__quarantine, child tables), and rollback on a mid-load failure.