docs / destinations / filesystem_s3

Parquet files in your S3 bucket, written under a role you own and can revoke.

Every destination receives the same layout; see the tables Ingest creates.

Setup

Parquet files in a bucket you own, laid out under the prefix you give as Base URI (s3://your-bucket/warehouse). Nothing is created for you and nothing is owned by Ingest: the bucket, the lifecycle rules and the bill are yours.

Grant access with a role: no keys to mint or rotate

Ingest never asks for your AWS keys. You create a role in your account that Ingest's identity for your workspace may assume, and you can revoke it at any time by deleting the role; nothing on Ingest's side needs to change, in either direction.

1. In your AWS console: IAM → Roles → Create role → Custom trust policy, and paste, with your workspace's values from the destination page in place of the two placeholders:

   {
     "Version": "2012-10-17",
     "Statement": [{
       "Effect": "Allow",
       "Principal": { "AWS": "<ingest-tenant-role-arn>" },
       "Action": "sts:AssumeRole",
       "Condition": { "StringEquals": { "sts:ExternalId": "<your-workspace-slug>" } }
     }]
   }
   

   The trust names YOUR workspace's Ingest role specifically, not Ingest's whole account, so no other workspace's session can use this role even if the ARN leaks. The external id is your workspace slug; it is not a secret, it is a pin.

2. The role's name must start with IngestAccess-, for example IngestAccess-lake. Ingest's own permission to assume customer roles is scoped to that name, so a differently-named role cannot be assumed no matter how its trust policy reads.

3. Attach a permissions policy scoped to the prefix in your Base URI:

   {
     "Version": "2012-10-17",
     "Statement": [
       { "Effect": "Allow",
         "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload"],
         "Resource": "arn:aws:s3:::your-bucket/warehouse/*" },
       { "Effect": "Allow",
         "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
         "Resource": "arn:aws:s3:::your-bucket",
         "Condition": { "StringLike": { "s3:prefix": ["warehouse/*"] } } }
     ]
   }
   

   If the bucket uses SSE-KMS, add kms:GenerateDataKey and kms:Decrypt on the key: a missing KMS grant is the most common AccessDenied that looks like a bucket-policy problem.

4. Paste the role's ARN into the destination's IAM role field.

S3-compatible stores

MinIO, Cloudflare R2 and friends have no IAM to assume. Leave the role field empty and supply the endpoint and an access key pair instead; those fields are on the destination page.

Then test it

Test connection creates a scratch schema and a table, writes a few rows through the same load path a real run uses, reads them back to check the count is exactly right, round-trips pipeline state, and drops the lot. An AccessDenied here names the role rather than the bucket policy: check the trust policy's principal and external id first, then the permissions policy's prefix.

Staging

needs_staging: none. Parts are written straight to your prefix; there is no intermediate hop and no Ingest-owned store at any point.

Writes

Plain files cannot merge: a resource with a merge disposition is excluded before deploy, with the reason shown. Pick the Iceberg option for connectors that need upserts.

Conformance

Every adapter passes the same suite before it ships: create a table with every canonical type, append 100k rows, atomic replace, merge (latest dedup_sort wins; child tables delete-then-insert by root id), schema evolution, NULL-key rejection, package replay without duplicates, state and schema round-trip, row-count read-back, identifier edge cases, double-underscored table names (__quarantine, child tables), and rollback on a mid-load failure.