docs / destinations / filesystem_gcs

Google Cloud Storage

Parquet files in your GCS bucket, authorised by a service-account key you mint.

Every destination receives the same layout; see the tables Ingest creates.

Setup

Parquet files in a GCS bucket you own, under the prefix in Base URI (gs://your-bucket/warehouse).

A service account and key

  1. In the Google Cloud console: IAM & Admin → Service Accounts → Create service account in the project that owns the bucket. A name like ingest-lake is enough; skip the optional grant step.
  2. Grant it Storage Object Admin (roles/storage.objectAdmin) on the bucket: Cloud Storage → your bucket → Permissions → Grant access, paste the service account's email. Bucket-level, not project-level: Ingest writes, reads back and deletes objects under your prefix, and nothing else.
  3. Open the account → Keys → Add key → Create new key → JSON. The file downloads once.
  4. Paste the whole JSON file into Service-account key on the destination. It is stored as a secret in your own namespace and never returned to the browser.

Then test it

Test connection creates a scratch schema and a table, writes a few rows through the same load path a real run uses, reads them back to check the count is exactly right, round-trips pipeline state, and drops the lot. A 403 here is the bucket grant; a malformed-key error is caught before anything connects, naming the field.

Staging

needs_staging: none. Parts are written straight to your prefix.

Writes

Plain files cannot merge: a resource with a merge disposition is excluded before deploy, with the reason shown. Pick the Iceberg option for connectors that need upserts.

Conformance

Every adapter passes the same suite before it ships: create a table with every canonical type, append 100k rows, atomic replace, merge (latest dedup_sort wins; child tables delete-then-insert by root id), schema evolution, NULL-key rejection, package replay without duplicates, state and schema round-trip, row-count read-back, identifier edge cases, double-underscored table names (__quarantine, child tables), and rollback on a mid-load failure.