S3 - Tangent

The S3 sink uploads objects to S3 with durability via a local write-ahead log (WAL).
Encoding, compression, object sizing, and concurrency are configured via common sink options.

Schema

string

required

Must be set to s3.

string

required

Name of the destination S3 bucket (do not prefix with s3://).

string

AWS region for the bucket (optional).
If omitted, the AWS SDK’s default resolution is used (env vars, instance metadata, etc.).

string

Filesystem path for the sink’s write-ahead log (WAL).
Defaults to /tmp/wal.

int

Maximum age (in seconds) before a WAL file is rotated and flushed to S3.
Defaults to 60.

object

Controls how Tangent serializes records emitted by this source. See Encoding.

object

Controls how Tangent compresses payloads emitted by this source. See Compression.

Example (NDJSON + Zstd)

tangent.yaml

sinks:
  s3_staging:
    type: s3
    bucket_name: my-bucket
    region: us-west-2
    # common options
    encoding: ndjson         # default
    compression:             # default shown explicitly
      type: zstd
      level: 3
    wal_path: /tmp/wal
    max_file_age_seconds: 60

Example (Parquet + Gzip, Arrow schema)

tangent.yaml

sinks:
  s3_parquet:
    type: s3
    bucket_name: analytics-data
    region: us-east-1
    encoding:
      type: parquet
      schema: ./schemas/arrow_schema.json
    compression:
      type: gzip
      level: 6
    wal_path: /var/lib/tangent/wal
    max_file_age_seconds: 120

Durability & Delivery

At-least-once delivery: records are appended to a local WAL before upload.
On restart or failure, the WAL is replayed and any incomplete objects are retried.
Rotation is triggered by size (object_max_bytes) or time (max_file_age_seconds).

tangent.yaml

Runtime

Plugins

DAG

Sources

Sinks

S3

Schema

Example (NDJSON + Zstd)

Example (Parquet + Gzip, Arrow schema)

Durability & Delivery

See also

​Schema

​Example (NDJSON + Zstd)

​Example (Parquet + Gzip, Arrow schema)

​Durability & Delivery

​See also

Schema

Example (NDJSON + Zstd)

Example (Parquet + Gzip, Arrow schema)

Durability & Delivery

See also