Skip to main content
The S3 sink uploads objects to S3 with durability via a local write-ahead log (WAL).
Encoding, compression, object sizing, and concurrency are configured via common sink options.

​
Schema

​
type
string
required
Must be set to s3.
​
bucket_name
string
required
Name of the destination S3 bucket (do not prefix with s3://).
​
region
string
AWS region for the bucket (optional).
If omitted, the AWS SDK’s default resolution is used (env vars, instance metadata, etc.).
​
wal_path
string
Filesystem path for the sink’s write-ahead log (WAL).
Defaults to /tmp/wal.
​
max_file_age_seconds
int
Maximum age (in seconds) before a WAL file is rotated and flushed to S3.
Defaults to 60.
​
encoding
object
Controls how Tangent serializes records emitted by this source. See Encoding.
​
compression
object
Controls how Tangent compresses payloads emitted by this source. See Compression.

​
Example (NDJSON + Zstd)

tangent.yaml
sinks:
  s3_staging:
    type: s3
    bucket_name: my-bucket
    region: us-west-2
    # common options
    encoding: ndjson         # default
    compression:             # default shown explicitly
      type: zstd
      level: 3
    wal_path: /tmp/wal
    max_file_age_seconds: 60

​
Example (Parquet + Gzip, Arrow schema)

tangent.yaml
sinks:
  s3_parquet:
    type: s3
    bucket_name: analytics-data
    region: us-east-1
    encoding:
      type: parquet
      schema: ./schemas/arrow_schema.json
    compression:
      type: gzip
      level: 6
    wal_path: /var/lib/tangent/wal
    max_file_age_seconds: 120

​
Durability & Delivery

  • At-least-once delivery: records are appended to a local WAL before upload.
  • On restart or failure, the WAL is replayed and any incomplete objects are retried.
  • Rotation is triggered by size (object_max_bytes) or time (max_file_age_seconds).

​
See also