Skip to main content
The compression block defines how Tangent compresses serialized records before they’re written to disk or uploaded to S3.
Compression can be configured globally under encoding, or inline within a specific sink.

​
Schema

​
compression.type
string
required
Specifies which compression algorithm to use.Options
  • none — no compression
  • gzip — standard Gzip compression
  • zstd — modern Zstandard compression (fast, high ratio)
  • snappy — Snappy block compression (Avro-only)
  • deflate — Deflate stream compression (Avro-only)
​
compression.level
int
Compression level (optional).
Defaults vary by algorithm:
AlgorithmDefaultRangeNotes
gzip60–9Higher = smaller output, slower
zstd31–22Higher = smaller output, slower
snappy——Fixed speed/ratio
deflate——Typically used for Avro data

​
Behavior

  • For NDJSON and JSON, Tangent compresses the entire file object.
  • For Avro and Parquet, compression applies to data blocks inside the file. The file itself is not additionally wrapped.
  • Tangent automatically appends appropriate file extensions:
    • .gz for Gzip
    • .zst for Zstd

​
Examples

​
Default (Zstandard level 3)

tangent.yaml
compression:
  type: zstd
  level: 3

​
Gzip with level 9

tangent.yaml
compression:
  type: gzip
  level: 9

​
No compression

tangent.yaml
compression:
  type: none
ScenarioRecommended Compression
High-throughput pipelineszstd (fast and efficient)
Archival or long-term storagegzip
Avro or Parquet encodingzstd or snappy
Local testingnone

​
Defaults Summary

FieldDefaultDescription
compression.typezstdModern default for speed and ratio
compression.level3Balanced performance

​
See also