Skip to main content
Sources define where Tangent reads data from — for example, a local Unix socket, Amazon SQS queue, file, or Kafka/MSK topic.
Each source reads raw bytes, applies a configured decoding policy, and emits structured records into Tangent’s pipeline.

​
Source Types

TypeDescriptionExample
socketLocal Unix domain socket (fastest for local ingestion)/tmp/sidecar.sock
fileRead structured logs from local or mounted files/var/log/app/access.log
mskConsume from Amazon MSK (Kafka) topicsb-1.example.com:9092
sqsPoll messages from Amazon SQS queueshttps://sqs.us-east-1.amazonaws.com/...

​
Common Configuration

Every source supports a few shared parameters:
​
type
string
required
Selects the source type.
One of: socket, file, msk, or sqs.
​
decoding
object
Defines how incoming bytes are parsed into structured records.
See Decoding for format and compression options.

​
Example (MSK Source)

tangent.yaml
sources:
  kafka_in:
    type: msk
    bootstrap_servers: "b-1.example.com:9092,b-2.example.com:9092"
    topic: logs
    group_id: tangent-node
    decoding:
      format: ndjson
      compression: auto

​
Behavior

  • Each source runs in its own async worker, managed by Tangent’s runtime.
  • Data is read continuously, batched in memory, and flushed based on runtime settings (batch_size / batch_age).
  • Decoding and optional re-encoding happen before the data enters the plugin or sink DAG.

​
See also