Overview

Sources define where Tangent reads data from — for example, a local Unix socket, Amazon SQS queue, file, or Kafka/MSK topic.
Each source reads raw bytes, applies a configured decoding policy, and emits structured records into Tangent’s pipeline.

Source Types

Type	Description	Example
`socket`	Local Unix domain socket (fastest for local ingestion)	`/tmp/sidecar.sock`
`file`	Read structured logs from local or mounted files	`/var/log/app/access.log`
`msk`	Consume from Amazon MSK (Kafka) topics	`b-1.example.com:9092`
`sqs`	Poll messages from Amazon SQS queues	`https://sqs.us-east-1.amazonaws.com/...`

Common Configuration

Every source supports a few shared parameters:

string

required

Selects the source type.
One of: socket, file, msk, or sqs.

object

Defines how incoming bytes are parsed into structured records.
See Decoding for format and compression options.

Example (MSK Source)

tangent.yaml

sources:
  kafka_in:
    type: msk
    bootstrap_servers: "b-1.example.com:9092,b-2.example.com:9092"
    topic: logs
    group_id: tangent-node
    decoding:
      format: ndjson
      compression: auto

Behavior

Each source runs in its own async worker, managed by Tangent’s runtime.
Data is read continuously, batched in memory, and flushed based on runtime settings (batch_size / batch_age).
Decoding and optional re-encoding happen before the data enters the plugin or sink DAG.

tangent.yaml

Runtime

Plugins

DAG

Sources

Sinks

Source Types

Common Configuration

Example (MSK Source)

Behavior

See also

​Source Types

​Common Configuration

​Example (MSK Source)

​Behavior

​See also

Source Types

Common Configuration

Example (MSK Source)

Behavior

See also