Daemon Protocol

• Transport
• Framing
• Exchange
• Lazy Start
• Splunk-facing Protocols
  • Chunked Custom Search Command Protocol v2
  • Alert Action Contract
• Why Not gRPC
• Implementation

How the per-search client binaries talk to the daemon. You do not need this to use BabySOARus; it is here because the protocol is small enough to describe completely, and knowing it makes the logs legible.

Transport

A Unix domain socket at <app>/var/daemon.sock, mode 0600.

One connection per search or alert-action invocation, opened at the start and closed at the end. The connection is full-duplex: the client keeps sending events while results come back.

Framing

[4-byte little-endian payload length][1-byte frame type][payload]

The length counts only the payload. Read four bytes, read one byte, read that many bytes.

Frames within a connection are strictly ordered, so results match requests by arrival order. There are no correlation identifiers, because there is nothing to correlate.

Payloads are capped at 64 MB, which bounds memory if a peer sends a corrupt length prefix.

Exchange

client                          daemon
  |                               |
  |------------ Hello ----------->|  resolve function, take a warm instance,
  |                               |  call begin-run
  |<--------- HelloAck -----------|
  |                               |
  |------------ Event ----------->|  execute-batch
  |------------ Event ----------->|  (pipelined: the client does not wait)
  |<--------- Result -------------|
  |------------ Event ----------->|
  |<--------- Result -------------|
  |<--------- Result -------------|
  |                               |
  |------------- End ------------>|  end-run, return the instance to the pool
  |<---------- Done --------------|

The client keeps up to eight Event frames outstanding before waiting for results. That keeps the daemon busy while the client encodes the previous answer, and bounds memory at a few hundred events rather than the whole search.

A guest-reported error arrives as an Error frame in place of the Result for that batch. The connection stays open and later events are still processed, which is why one bad batch does not abort a search.

Lazy Start

The client tries to connect. If the socket is missing or refused, it starts the daemon as a detached process with setsid, redirecting stdio to var/daemon.log, then retries with a short backoff.

Two searches starting at the same instant may both try. The daemon takes an exclusive flock on var/daemon.lock before binding its socket, so the loser exits quietly and its client connects to the winner's socket.

The daemon is deliberately not a child of the search process. Splunk tears down a search's process group when the search ends, and the whole point is to outlive it.

Splunk-facing Protocols

The client binaries speak two Splunk protocols on the other side. Both are Splunk's, and both were implemented against the Splunk SDK for Python's own source rather than from prose documentation.

Chunked Custom Search Command Protocol v2

Used by exec and execgen on stdin and stdout.

chunked 1.0,<metadata_length>,<body_length>\n
<metadata_length bytes of UTF-8 JSON>
<body_length bytes of CSV>

1. Splunk sends a chunk with action: "getinfo" and an empty body. Its searchinfo carries the arguments, session key, management URI and search identifiers.
2. The command replies with a metadata-only chunk containing its configuration: type and generating.
3. Splunk sends action: "execute" chunks, each with a finished flag and a CSV body. The command replies to each with its own chunk, mirroring finished.

Bodies use Splunk's CSV dialect: comma delimited, " quoting with doubling, CRLF terminators, and a __mv_<field> companion column beside every field carrying the multivalue encoding $a$;$b$, with a literal $ escaped as $$.

Alert Action Contract

Used by execalert. Splunk runs the binary with --execute and a JSON payload on stdin, containing among other things a path to the results file. That file is Splunk's CSV, usually gzipped. Compression is detected by magic bytes rather than filename, because the extension is not guaranteed.

Why Not gRPC

The protocol carries JSON arrays over a local socket between two processes built from the same repository at the same version. Length-prefixed frames are about fifty lines of code on each side, add no dependencies, and have no schema-compatibility story to maintain because there is nothing to be compatible with.

Implementation

All four have test suites; the CSV and chunked modules in particular are covered against the exact encodings the Splunk SDK produces.