Operations
What runs, where it lives, and how to tell what it is doing.
The Daemon
One babysoarus-daemon process per host, holding the WebAssembly engine and the
pool of warm interpreters.
It is not managed by Splunk. The first search that needs it starts it, and it stays running afterwards. That is the entire performance argument: the expensive setup happens once rather than once per search.
# Is it running?
pgrep -af etc/apps/babysoarus/bin/babysoarus-daemon
# What is it doing?
tail -f $SPLUNK_HOME/etc/apps/babysoarus/var/daemon.log
# Restart it, waiting for searches already in flight to finish first
bin/stop-daemon.sh "$SPLUNK_HOME/etc/apps/babysoarus/bin/babysoarus-daemon"
Stopping it is graceful, not merely safe: a search already using it keeps running to
completion (up to a five-minute grace period, BABYSOARUS_SHUTDOWN_GRACE_SECS to change that)
rather than failing, and a new search started while the old one is draining gets a fresh
daemon automatically, because the socket is unlinked the moment the stop signal arrives.
Verified live: a ten-event search sleeping two seconds per event survived a stop issued
partway through, finishing normally rather than erroring (execution log, 2026-08-22).
bin/stop-daemon.sh sends the signal and then genuinely waits for the process to be gone,
which a bare pkill does not -- it returns immediately, before the daemon (which may now be
draining real work) has actually exited.
Only one daemon can run per app directory. It takes an exclusive lock on
var/daemon.lock before binding its socket, so two searches starting at the
same instant cannot produce two daemons: the loser exits quietly and its
client connects to the winner.
Reading the Log
At info the daemon is quiet. A healthy start looks like:
INFO mounted Splunk lookup directories count=6
INFO engine ready app_dir=/opt/splunk/etc/apps/babysoarus allocation=Pooling
INFO loaded precompiled Python executor path=.../wasm/exec.cwasm
INFO compiled component hash="35b654f1..." elapsed_ms=56
INFO loaded function function=example path=.../functions/public/example.wasm
INFO scanned functions directories functions=1
INFO watching functions directory dir=.../functions/public
INFO listening socket=.../var/daemon.sock startup_ms=121
Two lines are worth checking.
allocation=Pooling means the pooling allocator is in use. If it says
OnDemand, the host would not let it reserve address space and instantiation
will be slower. Usually a container memory limit or a low vm.max_map_count.
loaded precompiled Python executor means the shipped .cwasm was
usable. If instead you see precompiled Python executor is incompatible; falling back to compiling exec.wasm, the artefact was built by a different
Wasmtime version and every daemon start pays a few seconds of compilation.
Rebuild with make precompile.
Turn up detail with BABYSOARUS_LOG=debug, which adds one line per instantiation
and per pool hit.
Upgrading
make app
SPLUNK_HOME=/opt/splunk make install
$SPLUNK_HOME/bin/splunk restart
make install kills the running daemon before replacing binaries, so the old
one cannot keep serving old code. Splunk itself only needs restarting when
configuration files change.
Capacity
Sizing follows from three properties rather than from a throughput number:
- Per-search overhead is small and fixed. A warm search pays microseconds of daemon time before your logic runs, so it does not grow with your data.
- Throughput is dominated by your snippet, not by the sandbox boundary. Measure the searches you actually run; a benchmark of an empty loop tells you about the loop.
- Memory scales with pool size, not with search volume, because a pooled instance holds a booted interpreter and is reused.
Concurrent searches run genuinely in parallel across cores, so the daemon is rarely the bottleneck. Measure on your own hardware before sizing; the performance chapter explains how, and how much the figures move between runs. Where the daemon can be the bottleneck:
Too few warm instances. BABYSOARUS_POOL_SIZE defaults to 16 per function. If
debug logging shows constant instantiated cold instance, raise it.
Long-running guest calls. A snippet making a slow HTTP call holds its instance for the duration. That is correct, but many such searches at once will exhaust the pool. Batch the calls.
Memory. Each instance maps the interpreter image copy-on-write, so the
marginal cost per instance is much smaller than the 33 MB image suggests.
Lower BABYSOARUS_POOL_SIZE and BABYSOARUS_POOL_IDLE_SECS on constrained hosts.
Distributed Deployments
Deploy the app everywhere the command will run. Each host runs its own daemon and its own pool; nothing is shared between hosts and there is no coordination.
exec reports itself as distributable, so Splunk may run it on indexers.
execgen always runs on the search head, as generating commands do. If you
would rather exec always ran on the search head too, add local = true to
the [exec] stanza in commands.conf.
Guest state that accumulates within a search, such as the seen set in the
first-seen example, is per host. Under distributed search each indexer
sees only its own share. When that matters, do the aggregation in SPL with
stats and run the snippet after it, on the search head.
Saved Functions Reach Every Host on Their Own
Unlike guest state, a function you save in the editor is not per host. Saving writes to the daemon's KV store, which every member of a search head cluster already shares, then reaches each member's own disk the next time a search actually lands there.
That last step is driven by real search traffic, not a timer: the daemon holds no standing Splunk credentials, so a background poller checking the KV store at 3am would need to invent some. Instead, every search that arrives already carries the searching user's own session token, which is exactly the credential a KV store read needs, at exactly the moment the answer matters. A member with no traffic never syncs, and that costs nothing, since nothing on that host is asking to run the function yet.
In practice this means a save reaches every host with searches actually running against it within a couple of seconds, and a host that has been idle picks it up on its very next search rather than on some fixed interval. A slow or unreachable KV store degrades to stale, not stuck: reconciling has a five-second budget, and a run proceeds with whatever is already on disk once that budget runs out rather than hang the search waiting for it.
Backup
There is nothing to back up. var/ is a cache, and everything else came from
the build. Include functions/public/ in whatever manages your app content,
the same as any other app file.
Monitoring
The daemon exposes no metrics endpoint. What you can watch:
index=_internal source=*splunkd.log* component=ChunkedExternProcessor BabySOARus
Search-level timing is in the job inspector under command.exec, which counts
the time the command spent including the daemon round trip.
For the daemon itself, the log is the interface. BABYSOARUS_LOG=info reports
loads, reloads, evictions and connection errors, which is generally what you
want to alert on.