Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Troubleshooting

"Unknown Search Command 'Exec'"

Splunk has not loaded the app.

ls "$SPLUNK_HOME/etc/apps/babysoarus/default/commands.conf"
$SPLUNK_HOME/bin/splunk restart

If the file is there and a restart did not help, check that the app is enabled and that metadata/default.meta exports the commands to system scope.

"Babysoarus-daemon Binary Not Found"

The app is installed but bin/babysoarus-daemon is missing or not executable.

ls -l "$SPLUNK_HOME/etc/apps/babysoarus/bin/"
chmod +x "$SPLUNK_HOME/etc/apps/babysoarus/bin/"*

make install sets the modes; copying by hand sometimes does not.

"Could Not Connect to Babysoarus-daemon Within 30s"

The client started the daemon and it never bound its socket. The reason is in the log:

tail -50 "$SPLUNK_HOME/etc/apps/babysoarus/var/daemon.log"

Common causes:

No write permission on var/. The daemon runs as the Splunk user and needs to create its socket, lock file and caches there.

A stale socket owned by another user. Delete var/daemon.sock and retry.

The binary cannot execute. Wrong architecture, or a missing shared library. ldd bin/babysoarus-daemon will say.

"The Bundled Python Executor Is Not Loaded"

Inline Python was requested but the interpreter is missing or unusable.

ls -l "$SPLUNK_HOME/etc/apps/babysoarus/wasm/"

Both exec.wasm and exec.cwasm should be present. If they are not, the build skipped them:

make python precompile
SPLUNK_HOME=/opt/splunk make install

If the log says "precompiled Python executor is incompatible; falling back to compiling exec.wasm", the .cwasm was produced by a different Wasmtime version. Everything still works, but every daemon start pays a few seconds of compilation. Rebuild it with make precompile.

"Unknown Function 'X'; Available Functions: ..."

The name does not match a .wasm file in functions/public/ or functions/private/. The error lists what is loaded, which is usually enough.

If the file is there and not listed, it failed to load. The log says why:

grep 'failed to load function' "$SPLUNK_HOME/etc/apps/babysoarus/var/daemon.log"

The usual cause is a core module rather than a component. See Custom functions.

"ModuleNotFoundError" in Inline Python

The module was not imported when the interpreter component was built, so it does not exist inside the sandbox. This is a property of how the component is produced, not a path problem.

Add it to python/exec_component/requirements.txt if it is a third-party package, or to the import block in babysoarus_exec.py if it is standard library, then rebuild. See Dependencies.

"Guest Exceeded Its Execution Time Budget"

A batch ran longer than 30 seconds. Usually one of:

An accidental infinite loop. The deadline caught it, which is the point.

A slow HTTP call inside the per-event loop. Move it outside the loop and make one request for the batch.

Genuinely heavy work. Raise BABYSOARUS_TIMEOUT_EXECUTE_SECS, remembering it applies per batch of 32 events, or lower batch_size so each call does less.

Results Are Wrong in a Numeric Way

Almost always string arithmetic:

e['total'] = e['bytes_in'] + e['bytes_out']       # '100' + '200' = '100200'
e['total'] = int(e['bytes_in']) + int(e['bytes_out'])   # 300

Every field value is a string. See Inline Python.

"KeyError" on a Field That Is Definitely There

Most often, Splunk never sent it.

Splunk only materialises search-time extracted fields that something in the pipeline references. Your snippet is opaque to Splunk, so a field you read but never mention elsewhere arrives missing.

# Does not work: nothing references `action`.
index=auth | exec inline="for e in events: e['x'] = e['action']"

# Works.
index=auth | fields _time user src_ip action | exec inline="..."

After stats, table or eval, the fields are already materialised and this does not apply.

Failing that, the field is genuinely absent from some events, because Splunk fields are sparse:

user = e.get('user', 'unknown')

It can also be a naming mismatch: stats ... as x renames the field, and the snippet must use the new name.

A Search Hangs

Check the daemon is alive and doing something:

pgrep -af etc/apps/babysoarus/bin/babysoarus-daemon
tail -f "$SPLUNK_HOME/etc/apps/babysoarus/var/daemon.log"

With BABYSOARUS_LOG=debug the log shows every instantiation and pool hit, which tells you whether work is happening or the guest is stuck. A stuck guest is interrupted by the execution deadline, so a hang lasting more than a minute is more likely to be a slow HTTP call.

Everything Is Slower Than the Benchmarks

Check the daemon is using the pooling allocator:

grep 'engine ready' "$SPLUNK_HOME/etc/apps/babysoarus/var/daemon.log"

allocation=Pooling is what you want. allocation=OnDemand means it could not reserve address space, usually a container memory limit or a low vm.max_map_count, and instantiation will be slower.

Then check instances are being reused:

BABYSOARUS_LOG=debug   # then restart the daemon
grep -c 'instantiated cold instance' var/daemon.log
grep -c 'reusing warm instance' var/daemon.log

Mostly cold means the pool is too small for your concurrency. Raise BABYSOARUS_POOL_SIZE.

Splunk Shows "External Search Command Exited Unexpectedly"

The command crashed rather than reporting through the protocol. That is a bug; please report it. To gather detail:

grep BabySOARus "$SPLUNK_HOME/var/log/splunk/splunkd.log" | tail -50

The search's own search.log in the dispatch directory has the command's stderr.

Alert Action Does Not Run

Check it is enabled on the saved search:

grep -A5 'action.execalert' "$SPLUNK_HOME/etc/apps/*/local/savedsearches.conf"

Then check what it did:

grep babysoarus-alert "$SPLUNK_HOME/var/log/splunk/splunkd.log" | tail -20

The action logs the search name, the search id and the row count on every run. No line at all means Splunk never invoked it, which is a saved-search configuration problem rather than a BabySOARus one.

Starting Completely Fresh

pkill -f etc/apps/babysoarus/bin/babysoarus-daemon
rm -rf "$SPLUNK_HOME/etc/apps/babysoarus/var"

Everything under var/ is regenerable. The next search rebuilds it.