Detection-as-code with `babysoarus-ci`

Getting the Pieces
Running One Thing
Running a Test Suite
Benchmarking, Honestly
In a Pipeline
Not Built Yet

Testing a detection normally needs nothing running: no Splunk, no daemon, no network. babysoarus-ci loads the same runtime the editor's Tests panel and the built-in library both use, directly, as a single binary -- which is what makes running your detections in a pull request practical instead of a project of its own.

Getting the Pieces

Two things, and they come from different places.

An installed app directory, for the Python executor and your own saved functions. Any real install already has one -- point --app (or BABYSOARUS_APP_DIR) at $SPLUNK_HOME/etc/apps/babysoarus, or at wherever babysoarus.spl was extracted for a CI job that has no Splunk to install into at all.

The babysoarus-ci binary itself, which ships in the same package, at bin/babysoarus-ci. So both pieces come from the one babysoarus.spl you already have: extract it, and the app directory and the binary are both inside. A CI job needs no toolchain, no Splunk and no network -- one archive, one extraction, done.

export BABYSOARUS_APP_DIR=/path/to/installed/BabySOARus
babysoarus-ci --version

Running One Thing

echo '[{"n": 5}]' > events.json
babysoarus-ci run --inline "for e in events:
    e['doubled'] = int(e['n']) * 2" --events events.json

--function NAME works the same way in place of --inline CODE, against a saved function instead of a snippet typed on the command line. This is the primitive everything else is built from, and on its own it is enough to git bisect a detection that used to pass and does not any more.

Running a Test Suite

babysoarus-ci test tests/

Test definitions are exactly the test.json shape the editor uses -- the same file that runs in the browser runs here unchanged, because a test that only exists in a KV store cannot be reviewed in a pull request:

{
  "name": "doubles the value",
  "inline": "for e in events:\n    e['doubled'] = int(e['n']) * 2",
  "events": [{ "n": 5 }],
  "expect": { "count": 1, "contains": [{ "n": 5, "doubled": 10 }] }
}

A path may be a single definition, an array of them in one file, or a directory -- walked recursively, entries sorted, so two identical runs report in the same order.

Exit codes are the whole API, and 1 and 2 mean different things on purpose: 0 every test passed, 1 a test failed, 2 the tool could not even run (a missing app directory, an unreadable file). A CI system that cannot tell "your detection is broken" from "the runner is misconfigured" shows both as the same red cross. --json prints machine-readable output instead of the human-readable pass/fail list, for a job that wants to parse the result rather than grep it.

A definition that cannot even start (a typo in function, malformed JSON) is reported as that one test failing, not as the whole run aborting -- one bad file never hides the results of every good one.

Benchmarking, Honestly

# Once, to record a baseline:
babysoarus-ci bench --function hunting/beacon_jitter --events events.json \
  --samples 30 --save baseline.json

# On every later run, to gate against it:
babysoarus-ci bench --function hunting/beacon_jitter --events events.json \
  --samples 30 --max-noise 10 --baseline baseline.json

Every figure is a median with its 10th and 90th percentiles, never a bare number, and a null control -- the same function measured as both sides of a comparison -- runs first, so the noise floor is on screen before any figure that might be judged against it. --max-noise refuses to report at all once that floor is too high to trust; --baseline then fails only once a difference is larger than both runs' noise floors combined, not on any slowdown at all, which is what keeps a regression gate usable on shared CI hardware rather than disabled within a week for crying wolf.

In a Pipeline

- name: babysoarus-ci
  run: |
    babysoarus-ci test tests/

The whole point is that this step needs nothing else in the job: no Splunk service container, no network egress, no Python interpreter on the runner -- the one inside the WebAssembly component is the only one that ever runs your code, sandboxed exactly as it is in production.

Not Built Yet

babysoarus-ci fuzz and babysoarus-ci lint are named in the usage text and refused outright rather than treated as typos -- the message says planned, not built yet rather than unknown command.

Keyboard shortcuts

BabySOARus