Writing Test Cases

dolphin is octo's test runner. A test is a debug config with assertions: the same input, mocks and spies you would pass to octo invoke by hand, plus what should have happened.

Tests live beside the flow, named for it the way Go names its own: orders.yaml is tested by orders_test.yaml. octo skips _test.yaml when it loads a directory.

Set up

task build                       # builds bin/octo and bin/dolphin
export OCTO_PATH=$PWD/bin/octo   # dolphin runs your tests by running the real octo

dolphin shells out to octo, so it has to find one: $OCTO_PATH, then ./octo, then your PATH. See installation.

A first suite

samples/error-handling.yaml charges a card over HTTP and has a flow-level error: chain to catch the failure. Its payments connector points at a discard port, so the real call always fails to connect. Mocking that one block lets the same flow show both paths:

flow: charge-flowlevel

# Every case runs with this, unless it says otherwise. This is where "no test in
# this file ever reaches the payment API" is said once.
mocks:
  charge-flowlevel.call-charge:
    cases:
      - when: 'body.amount > 100'
        error: card declined
    default:
      body: { id: ch_1, status: captured }

inputs:
  small:
    data: { amount: 10 }
  large:
    data: { amount: 500 }

cases:
  - name: a charge the bank accepts comes back captured
    input: small
    expect:
      body: { id: ch_1, status: captured }
    spies:
      charge-flowlevel.call-charge:
        count: 1
        records:
          - input:
              that: ['body.amount == 10']
            output:
              body: { id: ch_1, status: captured }

  # The point of the sample: a failing block redirects to the flow's `error:` chain,
  # whose output BECOMES the flow result, so the flow completes rather than failing.
  - name: a declined card takes the flow error path and answers 502
    input: large
    expect:
      # set-variable's `value` is CEL, so "502" in the flow evaluates to the NUMBER
      # 502. The quotes there are YAML's, not CEL's.
      vars: { httpStatus: 502 }
      that:
        - 'body.error.contains("card declined")'
        - 'body.failedBlock == "call-charge"'
        - 'body.flow == "charge-flowlevel"'
    spies:
      charge-flowlevel.call-charge:
        count: 1
        records:
          - error: card declined

The same mock in the editor's Testing tab. It is the same file either way:

Point dolphin at the flow and it finds the suite beside it:

./bin/dolphin test samples/error-handling.yaml

ok   samples/error-handling_test.yaml  (charge-flowlevel)  73ms

2 passed, 0 failed, 0 errored, 0 skipped  (73ms)

The declined case asserts on body.failedBlock: a mocked failure is indistinguishable from the real block failing, so an error chain that branches on which block died behaves as in production. The spy proves the block was reached with the right amount and that it failed; without it, a flow that skipped the charge and returned the error body for another reason would still pass.

When it fails

Change the expected httpStatus to 500 and run it again:

FAIL samples/error-handling_test.yaml  (charge-flowlevel)  44ms
  --- FAIL: a declined card takes the flow error path and answers 502 (22ms)
        expect.vars.httpStatus: want 500, got 502
        reproduce: bin/octo invoke --config samples/error-handling.yaml \
          --flow charge-flowlevel \
          --run-debug-config /tmp/dolphin-2432200989/error-handling_test.1.yaml \
          --timeout 30s --envelope-out /tmp/dolphin-2432200989/error-handling_test.1.outcome.json

1 passed, 1 failed, 0 errored, 0 skipped  (44ms)

the runs are in /tmp/dolphin-2432200989

The editor reports the same failure, because it runs the same binary:

The failure names the case, says what it wanted and got, and hands you a real, runnable octo command that reproduces it. The debug config for a failing case is kept on disk so you can edit it, re-run it, or break on a block inside it.

Prove the loop looped

spies assert on what happened inside the flow. A block in a foreach body is crossed once per item and every crossing is recorded, so the count is the iteration count.

samples/builtins-demo.yaml seeds two orders and classifies each against a threshold:

flow: demo

# The flow's source is a cron ticker. Sources do not run under `invoke`, so the
# message the ticker would have produced is supplied here instead.
inputs:
  ticked:
    data: { firedAt: "2026-07-10T12:00:00Z" }

cases:
  - name: it seeds two orders and classifies each against the threshold
    input: ticked
    expect:
      body:
        firedAt: "2026-07-10T12:00:00Z"
        orders:
          - { id: 1, amount: 50 }
          - { id: 2, amount: 250 }
      that:
        - '!has(vars.threshold)' # drop-threshold cleaned the scratch variable up
    spies:
      demo.each-order[body].classify-order:
        count: 2
        records:
          - input:
              that: ['vars.order.id == 1', 'vars.order.amount == 50']
          - input:
              that: ['vars.order.id == 2', 'vars.order.amount == 250']

  - name: an empty order list takes the else branch and never classifies
    input: ticked
    mocks:                       # overrides the address for this case only
      demo.seed-orders:
        default:
          body: { firedAt: "2026-07-10T12:00:00Z", orders: [] }
    spies:
      demo.each-order[body].classify-order:
        count: 0                 # the assertion: the foreach body never ran
    expect:
      body: { firedAt: "2026-07-10T12:00:00Z", orders: [] }

count: 0 is how a case proves a branch was not taken or an error path not reached. Records are positional: the first entry is the first crossing. Asserting on fewer crossings than happened is fine; naming more than happened is a failure.

Addresses are the same block-address grammar the debugging flags use, and they work unquoted as YAML keys.

What a case asserts

Omitting expect entirely still asserts that the flow completed and did not fail.

dropped and error are exclusive with each other and with the message fields. body is exact because it is the flow's answer; vars is a subset because variables are scratch space (that: ['vars.size() == 2'] pins them exactly). The CEL is the same CEL every block uses, over body, vars, eventID, correlationID and now. The test file reference has the full field list.

The environment

A config's env is resolved when the config is loaded, before any block runs, so a flow whose connector reads ${ANTHROPIC_API_KEY} cannot be built without one, and mocking the block that would use it does not help. Give the suite its own environment:

env:
  ANTHROPIC_API_KEY: test-key
  DB_DSN: 'file::memory:'

A test key is fake by construction, so it can be committed. For a value several suites share, put it in a file and pass --env-file. A suite's own env: wins over it, and both win over whatever you have exported.

Things worth knowing

One case is one octo process, because mocks and spies are baked into the flow tree when it is built. Cases are fully isolated, and dolphin runs them in parallel, one per CPU (--parallel 1 to serialize).

Sources do not run under invoke: no cron ticks, no HTTP request arrives, no queue delivers. A case seeds what the source would have produced, including the variables an HTTP source copies out of the request; that is what input.vars is for.

Mocking a composite removes everything inside it, so you cannot mock an ai-router and then assert on which route ran. Mock a block inside the composite instead (charge.resilient-charge[process].build-charge).

A message matching no mock case fails the block; it does not fall through to the real one, which is no longer in the flow. Give the spec a default for a catch-all.

A case can un-mock with address: null, so the real block runs for that case. mocks: {} is not that: an empty map contributes nothing. See un-mocking.

skip: "a reason" reports the case as skipped and never runs it. The reason is required.

A cache-scope hit skips its body. Each case is a fresh process with an empty in-memory cache, so a case can only exercise the miss.

The schema

To have your editor complete a suite and a validator check it:

./bin/dolphin schema --format yaml

It is drift-tested against the Go structs. The test file reference is that schema, rendered.

See also

• Test file reference: every key of a suite, field by field.
• Running tests: dolphin test's flags, the exit codes, and CI.
• Debugging a flow: the mocks, spies, and breakpoints a test is built out of, driven by hand.
• Testing in the editor: the same suites, written in a form and run from the Testing tab.
• CLI reference: the full mock spec and address grammar.