Your First Test

Set up

You need both binaries. From a clone of the repository:

task build                          # writes bin/octo and bin/dolphin
export OCTO_PATH=$PWD/bin/octo      # the octo dolphin will drive
export PATH=$PWD/bin:$PATH          # so `dolphin` is on your path

dolphin runs your tests by running the real octo, so it needs to find a binary. See installation for the release tarballs if you would rather not build.

Make a scratch directory with one flow in it:

mkdir ~/octo-first-test && cd ~/octo-first-test
cp /path/to/octo/samples/error-handling.yaml .

The flow under test is charge-inline. It POSTs a charge to an upstream inside a handle-errors block, so a failure becomes a "degraded" body rather than a 500. Its upstream is http://127.0.0.1:9, the discard port: every call fails to connect, so "it failed" is the only outcome the real flow can produce.

Get to green before you assert anything

Write this as error-handling_test.yaml, beside the flow:

flow: charge-inline

cases:
  - name: it runs to the end
    expect: {}

Run it:

dolphin test error-handling.yaml

ok   error-handling_test.yaml  (charge-inline)  68ms

1 passed, 0 failed, 0 errored, 0 skipped  (68ms)

dolphin found the suite by itself: you pointed at the flow file and it ran the _test.yaml beside it, as Go pairs orders.go with orders_test.go. octo skips _test.yaml when it loads a directory.

An empty expect is not an empty test. It asserts that the flow ran to the end and did not fail, which passes here because the handle-errors chain recovers.

Stand in for the block that calls the world

Replace the file with:

flow: charge-inline

mocks:
  charge-inline.charge[process].call-charge:
    default:
      body: { id: ch_1, status: captured }

cases:
  - name: a charge the bank accepts comes back captured
    input:
      data: { amount: 10 }
    expect:
      body: { id: ch_1, status: captured }

dolphin test error-handling.yaml

ok   error-handling_test.yaml  (charge-inline)  94ms

1 passed, 0 failed, 0 errored, 0 skipped  (94ms)

The mock replaces the call-charge block: the HTTP request never happens and the flow gets { id: ch_1, status: captured } as if the bank had said so. Everything around it ran for real.

The address charge-inline.charge[process].call-charge reads left to right: the flow, the handle-errors block named charge, its process slot, and the block inside it. See the address grammar. In the editor a picker lists the addressable blocks.

Break it on purpose

Change the expected id from ch_1 to ch_2, leave the mock alone, and run it again:

FAIL error-handling_test.yaml  (charge-inline)  77ms
  --- FAIL: a charge the bank accepts comes back captured (77ms)
        expect.body:
              want: {"id":"ch_2","status":"captured"}
              got:  {"id":"ch_1","status":"captured"}
        reproduce: /path/to/bin/octo invoke --config error-handling.yaml \
          --flow charge-inline \
          --run-debug-config /tmp/dolphin-143073336/error-handling_test.0.yaml \
          --timeout 30s \
          --envelope-out /tmp/dolphin-143073336/error-handling_test.0.outcome.json

0 passed, 1 failed, 0 errored, 0 skipped  (77ms)

the runs are in /tmp/dolphin-143073336

The failure has three parts: the case name in the words you wrote, want against got, and reproduce, a runnable octo command that re-runs exactly this case. The debug config for a failing case is kept on disk so that command works.

Put ch_1 back and confirm it is green again before moving on.

Test the path you wrote the flow for

charge-inline exists to survive a failing charge. Add a second case that overrides the mock for itself, so the block fails:

flow: charge-inline

mocks:
  charge-inline.charge[process].call-charge:
    default:
      body: { id: ch_1, status: captured }

cases:
  - name: a charge the bank accepts comes back captured
    input:
      data: { amount: 10 }
    expect:
      body: { id: ch_1, status: captured }

  - name: a declined card is recovered into a degraded body
    input:
      data: { amount: 500 }
    mocks:                              # replaces the file's mock, for this case only
      charge-inline.charge[process].call-charge:
        default:
          error: card declined
    expect:
      that:
        - 'body.status == "degraded"'
        - 'body.reason.contains("card declined")'
    spies:                              # and prove the block was actually reached
      charge-inline.charge[process].call-charge:
        count: 1
        records:
          - error: card declined

ok   error-handling_test.yaml  (charge-inline)  119ms

2 passed, 0 failed, 0 errored, 0 skipped  (119ms)

Three new ideas. error: on a mock fails the block exactly as a real failure would. that: holds CEL expressions over the result, each of which must be true; use it instead of an exact body when you do not want to pin a message word for word. spies: assert on what happened inside the flow: count: 1 says the block was reached exactly once, and the record says it failed. Without the spy, a flow that skipped the charge and returned a degraded body for another reason would still pass.

Run it the way CI will

Point dolphin at the directory and it runs every suite in it:

dolphin test .

ok   error-handling_test.yaml  (charge-inline)  149ms

2 passed, 0 failed, 0 errored, 0 skipped  (149ms)

The exit code is the contract a pipeline acts on:

Add --junit report.xml and your CI system will render the cases by name; see running tests.

The same file, in the editor

Start the editor over the directory you just worked in:

docker run -p 3000:3000 -v "$PWD:/work" juancavallotti/octo

The image ships dolphin and points the editor at it. The Testing tab opens the suite you just wrote as a form, and Run tests gives the same verdict because it shells out to the same binary.