Skip to content
← Insights

We killed our workflow engine mid-transaction. On purpose.

The process nobody draws

A loan application is not a chain of API calls. Four checks run in parallel. An agent reads the documents. An underwriter approves three days later. A regulatory cooling-off period runs out. Then come the booking, the email and the CRM update.

Most agent frameworks model the first two minutes of that process. The other three days are where an enterprise process actually lives: waiting, restarting after a crash, undoing what was done. Making every step "agentic" only adds uncertainty where predictability is required.

hwf, for hybrid workflow, is the prototype we wrote in C++20 to carry the whole process.

Five kinds of step

A process is a graph of steps, and a step is one of five kinds.

  • Code runs on a CPU pool.
  • Agent runs on an I/O pool under a resource limit, so that no more than four model calls are in flight at once.
  • Human runs on nothing. The instance is parked and a task lands in someone's inbox.
  • Timer is a durable due time.
  • Notify sends the email.

Each step can declare its dependencies, a condition, retries with exponential backoff, and a compensation function that undoes it. Independent steps run in parallel on a work-stealing scheduler.

State is the journal

Every state change is appended to a journal before anything acts on it. The state of a process is a fold over its events and nothing else. Most of the guarantees follow from that one decision.

  • A step is journaled as scheduled before any worker sees it.
  • After a crash, the steps that were in flight are marked as recovered and run again. Each side effect carries an idempotency key, so the downstream system deduplicates the second attempt.
  • Stop cancels in-flight steps cooperatively and keeps the completed work. Resume continues from there.
  • Rollback compensates completed steps in reverse dependency order. If a compensation keeps failing, the process goes to a failed state for a person to fix.

The demo script runs eight scenarios. The one we show first: kill -9 the engine after the loan booking has reached the simulated core banking system, but before the journal has recorded it. On restart the step runs again and the booking is deduplicated. One loan, not two.

Timeline
The engine dies between the booking and its record. The loan is still booked once.
Journal 01
The booking step is written as scheduled, before any worker sees it.
Engine 02
The booking is sent to core banking with its idempotency key.
Crash 03
kill -9. The engine dies before the journal records the result.
Restart 04
The journal is replayed. The step was in flight, so it runs again.
Core banking 05
Same key, seen twice. The second booking is discarded.
Journal 06
The step is recorded as done. One loan, not two.
The journal is written before any action, and every side effect carries an idempotency key. Replaying after a crash repeats the step, and the downstream system discards the duplicate.

What we measured

On an Apple M5 Max, with the engine's own benchmark.

50 to 70µs
per step, each one durably committed before the next starts
260,000
steps per second on a burst of 5,000 instances of 18 steps
7×
the cost of durability, compared with raw scheduling at peak

The raw scheduler, with empty tasks and no durability, runs 1.7 to 2 million tasks per second. Durability therefore costs about seven times raw scheduling at peak. That is far from being the bottleneck once a step calls a model for seconds or waits for a person for hours.

One caveat. On this machine our full-sync mode stops short of a complete flush to the disk, so the numbers for true power-loss safety would be lower. A Linux server on NVMe storage is the honest setting, and we have not published that run yet.

The engine passes thread-safety and memory-safety analysis with no finding.

What it is not yet

It is a prototype. The list of what production needs is long.

  • Agent steps block an I/O thread while they wait. Thousands of concurrent model calls need async I/O.
  • There is one engine process. Scaling out means partitioning instances across engines and moving the journal to a store built for it.
  • Workflows are C++ code. A product needs a declarative format.
  • Instances are pinned to a workflow version, but there is no migration tooling.
  • The journal has no snapshot or compaction for very long processes.
  • There is no HTTP API and no inbox screen.

We publish the limits next to the numbers because an architect needs both to judge a design.

For our research on the workflow model itself, read the 2yFlow technical report.

Let's structure your next intelligent system.

Whether you are launching an AI initiative, modernizing architecture, or aligning technology with execution, CMX helps you create the structure required to move with confidence.