Benchmarks

How far one engine goes

Every release, we put the systema engine alone in virtual machines of fixed size and drive it with real SDK workers until it stops keeping up. These are the results: what each size sustains, what it costs in CPU and memory on the way, and where it runs out.

Engine
systema 0.25.0
Measured
Oct 20, 2026
Machine
Local workstation, 16 CPUs
Sizes
3 VMs and the host, 8 scenarios

Pick a machine size

Every figure and chart below follows it. The other sizes stay in the charts, behind, to compare.

Calls per second through the engine
97 k/s
Worker to engine to worker and back, 512 calls in flight. p99 17 ms.
Engine CPU per call
20 µs
About 51k calls a second for each core the engine has.
Observer activations per second, sustained
50 k/s
Every event delivered. Offer 100 k/s and 74% arrive; the rest is dropped, and counted.
Calls a worker makes to its own functions
4.7 M/s
In-process, unobserved: the engine spends 0.00 CPU on them.
Engine memory per connected worker
161 KiB
5,000 idle workers held: 786 MiB on top of 40 MiB at rest.
Engine memory per observer
10 KiB
64,000 observers held: 638 MiB.

AMD Ryzen 7 9800X3D 8-Core Processor · 16 CPUs. A size of N vCPU · M is the engine with N virtual CPUs and an M memory cap, the load generator beside it on 6 vCPUs of its own; how we measure.

Calls

Calls through the engine

A worker calls a function another worker hosts. The call goes to the engine, the engine routes it to the host, and the answer comes back the same way: four WebSocket frames per call.

Callers keep more and more calls in flight, from 1 to 2,048. Throughput grows while the engine has CPU to spare, up to a peak: the size’s capacity.

SizePeakp99 thereCPU per call
1 vCPU · 512 MiB63 k/s351 ms16 µs
2 vCPU · 1 GiB97 k/s17 ms20 µs
4 vCPU · 2 GiB171 k/s6.2 ms22 µs
Host · 8 CPUs424 k/s8.3 ms15 µs

Calls per second, by calls in flight

64-byte payloads. Each size gets a fresh engine.

Calls per second through the engine, by calls in flight, per machine size
MachineCalls in flightCalls per second
1 vCPU14.1k
1 vCPU815k
1 vCPU3228k
1 vCPU12846k
1 vCPU51255k
1 vCPU2,04863k
2 vCPU13.6k
2 vCPU828k
2 vCPU3261k
2 vCPU12885k
2 vCPU51297k
2 vCPU2,04892k
4 vCPU13.6k
4 vCPU827k
4 vCPU3297k
4 vCPU128160k
4 vCPU512171k
4 vCPU2,048152k
Host128k
Host8119k
Host32219k
Host128330k
Host512403k
Host2,048424k
How to read it. Each line climbs while its engine has idle CPU and peaks when it has none; pushed further, the smaller sizes give some of it back. More cores move the peak up while the CPU per call stays about the same, so capacity grows with them.

p99 latency, by calls in flight

p99 latency of calls through the engine, by calls in flight, per machine size
MachineCalls in flightp99 latency
1 vCPU1289 µs
1 vCPU8769 µs
1 vCPU322 ms
1 vCPU1284.5 ms
1 vCPU51270 ms
1 vCPU2,048351 ms
2 vCPU1313 µs
2 vCPU8401 µs
2 vCPU32878 µs
2 vCPU1282.6 ms
2 vCPU51217 ms
2 vCPU2,04866 ms
4 vCPU1313 µs
4 vCPU8416 µs
4 vCPU32574 µs
4 vCPU1281.4 ms
4 vCPU5126.2 ms
4 vCPU2,04830 ms
Host165 µs
Host8146 µs
Host32464 µs
Host1281 ms
Host5122.4 ms
Host2,0488.3 ms
How to read it. Past the peak, more calls in flight only wait in line: latency grows with them while throughput does not. With one call in flight, a call takes longer in the VMs than on the host: the same engine, so that is what virtualization costs at low load.

Observers

What observing a function costs

Every function is observable. Here each called function gains 1 to 64 observers without a filter, so every call activates all of them, fire-and-forget, while 64 callers keep calling.

An activation is a call the engine makes on its own, and it costs about as much as one. So the engine splits the same CPU between the calls and their activations: the more observers per call, the fewer calls fit.

Activations per second, by observers per function

Observer activations per second, by observers per function, per machine size
MachineObservers per functionActivations per second
1 vCPU126k
1 vCPU439k
1 vCPU1671k
1 vCPU6451k
2 vCPU143k
2 vCPU439k
2 vCPU1638k
2 vCPU6432k
4 vCPU160k
4 vCPU4119k
4 vCPU16178k
4 vCPU64144k
Host1110k
Host4326k
Host16308k
Host64217k
How to read it. The line holds near the size's capacity whatever the fan-out. When an observer's target cannot keep up, its queue fills and the excess is dropped, counted as OVERFLOW; the tooltip says how much arrived.

The observed calls, by observers per function

Calls per second of an observed function, by observers per function, per machine size
MachineObservers per functionCalls per second
1 vCPU040k
1 vCPU126k
1 vCPU49.8k
1 vCPU166.3k
1 vCPU645.1k
2 vCPU081k
2 vCPU143k
2 vCPU435k
2 vCPU1634k
2 vCPU6429k
4 vCPU0135k
4 vCPU178k
4 vCPU430k
4 vCPU1647k
4 vCPU6444k
Host0251k
Host1222k
Host488k
Host1670k
Host6458k
How to read it. Calls slow down as observers are added because they share the engine's CPU with the activations, not because an activation waits for anything: the caller has its answer before any observer runs.

Delivery

How much it delivers in full

A source like cron or a database calls its event function (cron::on-tick, database::on-row-changed) for each event, and systema delivers each call to the observers whose args and filters match. We offer a steady rate, from 1k to 500k events a second over 16 observers, and count what arrives.

Up to some rate, everything does. That rate is the size’s sustained capacity, and past it the engine does not fall over: it delivers what it can and drops the rest, counted.

SizeSustainedPast it
1 vCPU · 512 MiB50 k/s75% of 100 k/s
2 vCPU · 1 GiB50 k/s74% of 100 k/s
4 vCPU · 2 GiB100 k/s61% of 200 k/s
Host · 8 CPUs100 k/s90% of 200 k/s

Activations delivered, by events offered

Per second, over 16 observers, on logarithmic scales: a line on the diagonal delivers everything.

Observer activations delivered per second, by events offered per second, per machine size
MachineEvents offered per secondActivations delivered per second
1 vCPU1k999
1 vCPU5k5k
1 vCPU20k20k
1 vCPU50k50k
1 vCPU100k69k
2 vCPU1k1000
2 vCPU5k5k
2 vCPU20k20k
2 vCPU50k50k
2 vCPU100k68k
4 vCPU1k1000
4 vCPU5k5k
4 vCPU20k20k
4 vCPU50k50k
4 vCPU100k100k
4 vCPU200k115k
Host1k1000
Host5k5k
Host20k20k
Host50k50k
Host100k100k
Host200k177k
How to read it. On the dashed line, everything offered arrives. Where a size's line leaves it, its engine is full: it keeps delivering at capacity, and the excess is dropped and counted instead of queueing without bound. While the observers' queues are full, the engine stops reading the reporting worker, whose SDK drops what it cannot hold (BUFFER_FULL).

Filters

Filters at scale

Observers usually filter: a cron expression, a table, a tenant. Here one function has 16 to 4,000 observers, each filtering on a different value, and every call matches exactly one of them.

What grows is the filtering, not the deliveries: the engine checks every observer’s filter on every call to the function.

Calls per second, by filtered observers on the function

Calls per second of a function with filtered observers, by number of observers, per machine size
MachineFiltered observersCalls per second
1 vCPU1620k
1 vCPU25619k
1 vCPU1,02414k
1 vCPU4,0005.6k
2 vCPU1640k
2 vCPU25636k
2 vCPU1,02429k
2 vCPU4,00024k
4 vCPU1660k
4 vCPU25659k
4 vCPU1,02464k
4 vCPU4,00058k
Host16153k
Host256159k
Host1,024164k
Host4,000124k
How to read it. Each filter check costs about 16 ns in the in-process benchmark, so a handful is free and thousands on one hot function are not. The same thousands spread over many functions cost nothing extra: only the called function's observers are checked.

Memory

What it holds, and how much fits

The engine’s resident memory at rest as idle workers connect and observers pile up. Each VM size’s memory cap is drawn across: where a line would cross it, that size runs out.

SizeIdle workersObservers
1 vCPU · 512 MiB~2,900 fit~46,000 fit
2 vCPU · 1 GiB~6,300 fit~99,000 fit
4 vCPU · 2 GiB~12,600 fit~262,000 fit
Host · 8 CPUs168 KiB each8 KiB each

Fits: the size’s cap, less the engine at rest, over the measured cost of one. Real workers register more functions, and real observers carry bigger filters.

Engine memory, by idle workers connected

Each worker announces itself and registers two functions, then only answers pings.

Engine resident memory by idle workers connected, per machine size
MachineIdle workersResident memory
1 vCPU038 MiB
1 vCPU10058 MiB
1 vCPU1,000200 MiB
2 vCPU038 MiB
2 vCPU10058 MiB
2 vCPU1,000201 MiB
2 vCPU5,000825 MiB
4 vCPU038 MiB
4 vCPU10059 MiB
4 vCPU1,000204 MiB
4 vCPU5,000837 MiB
4 vCPU10k1.6 GiB
Host043 MiB
Host10063 MiB
Host1,000213 MiB
Host5,000864 MiB
Host10k1.6 GiB
How to read it. Memory grows in a straight line with the workers: the slope is the cost of one, and it is nearly the same on every size. Where a size's cap crosses the line is how many idle workers it holds.

Engine memory, by observers held

Spread over 16 functions, a small filter each, 1,000 per observing worker.

Engine resident memory by observers held, per machine size
MachineObservers heldResident memory
1 vCPU039 MiB
1 vCPU1,00052 MiB
1 vCPU4,00077 MiB
1 vCPU16k205 MiB
2 vCPU040 MiB
2 vCPU1,00050 MiB
2 vCPU4,00078 MiB
2 vCPU16k198 MiB
2 vCPU64k678 MiB
4 vCPU040 MiB
4 vCPU1,00050 MiB
4 vCPU4,00073 MiB
4 vCPU16k183 MiB
4 vCPU64k530 MiB
Host044 MiB
Host1,00053 MiB
Host4,00076 MiB
Host16k175 MiB
Host64k542 MiB
How to read it. Again a straight line. A line that stops short crossed its size's cap: the engine was stopped by the kernel there, and the label says so.

Overload

When the targets are slow

Observers whose target takes 20 ms, behind calls that come far faster. Each observer runs 64 activations at a time and queues up to 1,024 more; past that, events are dropped and counted.

The calls hold at 64 in flight while slow observers are added: 4, 64, 256. Every one of them keeps a full queue. A queued event is held once, however many observers wait on it, and all the queues together are capped at 64 MiB of events.

Peak engine memory, by slow observers

Peak engine memory by slow observers with full queues, per machine size
MachineSlow observers on the functionPeak resident memory
1 vCPU451 MiB
1 vCPU64104 MiB
1 vCPU256278 MiB
2 vCPU452 MiB
2 vCPU64123 MiB
2 vCPU256329 MiB
4 vCPU453 MiB
4 vCPU64137 MiB
4 vCPU256516 MiB
Host464 MiB
Host64170 MiB
Host256553 MiB
How to read it. The queues are bounded per observer and all together, so what still grows with the number of slow observers is their activations in flight, 64 at a time for each. The calls themselves keep flowing; the slow observers lose events, every one counted as OVERFLOW in the observer's stats.

Local calls

Calls that never leave the worker

A worker calling a function it hosts runs it in its own process. The engine hears about such a call only when an observer’s filter matches it, in batched reports.

16 tasks in one worker call its own function as fast as they can: unobserved, with an observer matching 1% of the calls, and with one matching them all.

Local calls per second

2 vCPU · 1 GiB

  • Unobserved4.7 M/s

    engine 0.00 CPU · nothing on the wire

  • Observed, 1% match3.6 M/s

    engine 0.95 CPU · 36 k/s activations delivered

  • Observed, 100% match2.7 M/s

    engine 1.12 CPU · 41 k/s activations delivered

How to read it. Unobserved, the engine does nothing at all: no frame reaches it. Observed, it pays only for the reports and activations of the calls that match.

Micro-benchmarks

Single paths, every release

Criterion timings of single code paths, in one process, on every release since v0.11.6: a standard GitHub runner, 4 cores. Lower is faster; each line is a benchmark’s history, and its last point is v0.25.0-rc.2, measured Oct 19, 2026.

WebSocket round trip

One call from a worker, through the engine, to another worker and back, over real sockets.

  • invoke_echo
    91 µs

Concurrent calls

N calls at once to a function hosted in the engine, waiting for all of them.

  • 1
    2.2 µs
  • 8
    16 µs
  • 32
    68 µs
  • 128
    269 µs

Payload size

One call carrying 1 KB to 1 MB of JSON.

  • 1kb
    1.5 µs
  • 10kb
    2.1 µs
  • 100kb
    7.4 µs
  • 1mb
    68 µs

Observer fan-out

One call delivered to N observers at once.

  • 1
    22 µs
  • 8
    42 µs
  • 32
    114 µs
  • 128
    436 µs

Registration churn

Registering and removing functions, workers and observers by the hundred and the thousand.

  • functions_register_remove/100
    42 µs
  • functions_register_remove/1000
    416 µs
  • functions_register_remove/5000
    2.2 ms
  • triggers_register_unregister/100
    108 µs
  • triggers_register_unregister/1000
    1.1 ms
  • triggers_register_unregister/5000
    5.7 ms
  • workers_register_unregister/100
    214 µs
  • workers_register_unregister/1000
    7 ms
  • workers_register_unregister/5000
    48 ms
Core runtime1 benchmark

Calling a handler registered in the engine.

  • engine_call_registered_handler
    1.9 µs
Key-value contention4 benchmarks

N writers on the same key at once.

  • 1
    321 ns
  • 4
    1.2 µs
  • 16
    4.9 µs
  • 64
    23 µs
Key-value store10 benchmarks

Reads, writes, updates and listings in the built-in store.

  • delete
    537 ns
  • get_hit
    254 ns
  • get_miss
    87 ns
  • list_100_items
    28 µs
  • list_groups
    208 ns
  • list_keys_with_prefix
    288 ns
  • set_overwrite
    422 ns
  • update_increment
    372 ns
  • update_merge
    759 ns
  • update_set_field
    411 ns
Protocol: decode8 benchmarks

Parsing each wire message from JSON.

  • InvocationResult
    1 µs
  • InvocationResult_Error
    489 ns
  • InvokeFunction
    1 µs
  • Ping
    95 ns
  • Pong
    93 ns
  • RegisterFunction
    455 ns
  • RegisterTrigger
    591 ns
  • WorkerRegistered
    252 ns
Protocol: round trip8 benchmarks

Encoding and decoding each wire message.

  • InvocationResult
    1.4 µs
  • InvocationResult_Error
    669 ns
  • InvokeFunction
    1.3 µs
  • Ping
    107 ns
  • Pong
    107 ns
  • RegisterFunction
    544 ns
  • RegisterTrigger
    773 ns
  • WorkerRegistered
    394 ns
Protocol: encode8 benchmarks

Turning each wire message into JSON.

  • InvocationResult
    324 ns
  • InvocationResult_Error
    189 ns
  • InvokeFunction
    313 ns
  • Ping
    22 ns
  • Pong
    23 ns
  • RegisterFunction
    101 ns
  • RegisterTrigger
    169 ns
  • WorkerRegistered
    136 ns
Pub/sub fan-out4 benchmarks

One message to N subscribers.

  • 1
    22 µs
  • 8
    40 µs
  • 32
    80 µs
  • 128
    265 µs
Queue: concurrent producers4 benchmarks

N producers enqueueing at once.

  • 1
    4.8 µs
  • 4
    17 µs
  • 16
    68 µs
  • 64
    264 µs
Queue: payload size4 benchmarks

Enqueueing jobs of 1 KB to 1 MB.

  • 1kb
    4.3 µs
  • 10kb
    4.8 µs
  • 100kb
    11 µs
  • 1mb
    73 µs
Queue: enqueue1 benchmark

Putting one job on a queue.

  • push_single
    4.9 µs
Worker cleanup9 benchmarks

Tearing down a worker that leaves with its functions and calls in flight.

  • functions/10
    5.1 µs
  • functions/50
    16 µs
  • functions/200
    54 µs
  • invocations/10
    2.3 µs
  • invocations/50
    3.1 µs
  • invocations/200
    6.1 µs
  • triggers/10
    10 µs
  • triggers/50
    38 µs
  • triggers/200
    133 µs
Invocation tracking, concurrent4 benchmarks

The same with N calls at once.

  • 1
    594 ns
  • 8
    4.5 µs
  • 32
    22 µs
  • 128
    91 µs
Invocation tracking2 benchmarks

Recording which worker each call belongs to.

  • add_remove_sequential
    518 ns
  • add_with_200_existing
    520 ns

Method

How we measure

Everything on this page comes from one open-source tool that runs the same way on your machine as in CI, and every number links back to the run that produced it.

microVM · KVM

systema engine

N vCPUs, pinned

cgroup cap M, no swap

⇄ loopback

Load generator

6 vCPUs, 2 GiB

SDK workers and callers

On the host, the engine’s vCPUs and the load generator’s run on different physical cores, so neither ever slows the other.
Machine sizes
Each size is a microVM (libkrun on KVM, the virtualization systema's own sandboxes use). The engine runs pinned to its N virtual CPUs, in a cgroup capped at M with no swap: past it, the kernel stops it, and the run says where. The load generator runs in the same VM on 6 vCPUs and 2 GiB of its own and talks to the engine over the VM's loopback, so it never takes the engine's CPU and nothing proxies their sockets.
The host
The same engine as a plain process, pinned to half of the machine’s cores and not capped: the reference without a VM.
The load
Real SDK workers (the Rust SDK) call, host, observe and report observed calls, as an application would. Idle workers are raw WebSockets, so ten thousand fit in one process.
A step
10 seconds of load after a warm-up, with the engine’s CPU, memory and threads read from /proc every 250 ms. Every scenario starts on a fresh engine.
The engine
As it ships: default configuration, every span recorded, logs at info. It is the Linux glibc build, the one in the Docker image.
What it is not
A shared CI machine, so runs differ by several percent: compare lines within a run, and runs by their trend. The numbers are for this machine; a faster CPU moves every line.

Run it yourself

x86_64 Linux with KVM, in the workers repo next to sandbox-libkrun, against an engine built from systema. The report lands in target/loadtest/.

cd sandbox-libkrun && BUILD_TARGET=x86_64-unknown-linux-gnu scripts/build-init.sh
cargo build --release -p systema-loadtest
target/release/systema-loadtest suite --engine <systema>/target/release/systema --profiles 2x1024 \
  --init target/x86_64-unknown-linux-musl/release/systema-init --krunfw firmware/libkrunfw-linux-x86_64.so

The load test · The pipeline · Raw data · Micro-benchmarks · Docs