Event-driven architecture in exchanges: Kafka, Flink and the trading data pipeline
Every order that enters an exchange is an event, and every event passes through a chain of matching engines, risk checks and a data pipeline. This article breaks that architecture into layers and shows where the risks sit.
- The order path is a real-time system; the data pipeline is a deferred system — the two must not be mixed.
- FIX remains the standard for client connectivity, but the internal path has moved to SBE or similar binary formats.
- Kafka serves as a journal and distribution layer, not as a matching engine: any latency felt there is a business matter.
- The central risk is consistency: what happens when the event is recorded but the consumer has not acknowledged it.
One order, a million times a second
A single order at an exchange looks like one action: the client sends a buy, the exchange returns an execution report. In practice a chain of decisions passes between those two messages, and each one of them can reject the order, amend it or split it into several fills. I break that chain into layers here, because every layer adds latency and enlarges the attack surface.
The first stage is the client gateway. It authenticates the connection, checks the session sequence number, decodes the message, and verifies that the instrument exists, that the quantity respects the lot size and that the price respects the tick size. Then pre-trade checks come in: a credit line against the clearer, an exposure limit, a price collar around the reference price and a value cap per order. The gateway also throttles the session, timestamps arrival and assigns a running number, and only then is the order sent to the matching engine for that instrument. From there the matching engine is the only component that holds the order book: it matches by price and time priority, emits an execution report for every change, and emits a market data update for every change in the book. The execution report and the update are written to the same event stream, and from there they continue along two entirely different paths — the feed goes out by broadcast, and the executions continue to clearing, settlement and the risk systems.
Latency is measured in microseconds because the price a client displays is a free option it has granted to the market. The longer a quote stands in the queue, the greater the chance that someone picks it off after the market has moved. A slow exchange is not an inconvenience; it is a direct cause of loss for whoever is quoting. And because the quoting producer knows this, it widens its spreads or reduces its sizes, and that is already damage to the whole market. So every hop on the order path gets its own budget rather than one large budget: it is easier to maintain four budgets of tens of microseconds than a single budget of a hundred microseconds.
Hence the distinction the whole article rests on. The order path is a real-time system: correctness there depends on time, and a late answer is a wrong answer. The data pipeline is a deferred system: it must finish, but it need not care when. In analytics, a retry, a backfill and reading the same event twice are routine. On the order path each of them is a fault. The two can be connected, but they must not be confused, and the whole discussion of Kafka and Flink is precisely about that connection.
Trading protocols: FIX, FAST and SBE
The trading protocol is the contract between the client and the exchange, and choosing one is a choice between readability and speed. FIX is tag=value text: 35=D is NewOrderSingle, 35=8 is ExecutionReport, and every field carries its own number. Such a message can be read by eye, it can be reconstructed from a log file, and it carries a session layer with sequence numbers, heartbeats, resend requests (35=2) and gap fills (35=4). That is why FIX has survived since the 1990s: it is boring, and therefore it works. The price is clear: every field arrives as text, which means decoding, number conversions and string allocations on every message; repeating groups are unambiguous only as long as every side interprets them the same way; and a throughput of a million messages per second turns the decoding itself into a bottleneck.
FAST (FIX Adapted for STreaming) was born for exactly that. It encodes the same content model in a binary format, with a template for each message and a delta against the previous message: a field that did not change is simply not sent, so a repeated message takes up almost no space. That suits market data, where most updates repeat the same price with a change in quantity. The price is that the decoder becomes stateful: a lost message does not lose one data point, it loses the ability to decode everything after it until a snapshot arrives.
SBE (Simple Binary Encoding) goes a step further and gives up the dynamic template. The schema defines a fixed-size struct, and a generator tool produces a decoder and an encoder from it. Every field sits at a fixed offset from the start of the message, there are no tags on the wire, no strings and no allocations: the decoder reads directly from the buffer, and in a correct implementation without a copy from the network card’s memory either. A block length and a schema version sit at the head of every message, and that is the only way to add a field without breaking existing readers. The price is the opposite of FIX’s price: nothing on the wire explains itself. The wrong schema version will read garbage instead of throwing an exception, monitoring tools must have a decoder, and every consumer must hold exactly the same schema file.
| Protocol | Format | Typical latency | Main use |
|---|---|---|---|
| FIX | tag=value text, session layer with sequences | tens to hundreds of microseconds to decode a message | client connectivity, order entry, execution reports, post-trade |
| FAST | binary with templates and deltas | single-digit microseconds to decode | previous-generation market data feeds, still present at some firms |
| SBE | fixed-size struct, code generated from a schema | nanoseconds for field access | internal path, modern market data, order entry |
The numbers in the table are orders of magnitude for decoding, not vendor measurements. Even so, FIX remains the standard towards the client, and the reason is social rather than technical: on the client side sit dozens of firms and software vendors, and none of them will regenerate a codec every time the schema changes. The decoding cost of FIX is paid at the edge layer, across hundreds of sessions, and not in the core across millions of messages per second. The internal path moved to binary because the same team and the same schema file sit at both of its ends. The gateway is where FIX ends and becomes SBE, and it is also the trust boundary: the last place that speaks text.
The matching engine
The matching engine is the heart of the system, and in a sense also the simplest part of it: a deterministic state machine whose state is the order book. It does not read from disk in the middle of a match, does not wait for a reply from the network and does not coordinate with another service. It receives an order, changes state and emits events. Everything else happens in the surrounding systems. The diagram below shows the full path and the budget for each hop along the way.
The order book
The order book is not a table in a database, and that is not an aesthetic choice. A B-tree lookup costs whole microseconds, and on the order path that is the entire budget of a hop. The book is a purpose-built memory structure: an array of price levels around the reference price, and within each level a FIFO queue of orders. Adding an order is an append to the tail of a linked list; a cancel is an unlink through a back-pointer that the order itself holds, so it requires no scan. Order objects come from a pre-allocated pool, so there are no allocations and no GC on the hot path.
Above the book sits the market data layer, and it does not recompute the book. It derives a delta from it for every change. That matters for audit: the feed and the executions are born in the same place and in the same order, so the picture the market saw can be reconstructed. An aggregated book (L2) is enough for most consumers; a per-order book (L3) is needed by anyone who wants to know who is standing in the queue.
Determinism and recovery
A deterministic engine is one that always produces the same output from the same ordered input. No system clock inside the match, no threads, no randomness and no dependence on the order in which messages arrived from different networks. Order is decided before the engine, in the sequencer, and what reaches the engine is already an ordered stream with running numbers.
Recovery is therefore a relatively simple operation: keep a journal of inbound orders, keep a snapshot of the book every few minutes or every few events, and on start-up load the snapshot and run the tail of the journal. Recovery time is the length of that tail, so it is measured and capped in advance. The test I demand from any team building such an engine is a replay of a full trading day in a test environment with a hash comparison over the entire output: if the hash is identical, the engine is deterministic in practice and not only in its claims.
A gap in the sequence numbers is a serious event. It cannot be skipped, retrying is forbidden, and matching must not continue. An engine that carries on past a gap produces a book that never happened, and everything built on it is silently wrong. Halting one instrument is an outage; a lying book is continuing damage.
Single writer
Every instrument has a single writer. Not because it is easier, but because that is what makes order describable at all. Two threads matching in the same book need a lock, and a lock is both latency and a source of non-determinism: whoever acquired it first is not necessarily whoever arrived first. So the engine runs in a single thread per instrument, with no locking at all, and concurrency lives between instruments — one instrument’s book does not depend on another’s.
That decision has a consequence for the whole architecture. The real serialisation point is the sequencer, not the book. If order is decided there, the Kafka partition key has to derive from the same decision. A key by client account looks sensible — all of a client’s activity in one place — but it splits the instrument across partitions and breaks exactly the property that is most needed.
Kafka in the data pipeline
Kafka is not a message queue and not a matching engine. It is a replicated log split into partitions, written only at the end and read by offset. The exchange writes every event once, and each consumer reads at its own pace. That is what lets the order path stay in the microsecond range: the engine does not wait for any consumer, does not wait for an acknowledgement from clearing and does not wait for data. It writes, moves on, and consumers catch up without stopping the market.
The log is the source of truth and not just a pipe. If clearing falls over, it does not ask the exchange to send anything again — it reads from where it stopped. If the risk engine comes up in a new version, you run the same offset range through it and get the same results. That is also what makes the log an audit trail: it does not "record" what happened, it is what happened.
The key is the instrument
Kafka guarantees order within a partition and not beyond it. That is not a limitation to work around; it is exactly the guarantee you need. A cancel order has to be read after the order it cancels, and two different instruments do not depend on each other. So the key is the instrument identifier, and every consumer gets the order it needs for free. A key by client account sounds tidy, but it actually splits an instrument across partitions and loses the order that the matching engine decides.
Retention is a business decision, not a technical setting. A topic of order events is retained by time and size and is not compacted, because compaction keeps the last value per key and deletes the history — exactly what must not be deleted. Compaction suits a state topic, such as reference data or "the last known order". Using tiered storage to lower cost does not exempt you from the requirement that the history be kept somewhere that has been audited.
"Exactly-once" is a property of the consumer
No broker provides end-to-end exactly-once, because it does not know what happened on the other side of the network. What you can get: an idempotent producer (enable.idempotence=true) that recognises a retry and preserves order, and a transaction that lets you write to several partitions and mark a consumption offset in the same operation, with read_committed on the reading side. That guarantees the message is written once. It does not guarantee the business action happens once.
Once, on the business side, comes from the consumer, and usually from an identifier rather than from a mechanism. An execution carries a unique ExecID, and the clearing table writes by that key while ignoring duplicates; or you keep a dedupe record by (partition, offset) in the same transaction as the write itself. If no such business identifier exists — for example, a money transfer without an idempotency-key — no Kafka setting will save you. Even acks=all with min.insync.replicas=2 protects against data loss, not against business duplication.
// orders.events.v3 · key = symbol · value = ExecutionReport (Avro, registry v3)
Properties p = new Properties();
p.put("bootstrap.servers", "kafka-a.trading:9093,kafka-b.trading:9093");
p.put("security.protocol", "SSL");
p.put("ssl.keystore.location", "/etc/trading/producer.p12");
p.put("acks", "all"); // an ack from the leader alone is not enough
p.put("enable.idempotence", "true"); // retries keep one copy per record
KafkaProducer<String, ExecutionReport> producer =
new KafkaProducer<>(p, new StringSerializer(), new SpecificAvroSerializer<>());
// the engine never blocks on the broker: journal locally, publish after
ProducerRecord<String, ExecutionReport> rec =
new ProducerRecord<>("orders.events.v3", exec.getSymbol(), exec);
producer.send(rec, (meta, ex) -> {
if (ex != null) journal.markUnpublished(exec.getExecId());
});
One note on that code: it is asynchronous on purpose. The engine records the event in the local journal before it publishes, and the callback only marks a failure for later completion. If the broker is slow, the log is delayed and the market keeps trading. That is the central choice in this architecture.
Flink and real-time checks
Some risk checks cannot run on the gateway’s pre-trade path, or at least not only there. A cumulative daily position limit, exposure across several instruments, and a collar based on the price history of the same second — all of these require holding state over the stream rather than looking at a single order. Those are done in Flink, as a second line of defence.
The "second line" distinction is not semantics. The gateway checks what can be checked in microseconds and rejects the order. The stream checks what accumulates over minutes, alerts, and sometimes sends refreshed limits back to the gateway. If the gateway depends on the stream, any delay in the data pipeline becomes a halt in trading.
State, checkpoints and restarts
A Flink job is effectively one large state that evolves with the stream: position per account, last price per instrument, breach counters, and limits updated from outside. That state is kept in a periodic checkpoint, cut into the stream with barriers and containing the Kafka offsets as well. On recovery the job starts from the checkpoint state rather than from the beginning, so it does not alert again on what it has already alerted. A savepoint is the same mechanism in a deliberate form, before a deploy.
Event time versus processing time
event time is the exchange time inside the message. processing time is the clock of the machine processing it now. For an analysis of "what happened" you always want event time, because it is the only thing that lets you run the same day again and get the same output. The price is waiting for the watermark: to know there are no more events from before a given point, you have to wait for a declared lateness bound. In a risk job that means a one-second window may publish perhaps half a second after it ended — enough for an alert, not enough for an immediate halt.
CREATE TABLE executions (
event_ts TIMESTAMP(3),
account_id STRING,
symbol STRING,
side STRING,
qty BIGINT,
price DECIMAL(18, 4),
WATERMARK FOR event_ts AS event_ts - INTERVAL '200' MILLISECOND
) WITH (
'connector' = 'kafka',
'topic' = 'orders.events.v3',
'properties.bootstrap.servers' = 'kafka-a.trading:9093',
'properties.group.id' = 'risk-checks',
'scan.startup.mode' = 'committed-offset',
'format' = 'avro-confluent',
'avro-confluent.url' = 'http://schema-registry.trading:8081'
);
INSERT INTO risk.alerts
SELECT account_id,
symbol,
SUM(CASE WHEN side = 'BUY' THEN qty ELSE -qty END) AS net_qty,
MAX(price) AS last_price,
TUMBLE_START(event_ts, INTERVAL '1' SECOND) AS window_start
FROM executions
GROUP BY account_id, symbol, TUMBLE(event_ts, INTERVAL '1' SECOND)
HAVING ABS(SUM(CASE WHEN side = 'BUY' THEN qty ELSE -qty END)) > 50000;
For a price collar the query needs to hold a reference price per instrument, and that is done with broadcast state: a small stream of reference prices is updated, and the execution stream is checked against it. This is also where Flink shows its weakness: any state updated from outside is a failure point, because recovery from the checkpoint returns it to the state it had at checkpoint time and not to the current state. That gap is closed with an external source of truth that is read again on start-up, not by assuming the update will arrive.
Where it breaks
The pipeline does not break under average load. It breaks at the moments when something is recovered, upgraded or changed — and that is exactly when everything else has already run out of tolerance. These are the places where I meet this failure in practice.
Replay and backpressure
A replay storm is the state where a consumer resets its offset backwards — after a deploy, after a consumer group change, or after someone "just wanted to check" — and streams two days of events back in. If that consumer is a risk job or a clearing system, it re-sends alerts, writes again, and loads everything behind it. At the same time, a recovering matching engine runs its journal again and emits every event of the day; if the re-publication also goes to Kafka, the two streams meet on the same consumers.
Backpressure in this pipeline does not put a brake on the producer, because a matching engine cannot block. It will not "wait for Kafka", because that is a halt in trading. The right way is a local write before publishing, a bounded queue, and a written policy for what is dropped first: market data before executions, never the reverse. Pressure is measured as lag per partition.
A schema without a registry
Once you have swapped text for binary, the schema is the only contract left. Without a registry and without compatibility checking in the build, an innocent change is interpreted silently on the consumer side: qty in units instead of lots, a side field receiving a third value that is not mapped, or a price moving to a scaled integer. That is the worst shape of failure, because the data looks plausible.
The solution is not "publish the schema in a document". It is a registry with two-way compatibility checking, CI that fails when it breaks, and a version identifier that appears in the event itself. Renaming a topic to .v4 is legitimate, but it splits the history and doubles the storage, so it is a last resort rather than a routine.
The hot partition
A key by instrument gives order, but it also gives uneven distribution. One instrument — an index, an index future, or the hot name of the day — can be most of the messages, and one partition leans on one core. A partition cannot be split without losing order, so the only way is to reduce what goes in: separate topics by asset class, dedicated partitions for the hot instruments, and aggregation in the engine instead of sending every change. The wrong fix is to add consumers; that does not move the partition.
Who may write
This is the part where I am not willing to compromise. A topic of order and execution events is not just another topic; whoever writes to it decides what is traded and what is settled. A producer that forges an ExecutionReport can create a fill that never happened, and if clearing reads from the same stream — the money moves. That is not a data fault, it is fraud with a clean log.
So: a service identity per producer with its own certificate, mTLS between the producer and the broker, and an ACL that permits writes only to the engine’s identity and on one topic only. Risk, control and analytics services get read-only access. There is no shared service account, no admin key in the hands of the development team, and publish quotas so that a broken producer cannot take over the broker. The policy itself is worth managing as code, so that a permission change is reviewed like a code change.
- replay storm: resetting an offset backwards returns whole days of events to the live stream. Bound the ranges and measure in advance how long a full replay takes.
- backpressure: lag per partition is the early sign; the engine does not block — it writes locally and drops market data before executions.
- schema: without a registry and compatibility checking, a field change is interpreted silently. A version in the event and enforcement in CI.
- hot partition: one instrument can carry most of the traffic, and a partition does not split. Aggregation in the engine and routing by asset class.
- permissions: write access to the order topic equals the ability to decide what is traded. A separate identity per producer, a narrow ACL and quotas.
If the log is the source of truth, it is also the evidence. In practice that means: a retention period that covers the audit period, a topic that is not compacted, a schema version identifier on every event, and a read permission separate from the write permission. A hash chain over the messages, with a key managed in an HSM, turns the question "did anyone change the history" into a question that can be answered. In Singapore, the operational accountability framework set by MAS — its Technology Risk Management Guidelines and MAS Notices 644 and 655 — does not dictate which topic to retain, but it certainly asks what happened at any given moment, and that is exactly what the log answers.
A checklist for event-driven architecture
I run this list before I sign off on an order path architecture, and every item on it is a place where I have seen a system break.
- Separate the order path from the data pipeline physically and logically: two models of latency and of failure, and never a dependency of the first on the second.
- Fix one sequencer per instrument, and demand nothing but determinism from the engine — no clock, no threads and no network dependency.
- Keep a local journal before publishing to Kafka, and treat publication as a completion that can be delayed rather than a condition for trading.
- Set
acks=all,enable.idempotence=true, a replication factor of 3 andmin.insync.replicas=2on every producer, and know what that does not give you. - Choose one partition key — the instrument identifier — and document its consequences for order and for load distribution.
- Do not enable compaction on a topic of order events; compaction keeps state, not history.
- Register every schema in a registry, enforce two-way compatibility in CI, and stamp the version into every event.
- Allow write access to the order topic to a single service identity, and check the ACL on every deploy and not only at setup.
- Measure lag per partition, full replay time and p99 per hop — not averages.
- Practise recovery: replay a whole day in a test environment, compare a hash over the output, and measure recovery time against the target.
No item on the list requires a new product; all of them require a decision written down in advance. That is also the difference between an architecture that has been tested and one that is discovered during an incident.
Key takeaways
- The order path is a real-time system and the data pipeline is a deferred system. They connect through the log, but one must never be a condition for the other’s existence.
- FIX stays at the client layer because its cost is paid across hundreds of sessions; the internal path moved to binary because there it is paid on every hop.
- The matching engine is a deterministic function of an ordered stream of orders, and anything that breaks determinism — a clock, a second thread, a network call — is a bug even if the tests pass.
- Kafka gives order within a partition and a single write; business "once" is a property of the consumer and comes from an idempotent identifier.
- The big risk is not performance but permissions: whoever writes to the order topic decides what is traded, and that is a trust boundary, not a convenience setting.
Sources and links
- Apache Kafka — Documentation: the design of the log, the idempotent producer, transactions, retention and compaction.
- Apache Flink — Documentation: checkpoints and savepoints, event time and watermarks, state backends and Flink SQL.
- FIX Trading Community — the FIX and FIXT specifications: the session layer, sequence numbers and resend messages. No direct link, because specification addresses change between versions.
- SGX — Singapore Exchange: a local example of trading infrastructure and the flow of market data.
- MAS — Monetary Authority of Singapore: the Technology Risk Management Guidelines and MAS Notices 644 and 655 on cyber risk management — the operational framework within which the order path and the audit trail sit.
- RFC 8705: OAuth 2.0 client authentication with mTLS and a certificate-bound token — for identifying producers on the event bus.
- Zero Trust in financial systems: from mTLS to service mesh: mTLS and service identity in the inner layers, including the boundary between the gateway and the matching engine.