Architecture

Zero Trust in financial systems: from mTLS to service mesh

Zero Trust is not a marketing slogan but an operational assumption: every request, including one that comes from inside, needs authentication, authorisation and encryption. Here is how that is applied in practice in trading infrastructure, without stopping the business.

Article summary
  • Zero Trust starts with mTLS and certificate-based service identity — not with a WAF and not with a firewall.
  • Without automation for issuing and renewing certificates, mTLS becomes the largest source of outages in the organisation.
  • A service mesh gives you transparent mTLS; the choice between Istio and Linkerd is decided by team size, not by capability.
  • Authorisation has to be defined per service and per path — mTLS on its own does not stop logical access.
Zero Trust system — encrypted connections between microservices

What has changed since the perimeter model

The perimeter model was comfortable to operate: a firewall at the edge, a VPN for staff, an internal network you could trust. It was cheaper too — one control at the entrance instead of thousands of service identities to issue, validate and revoke. The problem is that the model rested on an organisational assumption rather than a technical one: that whoever is inside is a well-behaved employee who will not attack the system. When that assumption breaks, there is no second control behind the firewall.

The first break is lateral movement. An attacker who lands on one server does not need to breach a firewall — they are already inside, and from there they climb between servers through central administration and shared SSH keys. At Equifax in 2017, which exposed the data of around 147 million people, the ease with which movement went from server to server is the main point. At SolarWinds in 2020 and in the 2016 SWIFT case — about US$81 million in Bangladesh — the pattern repeats: the attacker arrives through a supplier or a legitimate endpoint and benefits from the internal network trusting them.

The second break is the cloud. "Inside" and "outside" are concepts of a physical network; in Kubernetes every pod talks to every pod, and an IP address is a passing detail with no identity behind it. The third break is suppliers and third parties: an integration, a configuration file, a SaaS service, a contractor's access — each of those connections is an internal door that somebody else administers. Together they make the question "is the request coming from inside" meaningless, because a yes does not say who sent it.

From the Jericho Forum to NIST SP 800-207

The idea did not start with NIST. The Jericho Forum was already talking about de-perimeterisation in 2004 — the dismantling of the perimeter — and argued that protection should be attached to data and identity rather than to the network line. It was too early: in those years there were no tools that issued a certificate to every service and renewed it automatically.

NIST SP 800-207 from 2020 turned the idea into a framework: no implicit trust based on network location, every request is evaluated, and the decision is made in a policy decision point (PDP) and enforced at a policy enforcement point (PEP) sitting in the traffic path. The document mandates no product and describes no single architecture, only properties, and you can map an existing system against it without buying anything.

What Zero Trust actually requires, without the slogans: authentication — every request carries a cryptographic identity that can be verified; authorisation — that identity is permitted this action on this resource; encryption — traffic is encrypted between the two ends, not only at the entrance to the organisation. All three on every request, including between two services in the same namespace. If one is missing, this is an internal network with a new name. My test: if I cannot answer "who called what, under which identity, and was it authorised" for a Tuesday at 14:03, the architecture does not exist.

mTLS is the foundation, not the decoration

Ordinary TLS solves one problem: the client knows it is talking to the right server. The server knows almost nothing about the client — an IP address, and sometimes a token at the application layer. With mTLS both sides present a certificate and validate the chain against a certificate authority, and identity moves out of the application's configuration and into the handshake.

It sounds like a small change. The difference between "my service trusts a header called X-Service-Name" and "my service knows which identity was presented in the certificate" is the difference between a system that can be destroyed by one YAML file and a system that requires a private key. The header passes through every proxy, can be forged from any pod, and is usually not checked.

What a handshake actually looks like

In TLS 1.3 the client sends a ClientHello with cipher suites and an ephemeral key. The server replies with EncryptedExtensions, and from there everything is encrypted: the request to present a certificate (CertificateRequest), the server certificate, and a CertificateVerify that proves it holds the private key. The client validates the server's chain, sends its own certificate and the Finished message, and only once both sides have verified do they move on to application data.

Two operational consequences that are easy to miss. First: in TLS 1.3 the client certificate is encrypted, so an observer on the path sees no identities. Second: there is no renegotiation any more — the certificate is chosen during the handshake, and if the proxy does not know which one to present from the SNI it presents nothing and the connection fails. Resumption changes the picture too: on a resumed connection the identity is carried from the PSK rather than from the certificate, so the lifetime of the ticket has to be limited.

Four stages of mTLS between a client service and a server service, with both certificates issued by an internal certificate authority
Four stages, two directions: service A presents a client certificate and service B presents a server certificate, and both were issued by the same internal certificate authority — which is why each side validates the other's chain without calling an external party.

What mTLS gives you is cryptographic identity and encryption in transit. What it does not give you is authorisation: the clearing service with a valid certificate passes the handshake even when the request touches an endpoint that should not be open to it. A valid certificate from a compromised pod will not be stopped by mTLS, because it looks exactly like the certificate of a healthy pod.

Certificate management: cert-manager and Vault

A single certificate is easy to manage. Twenty services across four environments, each with several replicas that churn, is different work. cert-manager watches Certificate objects, talks to an issuer, obtains a certificate, stores it in a Secret and renews it before it expires. The issuer can be an internal CA, Vault or an external CA — in financial infrastructure it is almost always Vault or an internal CA backed by an HSM, because the CA's private key should not sit in a file. I wrote about the secrets themselves in Kubernetes secrets management.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: matching-engine-tls
  namespace: exchange-prod
spec:
  secretName: matching-engine-tls
  duration: 168h
  renewBefore: 24h
  privateKey:
    algorithm: ECDSA
    size: 256
    rotationPolicy: Always
  usages:
    - digital signature
    - server auth
    - client auth
  dnsNames:
    - matching-engine.exchange-prod.svc.cluster.local
  issuerRef:
    name: vault-pki
    kind: ClusterIssuer

Three details matter more than the rest. duration: 168h is seven days — short enough that a stolen certificate loses its value, long enough that the handshake does not become a dependency on the certificate authority. renewBefore: 24h starts renewal when the certificate is 144 hours old and leaves a full day to fix a failure. rotationPolicy: Always rolls the private key as well; without it you swap the envelope and keep the same key.

Automation is not a nice-to-have. A seven-day certificate renews 52 times a year per identity, and at a hundred service identities that is more than five thousand renewals a year. A renewal failure is not an exception but statistics: a network policy that changed, a Vault that went into sealed mode, a permission that was deleted. Without automation each of these is discovered only when the first request fails, and that happens on a Saturday.

Service mesh: Istio versus Linkerd

You can wire mTLS into every service in code. That is an operational mistake: every language, framework and version renews certificates in a different way. A service mesh moves the responsibility down to the infrastructure layer: the proxy presents the certificate and validates the other side, and the application carries on speaking HTTP over localhost.

The result is a uniform service identity, derived not from a host name but from the identity of the workload: in Kubernetes, from the service account. In Istio and Linkerd it is represented as a SPIFFE ID, such as spiffe://sanpu17.internal/ns/payments/sa/ledger-client. That is what lets you write "only this identity" instead of "only whoever comes from this IP address".

Sidecar versus ambient: who pays for the transparency

In the sidecar model a proxy is added to every pod: Envoy in Istio, linkerd2-proxy in Linkerd. The proxy intercepts traffic through iptables rules in the pod's namespace, performs mTLS against the other side's proxy, and passes the request to the application in plaintext over localhost. No code changes, but an extra process per pod and an extra hop on every request.

Istio also offers ambient: an L4 layer in one ztunnel per node, encrypted traffic between them, and an L7 proxy (waypoint) only where HTTP-level policy is needed. Less overhead, but more concepts to understand before you switch it on. Linkerd stays with sidecars and offers a small operational surface: one CLI and less room for error.

DimensionIstioLinkerd
Installation complexityseparate control plane (istiod), many CRDs and gateways; ambient requires installing ztunnel and waypointone CLI and a short install, few CRDs, sidecar by default
Proxy overheadEnvoy: more memory and CPU per pod; with ambient the overhead moves to the nodelinkerd2-proxy is light, lower memory use per pod
Policy modelPeerAuthentication, RequestAuthentication and AuthorizationPolicy, plus OPA integration through ext_authzServer, ServerAuthorization and AuthorizationPolicy; less expressive, less room for error
Monitoring and observabilityEnvoy telemetry, Kiali and integration with Prometheus and Jaegerbuilt-in dashboard and golden metrics per route, less depth at request level
Fit for team sizea dedicated platform team, or a managed control planea small team, even without a service mesh specialist

My comparison is not by feature count but by who maintains it on a Saturday: a team that knows Istio in depth will get more out of it, and a small team will get most of the benefit from Linkerd.

Enforcement: PeerAuthentication and PERMISSIVE

Enforcing mTLS itself is one object. The policy below declares that all traffic entering the payments namespace must use mTLS, and that a pod presenting no certificate gets no connection:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payments-prod
spec:
  mtls:
    mode: STRICT

A mesh-wide policy lives in the root namespace, usually istio-system, and a selector narrows it to a workload. A point that gets less attention: when the control plane falls over, an existing proxy keeps working with its configuration and certificates, but a new pod gets no identity. In other words the mesh is not one point of failure but two, with different behaviour.

PERMISSIVE has one legitimate place: the migration window. It accepts encrypted traffic and plaintext alike, so a service that has not yet received a sidecar keeps working. The problem is that people forget to turn it off: an entry point that presents no identity remains, and every request arriving through it gets an empty source.principal. A policy that relies on principals will not catch it: it will not deny, it will not match, and if the default is allow the request goes through. This is the most dangerous failure I know in a mesh: not a declared hole, but a feeling that there is mTLS while part of the traffic is still in plaintext.

The actual migration: PERMISSIVE, measure the traffic that is still unencrypted (in Istio istio_requests_total with connection_security_policy="none"), fix it, then STRICT with a target date. Without a date, PERMISSIVE stays the default.

Authorisation: from RBAC to OPA

Once you have identity, the remaining question is what it is allowed to do. Most organisations start with RBAC: a "payment-service" role that is permitted to call "ledger". That is a reasonable start that runs out quickly: a role does not distinguish GET /balance from POST /transfers, does not distinguish S$100 from S$10,000,000, and does not distinguish 10:00 from 23:00. Six months later there are dozens of roles, and nobody knows what they permit.

It is worth separating the tools. Kubernetes RBAC is about access to the API server, not about traffic between services — that is the job of NetworkPolicy at L3/L4 and the mesh's AuthorizationPolicy. Both work on identity and path alone. Amount, time and account type need somewhere else, and that is where OPA comes in.

Policy in Rego: time window, ceiling and CIDR

OPA is neither a proxy nor a mesh but a decision engine: you send it JSON input, it runs Rego and returns a decision. The sidecar assembles the input, so it can include the identity from mTLS, the request body and data loaded as bundles.

package payments.authz

import future.keywords.if
import future.keywords.in

# Default: deny. There is no implicit authorisation.
default allow := false

max_amount_usd := 250000

allow if {
	input.method == "POST"
	startswith(input.path, "/v1/transfers")
	trusted_principal
	trusted_range
	within_window
	amount_ok
}

# The identity comes from the sidecar's mTLS, not from an HTTP header that can be forged.
trusted_principal if {
	some principal in data.payments.allowed_principals
	principal == input.source_principal
}

# CIDR is an extra layer, not identity: an address can be obtained, a private key less easily.
trusted_range if {
	some cidr in data.payments.allowed_cidrs
	net.cidr_contains(cidr, input.source_ip)
}

within_window if {
	clock := time.clock(time.now_ns(), "Asia/Singapore")
	minutes := clock[0] * 60 + clock[1]
	minutes >= data.payments.window.open_minutes
	minutes < data.payments.window.close_minutes
}

amount_ok if {
	to_number(input.parsed_body.amount) <= max_amount_usd
}

The three controls differ in nature, and that is the point. trusted_principal is the real control: a service identity that cannot be presented without the private key. trusted_range is a secondary control: it gets in the way of an attacker from outside, but once a pod is compromised the address is exactly the legitimate address. within_window is a business control rather than a security one, and it creates a real problem for a customer who needs to move money at 23:00.

Another layer is OAuth: if the infrastructure uses tokens and not only certificates, RFC 8705 defines how to bind the mTLS certificate into the access token, so that it is tied to the identity it was issued for. This is also where most attacks in payments touch authorisation rather than authentication: BOLA, the first entry in the OWASP API Security Top 10 2023 edition, is the gap between "I am allowed to read an account" and "I am allowed to read this account". I went further into it in API security in payment infrastructure.

What mTLS does not give you: ABAC and ReBAC

Two complementary approaches handle context. ABAC decides by attributes: department, account type, the customer's risk rating, time, device type. It is more flexible than RBAC, but it needs a reliable data source for every attribute, and any attribute that is not maintained turns the policy into a lie. ReBAC decides by relationships: "this operator is authorised on this account because they are assigned to this branch". That is the model that suits distributed authorisation and access to customer accounts.

Both can be expressed in OPA, but the data does not live in OPA. You need a source that produces it and a decision about what happens when it lags, because a request waiting half a second for authorisation data is already a problem on the order path. In practice I prefer ReBAC for account access and leave ABAC for the time window and risk rating.

Policy as code: test it like code

A Rego policy that is not tested is a bug waiting for production. Every new rule needs three tests: a case that allows, a case that denies, and an edge case. opa test runs tests with with input as and with fixtures that mimic bundles, and opa test --coverage shows which lines were not covered. In the pipeline I run the same bundle through conftest against the deployment files.

The second part is version management: a signed bundle, a pinned version, and a rollback in one command — a policy you cannot revert in five minutes is a commitment.

What actually breaks

Zero Trust usually fails not in an attack but in operations. These three failures recur most, and two of them happened to me:

Three failures that reach production

  • A certificate expiring at the weekend. The renewal failed on Thursday night, there is no alert, and at seven on Saturday morning the matching engine cannot connect to the order book. Under STRICT there is no fallback to plaintext, and that is right security-wise, but it turns the certificate into an availability failure point. The first question is not "who revoked the certificate" but "why did nobody know".
  • The certificate authority is unavailable. Vault in sealed mode, or a new network policy that blocked egress to the issuer. Existing connections keep working until the certificate expires, and then every new pod gets stuck, usually during a deployment restart — that is, while recovering from a different incident. This is the least rehearsed failure, and when it happens it blocks the fix itself.
  • A wrong policy that denies a legitimate service. A new Rego rule, or an AuthorizationPolicy with source.principals that does not match the SPIFFE ID, or a policy that was deployed to prod before it ran in staging. Because the system is fail-closed, a typo is an outage. That is not a bug in the tool but the result of a model whose default is deny — and that is exactly the property we asked for.

In all three cases the pattern is the same: a cryptographically correct control with no on-call, no alert and no procedure. That is why I recommend writing the runbook before you switch on STRICT and testing it in a real exercise.

Runbook: alerting, circuit breaking and CA recovery

Three components in the runbook. The first is an alert based on a metric, not on memory: cert-manager exposes certmanager_certificate_expiration_timestamp_seconds. The same alert is defined by tier rather than by one number: short-lived service certificates alert if renewal has not completed within hours of the window opening, and long-lived certificates alert 30 days before expiry.

- alert: MeshCertificateExpiringSoon
  expr: (certmanager_certificate_expiration_timestamp_seconds - time()) / 86400 < 30
  for: 1h
  annotations:
    summary: "managed certificate expires in less than 30 days"
    runbook: "tls/cert-expiry"

The second is circuit breaking at the mesh level: if the order book does not answer, the payment service should fail fast rather than block its thread pool on retries. outlierDetection and maxConnections in a DestinationRule do that at the traffic layer, but they do not replace business limits and do not prevent double charging: they prevent the cascade effect.

The third is recovery of the certificate authority. The root should sit offline, the intermediate can be online with a backup, and the CA keys should sit in an HSM or a KMS with permissions separate from whoever issues certificates. You need an exercise where the CA is completely off and you measure how long it takes to bring it back, including obtaining the permission of whoever operates it. Without that exercise there is no RTO, there is hope. I wrote about the hierarchy itself in HSMs and key management.

Zero Trust without a runbook is a scheduled outage

The combination of STRICT mTLS, fail-closed policy and short-lived certificates is exactly the combination that takes systems down: each component is right on its own, but together they turn every expired certificate and every wrong rule into a full availability incident. If whoever is on call cannot answer within five minutes "which certificate has expired and who issues it", the architecture is not ready for production.

What has to be on call: a certificate dashboard per service, a command that shows the full chain, a controlled break-glass procedure, and automation that restores a certificate instead of a person. A good break-glass is not "switch off mTLS" but a temporary, limited permission that is written to the audit log and closes itself.

A 90-day implementation plan

I do not believe in two-year plans. A quarter is enough to bring one domain to STRICT mTLS with one authorisation policy, but not the whole organisation. The goal in the first 90 days is not full coverage but proof that you know how to operate this: certificates renew on their own, alerts exist, and someone handles them.

Stages and metrics

Six stages, each with a metric that comes out of it. A metric you cannot measure is a decision, not a metric.

  1. Days 1–15, mapping. Who talks to whom, on which port and which protocol, and who owns each service. Collect real traffic, not a diagram. Metric: the percentage of services with a defined owner, and the number of endpoints still speaking plaintext.
  2. Days 16–30, internal certificate authority. An offline root, an online intermediate, naming based on SPIFFE ID, and an API for issuance. Metric: end-to-end issuance time, and the number of manual issuances, which should tend to zero.
  3. Days 31–45, cert-manager and Vault. One issuer, one Certificate, one non-critical service. Metric: the percentage of renewals that succeeded without intervention, and the average time from creating a Certificate to a ready certificate.
  4. Days 46–60, a mesh in one namespace. Not the whole cluster. Metric: the p99 latency added compared with a direct connection, and the proxy's memory use per pod.
  5. Days 61–75, STRICT mTLS. PERMISSIVE with measurement first, then STRICT with a date. Metric: the percentage of connections with connection_security_policy="mutual_tls", which should be 100% in the domain you chose.
  6. Days 76–90, authorisation. Start with AuthorizationPolicy, add OPA in audit-only mode, and only then enforce. Metric: the number of denials reviewed by hand, and the number of wrong denials.

The most important metric of all does not appear in the list: recovery time. After each stage ask how long it would take to go back if it fails, and who does that at two in the morning. If the answer is "we don't know", the stage is not finished.

What not to do

The most common mistake is installing Istio on three services "to try it out". You get all the operational overhead of a mesh — control plane, a proxy in every pod, upgrades and telemetry — with no real coverage, because most of the traffic is still outside the mesh.

  • Do not install a mesh in a partial deployment without a defined domain. Either a whole namespace or nothing.
  • Do not switch on STRICT across the whole cluster on day one. Start with one domain, with a rehearsed rollback.
  • Do not run mTLS without renewal automation. A seven-day certificate renewed by hand is a scheduled outage.
  • Do not put OPA in the request path before it has run in audit-only mode and you have seen its decisions.
  • Do not give up an alert to "save noise". Noise on call is cheaper than an outage on a Saturday.

What this means for regulation in Singapore

I do not interpret regulation, and this is not the place to do it. But a few frameworks are relevant to anyone building this architecture in Singapore. The MAS Technology Risk Management Guidelines, together with MAS Notice 644 on cyber security management and MAS Notice 655 on cyber hygiene, set out a framework for risk management, for mapping critical systems, for third-party risk and for incident management. MAS is also the integrated regulator for the financial sector, so insurers and capital market entities meet corresponding requirements in their own context. Alongside them sit the PDPC — Personal Data Protection Commission under the Personal Data Protection Act 2012 (PDPA), the Cybersecurity Act 2018, administered by the CSA — Cyber Security Agency of Singapore — and the Payment Services Act 2019.

None of them mandates mTLS or names a product. The framework usually requires the outcome — risk management, operational resilience, audit, incident reporting — not the means. Your advantage: you choose the architecture, but you have to prove that it works. In practice the audit asks three questions: where identities are managed, how you know a certificate has not expired, and what happens when it does.

For anyone with exposure to Europe, DORA adds a layer: ICT risk management, a register of third-party providers, incident reporting and resilience testing. If your counterparty is in the EU, they will live by DORA and it will flow into contracts. And in payments, PCI DSS 4.0.1 is the current version of the standard, and it is relevant to anyone who touches card data. The PCI SSC document library and the MAS website are primary sources I read before relying on a summary.

Key takeaways

  • Zero Trust starts with service identity: mTLS gives every request a cryptographic identity, and that is the foundation for everything else.
  • A certificate without automation is a scheduled outage. cert-manager and an internal issuer are part of the control, not a luxury.
  • A service mesh makes mTLS transparent, but sidecar and ambient are an operational trade-off, not a matter of faith.
  • mTLS authenticates; it does not authorise. Context-aware authorisation — amount, time, identity, relationship — needs a policy engine such as OPA.
  • The real measure of the architecture is recovery time: if the on-call engineer does not know which certificate has expired, there is no Zero Trust.

Sources and links

  1. NIST SP 800-207 — Zero Trust Architecture — the document that defines the model: PDP, PEP and the absence of implicit trust.
  2. SPIFFE — the identity standard for services, the source of the ID scheme used by Istio and Linkerd.
  3. Istio — Security — identity, mTLS and policy, including PeerAuthentication.
  4. Linkerd — Mutual TLS — how Linkerd issues certificates and performs mTLS between proxies.
  5. cert-manager — the documentation for Certificate, issuers and the certificate lifecycle.
  6. HashiCorp Vault — Documentation — the PKI engine, sealing and the issuance of internal certificates.
  7. Open Policy Agent — the decision engine and the Rego language guide.
  8. RFC 8705 — client authentication in OAuth 2.0 using mTLS and certificate-bound tokens.