urMQTT: building an MQTT broker that is also an operating system

Devices in vehicles, in parking lots and in electronics cabinets have one thing in common: they send small messages, rarely, but there are many of them and they are always connected. For years we have been connecting them to MQTT brokers, and for years the same question has bothered us. Why do we need a whole Linux, OpenSSL, a language runtime, SSH, a package manager and only then a broker for something this simple? Each of those layers has its own vulnerabilities, its own memory and its own moving parts, while a fleet of devices needs only three things from a server: to hold a great many mostly idle connections, to authenticate every device reliably, and to carry messages without loss.

That is how urMQTT came about: a unikernel that is an MQTT broker. The kernel, TCP/IP, TLS 1.3, MQTT and an embedded database in a single Rust binary booted from a single ISO. There are no processes, no system calls, no shell. The first platform it is built and tested on is aarch64, and the same code is heading for x86_64, by UEFI and classic BIOS, and for real hardware: the goal is one ISO that boots on all of them. The project is open source under the MIT or Apache 2.0 license, and in this post we want to record where we have got to, how it is put together inside, and what we learned along the way.

What "unikernel" means exactly

A conventional broker is a program that runs on an operating system. urMQTT has no operating system beneath it: it boots itself through UEFI firmware and the Limine bootloader, sets up its own page tables, programs the interrupt controller and the timer, finds virtio devices over PCIe, obtains an address via DHCP and listens on a single port. Everything that would normally be done by the kernel, the drivers, the network stack and the application is here one ELF running at EL1.

A comparison with a typical installation speaks for itself:

	Typical broker on Linux	urMQTT
Exposed services	SSH, the broker, often a metrics or administration port as well	TCP 8883 only (MQTT over TLS 1.3 with mTLS)
Code in the image	kernel, userland, libc, OpenSSL, runtime	a single Rust ELF (plus vendored SQLite)
Remote management	shell, configuration files, package updates	none. A new version is a new ISO
Memory per idle connection	kilobytes (socket buffers, TLS objects, heap)	target ≤ 512 B
Multicore model	threads and locks, or a single event loop	one thread per core, no shared state, run to completion
Code integrity	mappings that are both writable and executable are possible	W^X, locked page tables, PAC/BTI

The priorities are, in this order: security, scalability, multicore performance, small footprint and an immutable installation. Whenever two goals conflict, the one higher on that list wins.

Where we are now

The project is in the pre-alpha phase, but seventeen development milestones have been closed, from M0 to M17. Today urMQTT serves MQTT 3.1.1 over TLS 1.3 with mandatory client certificates and keeps its state in SQLite on a raw disk. In short, here is what it already does:

• Kernel: its own exception vectors with decoded reports, its own page tables with W^X enforced, guard pages, PAC and BTI where the CPU has them, GICv3 and GICv2, the generic timer, SMP up to 64 cores.
• Hardware discovery from the DTB or ACPI into a single platform description, with identical results from both sources.
• Drivers: PCIe ECAM, MSI-X, virtio-net, virtio-blk, virtio-scsi and virtio-rng, and a CSPRNG seeded with hardware entropy.
• Networking: its own DHCPv4, ARP, ICMP and strict IPv4, then its own server-side-only TCP: SYN cookies, a hierarchical timer wheel, NewReno, ACK coalescing, no TIME_WAIT state.
• TLS: TLS 1.3 only, mandatory mTLS, device identity from the CN or a SAN URI; the handshake is done by rustls, followed by our own record layer with AES-128-GCM and ChaCha20-Poly1305. Session resumption with PSK tickets, a revocation list, admission control already at the handshake.
• MQTT: a zero-copy codec, fuzzed and tested with real mosquitto clients; QoS 0 and 1, wildcards, retained messages, last will (LWT), persistent sessions, an ACL restricted to devices/{id}/#, per-device rate limits.
• Storage: SQLite inside the kernel through a VFS over the raw disk, in WAL mode, with an integrity check at boot; a device registry, revocations, retained messages, persistent sessions with message queues for offline devices; a separate per-core telemetry log with a CRC on every record and recovery after a crash.
• Operations and monitoring: a watchdog timer for handlers, a health check, rate-limited logging, syslog and statsd over UDP, admission that survives reconnect storms.
• Delivery: bit-for-bit identical rebuilds verified in CI, an ISO and a raw GPT disk image, minisign signatures, instructions for KVM, Proxmox and Hetzner.

Each step is labeled with the milestone that delivered it, and the file docs/PROGRESS.md in the repository is the single source of truth about what actually works.

Architecture in three pictures

The whole design is described in docs/ARCHITECTURE.md, and the reasons for every major decision are recorded as ADRs in docs/DECISIONS.md. Here is an overview.

Layers

From top to bottom: the broker (sessions, topic tree, QoS, retained messages, wills, ACL, limits), the MQTT codec, TLS, TCP, the network layer, and then the drivers. Alongside that main path sits storage (SQLite and the telemetry log), and underneath everything are memory management, platform discovery and the configuration, which is compiled into constants at build time, together with the embedded CA certificate, the server certificate and the key. The hypervisor with its virtio devices is the only thing outside the image.

One thread per core, no shared state

Each data core runs its own loop with its own TCP stack, TLS states, a shard of the broker (sessions and the topic tree), a heap cache and a telemetry partition. The cores share no mutable state, only bounded lock-free rings, one for each ordered pair of cores. The highest core runs SQLite and nothing else.

The network card currently has a single pair of queues, held by core 0. It steers every MQTT flow to one of the worker cores according to the Toeplitz hash of its four-tuple, and sends out whatever the worker cores return. A session lives on its home core, determined by the hash of the client identifier, so takeover, wills and durability never cross cores. A connection that lands on another core keeps its TCP and TLS there, and forwards plain MQTT to the home core with credit-based flow control. Every routed message is also fanned out to the other shards of the broker, each of which keeps a replica of the retained messages.

Three rules hold this model together. Run to completion: a received packet is processed entirely on one core, from TCP through in-place TLS decryption, MQTT parsing and routing to the encrypted reply, with no task queues and no copying into application buffers. Adaptive polling: under load the receive queues are actively polled, and at idle the core falls back to interrupts and WFI; interrupt handlers only acknowledge the interrupt and mark the work. Watchdog timer: a 10 ms tick on each core catches handlers that exceed their budget.

The life of a PUBLISH

A device sends a TLS record. The network card hands it to the core's loop by descriptor, without copying. The loop checks TCP, decrypts the TLS in place and parses the MQTT from the borrowed buffer, then checks the ACL, the rate limit and the path through the topic tree in memory. If another core is interested in that topic, it receives a routing message with a reference to the payload and the topic identifier; the database receives a telemetry record or a retained-message update, in a batch. The PUBACK reply is encrypted directly into the outgoing buffer and merged with the TCP ACK, so a single segment goes out to the device. For the other core, the message costs one encryption per subscriber at send time.

What a connection costs

The target is at most 512 bytes per idle TLS connection, aiming for 256. While a connection is idle, nothing is held in a buffer for it: received data is processed directly from the network frame, and the send, record and MQTT buffers exist only while something is unfinished or unacknowledged. The kernel measures this itself: exact heap counters in requested bytes and in the blocks they occupy, written every ten seconds as heap above baseline per open connection.

Part	Size	How
TCP control block	96 B	packed flags, 16-bit RTT/RTO, a slot in the timer wheel, no buffers at idle
Slot in the TCP table	4–8 B	only the connection index, table at most half full
Connection and TLS 1.3 record state	152 B	secret and sequence per direction (40 B); key and IV are derived per record
MQTT connection	48 B	identity and will on the heap only while they exist
MQTT session, hot form	64 B	one subscription inline, interned node identifiers
Subscription devices/<id>/cmd/#	0 tree nodes	stored relative to the session identifier, shared by all devices

Measured under HVF with four virtual cores, three of them workers, with every device holding an mTLS connection and one QoS 1 subscription, including allocator rounding:

Connections	Bytes per connection
960	531 B
9,922	494 B
10,197	474 B

The network frame pool is a fixed ~2 MiB, which is roughly 200 B per connection at ten thousand connections and about 2 B at a million. We could not measure beyond that: QEMU's user-mode networking on macOS stopped accepting connections at 10,197, roughly at the limit of 10,240 descriptors of the QEMU process. The figure falls as the fixed parts are spread over more connections, so it does not grow with their number.

Security model

Security is the first priority and wins every conflict with the other goals. The model has several layers:

• One port. TCP 8883, TLS 1.3 only, mandatory client certificates signed by the CA embedded in the image. The MQTT client identifier must equal the device identifier from the certificate.
• No entry points. No shell, no console input, no SSH, no administration API, no dynamic code, no SQL reachable over the network and no filesystem.
• Memory safety. Rust, with forbid(unsafe_code) in the protocol crates; every unsafe block has a // SAFETY: comment that a lint checks, and external input never causes a panic: clippy in the crates on the data path forbids unwrap, expect, indexing and panic!.
• Exploit mitigation. W^X is enforced through descriptor encoding and the SCTLR_EL1.WXN bit and audited on every boot. After boot the page tables are read-only, with no writable alias. The lower half of the address space is disabled. Every stack has guard pages, and exceptions run on a separate stack. PAC and BTI are enabled when the CPU supports them, and smoke tests verify this.
• Abuse resistance. SYN cookies; admission rejects new connections, before any cryptography, when the heap is low, when the database is lagging, when too many handshakes are in flight per core or per source address, or when the handshake rate is too high. Per-device rate limits, bounded queues, a TCP window tied to broker backpressure, rate-limited logging, a watchdog timer and a health check.
• Immutability. The configuration, the CA and the server key are embedded in the image. Changing anything means a new image.

The threat model is recorded in docs/SECURITY.md.

Storage without a filesystem

The data lives on a separate disk, virtio-blk or virtio-scsi, with no filesystem on it. The layout is minimal and our own:

LBA 0      [ superblock: magic · layout version · uuid · region table · CRC-32C ]
LBA 8, 16  [ file size table, slots A/B (the newest valid one wins) ]
LBA 64     [ SQLite main database ½ ][ SQLite WAL ⅛ ][ journal 1 MiB ][ telemetry log: 8 partitions … ]
last 4 KiB [ scratch space for the block device self-test at boot ]

SQLite (vendored 3.54.1, with a digest check at build time) is compiled into the kernel with clang, without the standard library, and uses a fixed memsys5 heap. Its VFS maps three fixed files onto disk regions. It runs in WAL mode with locking_mode=EXCLUSIVE, so it needs no shared-memory index, and with synchronous=FULL; if quick_check finds an error at boot, the system halts safely. The database holds the device registry, revocations, retained messages and persistent sessions with QoS 1 message queues for devices that are not connected.

The database core, the one with the highest number, runs SQLite and nothing else. The data plane sends it state changes over a lock-free SPSC ring; it gathers them into one transaction every 5 ms or every 512 operations and returns the acknowledgments over a second ring. An empty disk is formatted on first boot; anything else is never formatted over.

Telemetry takes a different path. Every accepted PUBLISH on topics from storage.telemetry_topics goes into its core's partition of the telemetry log: a ring of segments with a CRC on every record, salted with the segment sequence so that old passes of the ring never reappear. The cores hand records to the database over their own ring, without copying, and it writes them out every 5 ms or every 256 KiB. At boot, recovery keeps every record that was written out, cuts off the unfinished tail and continues in a new segment. Retention is by size, through the ring, and by age, through retention_days. We verified this with a million records and with kill -9 in the middle of a write.

The numbers

The load tool in tools/loadgen does three things: a storm of TLS 1.3 handshakes with mTLS and CONNECT, long-lived connections with PINGREQ that publish telemetry every second, and reconnect storms. make bench on an Apple M4 Pro under HVF gives:

Worker cores	handshakes/s in the guest	scaling	through QEMU user networking (connections/s)
1	862	1.00×	789
2	1,724	2.00×	636
4	3,438	3.99×	536

The in-guest figure, where the client and the server are inside the same virtual machine and each worker core works on its own, shows that the data plane scales linearly. The network figure is limited by QEMU's single-threaded user networking, not by the guest; this is recorded as ADR-032.

We test reconnect storms by having all devices connect at once, retry rejected attempts with a backoff, and then all drop without a DISCONNECT. The heap_limit_mb= parameter in development mode makes admission treat the heap as smaller than it is, so a storm larger than the heap can be tried with few sockets. In one such round of 3,000 devices, 1,368 connected in 36 seconds and 5,619 attempts were cleanly rejected, with a limit of 8 MiB; the memory returned to baseline after the storm.

How we work

The project is driven by a written specification, PROMPT.md, with a milestone plan, and every milestone ends with the same ritual: make check and make smoke green, a single commit tagged M<n>, an entry in docs/PROGRESS.md and an ADR for every decision that is not obvious. The rules are short and we do not negotiate them: no panics on external input, unsafe only in hardware and FFI crates and always with a comment, no allocations on the hot path, a comment next to every atomic ordering, every limit comes from configuration, and deliberate temporary pieces fail loudly.

CI does more than we could manage by hand: fmt, clippy for the host and for aarch64-unknown-none, typos, actionlint and shellcheck; tests on Linux and macOS on arm64; a kernel build for the development and production configurations with the ELF size in the summary; a boot under QEMU with all smoke-test scenarios and saved serial logs; cargo-deny for advisories, licenses and sources; one minute of fuzzing on each of eleven targets, and ten minutes per target at night with a cached corpus. On top of that come CodeQL, OpenSSF Scorecard, Dependabot and a release with SLSA provenance. Every third-party action is pinned to a full commit SHA.

We take reproducibility literally: make repro builds HEAD twice in fresh worktrees and compares the ELF, the ISO and the raw image byte by byte. Releases are signed with minisign, and the artifacts can be verified with gh attestation verify. The production image contains the server's private key, so the ISO itself is a secret; the images attached to GitHub releases are development ones, with temporary keys, and are meant only for trying things out.

What we learned

A few lessons that changed the plan:

• Our own TCP paid off after all. The prototype ran on smoltcp and served us well up to M13. But a server holding hundreds of thousands of connections does not need a client TCP; it needs SYN cookies, a timer wheel and 96 bytes of state. In M14 we replaced smoltcp with our own server-side-only stack, with simulated-loss tests and segment fuzzing.
• Measure memory from the inside. Per-connection estimates were useless until the kernel started counting the heap in allocator blocks rather than in requested bytes. Only then did we see where the 512 bytes really go, and we brought it down to 494.
• The test tooling is part of the product. The biggest limit was not the kernel but QEMU's user-mode networking and the number of ephemeral ports on the development machine. For a hundred thousand connections and more you need vmnet-shared on macOS, or Linux with KVM, a tap interface and multiple client addresses.
• SQLite without an operating system is possible, but only under strict conditions: a fixed heap, exclusive locking without shared memory, a VFS that knows nothing about files, and admission that does not let the database lag.
• Reconnect storms are the normal state, not the exception. When a mobile network drops and comes back, thousands of devices knock at the same time. That is why admission rejects a connection before any cryptography, and every rejected connection costs almost nothing.

What comes next

Work after M17 is planned in three directions, and the order among them is recorded in docs/PLAN.md. None of this has been built yet.

• One ISO for multiple platforms. The same file urmqtt-<env>-<version>.iso and one raw GPT image that boot on aarch64 UEFI, x86_64 UEFI and x86_64 BIOS, in virtual machines and on real hardware, starting with a single Intel PC. Still an appliance with everything inside, reproducible and signed.
• A raw disk, no filesystem. A byte-level format specification, a host tool urmqtt-disk that shares its parsers with the kernel, region alignment to 1 MiB, growing the disk with the data, discard for thin-provisioned disks, A/B database snapshots, bit-rot checking and, optionally, AES-256-XTS encryption with hardware AES.
• Provably secure. Runtime revocation and a device allowlist over a signed control channel, without a new image; differential and conformance testing of our own TLS and TCP; KASLR, stack protection, kCFI, MTE and a hardened allocator; SQLite isolated at EL0; Secure Boot; SBOM and cargo-vet; and finally a package for an external audit.

Platforms that offer only non-virtio devices, such as AWS ENA or Azure MANA, are not supported for now. Raspberry Pi 4 and 5 as a KVM host, Hetzner CAX and Oracle Ampere A1 are documented and supported in code, but have not yet been tried on the real service.

Try it

The repository is at github.com/zeljan-alduk/urMQTT. With Rust on the pinned nightly toolchain and QEMU, make certs, make run and a single mosquitto client are enough for a first PUBLISH over mTLS on your own machine. The warning from the beginning holds at the end too: this is pre-alpha, and everything that works is written in docs/PROGRESS.md. Bug reports, design questions and security reports following the instructions in SECURITY.md are welcome. We will report back here on each next milestone.