The note up front stays the same: the project is pre-alpha, the first release is only for VPSs with KVM, and the numbers measure code, tests and specific experiments, not maturity in production.

What it looked like over time

From the previous post (31.10. at 23:00) to today (1.11. at 20:41) the repository has 59 new commits, 178 in total. All of them were made within a single calendar day, so the same warning applies as before: the project is built with Claude Code (commits carry Co-Authored-By: Claude), the plan, code, tests and ADRs are produced in short closed loops, and the numbers below say something about the amount of code and checks, not about how much human work is in them.

The numbers were computed with the same method as in the previous post: all .rs files outside vendor/ and fuzz/, #[test] functions, ## ADR- headings in docs/DECISIONS.md.

Work went on in parallel on several branches, so the order of commits is not strictly the order of work. ADRs are numbered up to 232, but the numbering has gaps (some were renamed when branches were merged); we count headings, not the highest number. Releases: v0.2.0 (1.11., 18:45 UTC) and v0.2.1 (1.11., 20:22 UTC), both marked as pre-release.

H6: SQLite in user mode

What we planned. SQLite is 254,268 lines of C that parse the database, the WAL and the journal from disk, and the disk may be crafted or modified. Until then it ran in the kernel, with kernel privileges. What we built (ADR-210 to ADR-212): SQLite, its libc parts and its heap were moved into a separate program, dbworker, which runs at EL0 (aarch64) or ring 3 (x86_64), in its own address space. The kernel no longer links SQLite. The worker talks to the kernel through a shared memory window and twelve system calls, and every length and offset that comes back is checked before use. A fault in the worker (illegal access, undefined instruction, an exceeded time budget of 30 s) ends the call, the worker is restarted and the database is opened from the WAL as at boot; after 5 restarts in a minute the storage is declared faulty rather than restarted endlessly.

The cost, measured on aarch64 under HVF, the same VM, a kernel with built-in SQLite versus a kernel with the worker, six alternating runs of each:

Disk writes (1,572) and flushes (319) are identical, as they must be. What is not isolated: the kernel still trusts what the worker returns, after checking its shape. A compromised worker can lie (for example, return an empty list of revoked certificates and thereby "un-revoke" them), but it cannot read kernel memory, TLS keys or ticket keys. There is no KPTI, so Spectre/Meltdown-type leaks are not prevented and we claim nothing about them; the PAC key is shared by EL0 and EL1. On x86_64 we did not measure the switching cost on KVM (under TCG the number would be meaningless), and fuzzing of the worker and the protocol itself is not finished. The storage test suite (79 scenarios, each of which runs SQLite in the worker) passes on aarch64 HVF and x86_64 TCG, and 76 on aarch64 TCG (one is skipped because TCG has no MSI-X).

H7 is closed, but not the way we announced. The lower bound on time (ADR-050) and SNTP with consensus (T1) cover the scope of H7; NTS (RFC 8915) remains out of scope and is recorded as a residual risk, not as something that works. We do not consider time security "solved", only its provable part done.

A new network option. Contabo does not give DHCP to a custom ISO, so until now the VPS needed a different image from the lab VM. Now network.mode = "auto" (ADR-214) asks DHCP first, and if there is no lease within the set time (dhcp_timeout_s, default 15 s) or it loses the lease, it applies the configured static address. Tried manually on QEMU (with DHCP it uses the lease, without a network it falls back to the static address after 5 s); there is no smoke scenario for it yet.

X11: a comparison with Mosquitto on the same VPS

The same kind of machine as in the previous post (Contabo, 4 vCPU AMD EPYC, 8 GiB), load from a Linux host 15 ms away, mutual TLS 1.3 with the same certificates on both sides, Mosquitto 2.0.21 on Debian 13 without persistence. The method and decisions are in ADR-213, the raw data in docs/perf-results/. Median of three runs.

What this means and what it does not. The handshakes are lower bounds for both servers: the load generator is a shared VM with 60 to 70 % stolen CPU time and at 128 threads it was close to its own limit, while urMQTT's worker cores were busy only 12 to 13 %. The ratio at 128 clients (about 2.2 times) is the most meaningful; at 32 and 64 it mostly measures clients waiting. The memory numbers do not count the same thing (kernel heap versus process RSS), so "about 50 times less" is an order of magnitude, not a precise ratio. A latency of 14 ms is one network round trip (about 15 ms), so "no measurable broker time"; the tails (p99 around 100 to 200 ms) are dominated by the shared hosts for both brokers. We do not claim differences below about 20 %.

What did not go in our favour and stayed in the tables: a single TCP flow through urMQTT carried 2 to 8 times less than through Mosquitto, with swings from zero to 7.7 thousand messages per second between runs, and the 20 ms handler watchdog cut live connections when the hypervisor stopped a vCPU (in the production setup action = "drop", 6 to 8 connections per run under QoS 0 flooding). Both are described in the next section because they had causes in our code.

Why the single flow was slow: causes, not excuses

The easiest thing would have been to say "shared VPS". We did not; with the same VPS and the same method we looked for causes. Five were found, four in the data path and one in the watchdog (ADR-221 to ADR-224):

The stops are real: on this VPS cores 0 to 2 had a median of 36 to 80 late ticks per 5 seconds, and the worst stop in the window had a median of about 70 ms and went up to 0.9 s (core 3, database and journal: up to 1.55 s), while the host reports no stolen time (host steal 0 us). That is why the watchdog needed a second clock, not a larger budget.

Watchdog: 148 dropped connections in 30 loaded runs before, 0 in 24 runs after; on a local rig that delays and drops frames, at 0.1 % loss a single 64 B flow of 14.3k messages/s (the X11 kernel) now reaches 41.0k, and at 0.5 % from 5.0k to 17.7k.

What this does not say. urMQTT is still not faster than Mosquitto on a single flow: at 64 B it publishes about 80 % of its speed and delivers about 40 %, because it drops QoS 0 when the subscriber's flow is slower than the publisher's (Mosquitto queues without a limit). At 1 KiB in the 16-pair flooding Mosquitto is ahead by about 15 %, which is within noise. The generator was again a shared VM with 58 to 77 % stolen time. The second round (heap locking moved out of the data path, send-side SACK and more) is on the unintegrated branch perf-single-flow-wip; its numbers are not claims yet and we do not give them here.

A promise we did not make, but kept: configuration is data

Until now every configuration was a new build: keys and limits were built into the image. C1 turns that around (ADR-215 to ADR-219): the kernel is built once, and the configuration is data.

• The urmqtt-cfg blob (C1a): a versioned, canonical record (fields in ascending tag order, a SHA-256 fingerprint, an optional detached Ed25519 signature), 70 fields with defaults and bounds, and a decoder that rejects anything malformed.
• The kernel reads it as a Limine module (C1b), before anything else, on aarch64 and x86_64. There is one released kernel for every ENV except dev (the aarch64 ELFs for release, prod and bench have the same SHA-256; verified only on aarch64). The released kernel rejects a development configuration, and without a configuration it prints "refusing to serve" and serves nothing.
• The installed disk carries the configuration and an update keeps it (C1c).
• urmqtt-config and the ISO template (C1d): pack, show, verify, and pack --iso replaces both copies of the configuration slot and the 24 module digests in a finished ISO. The result is byte for byte the same as the ISO that mkiso.sh --config builds. The v0.2.0 release template is 63,111,168 bytes.
• The trust statement (ADR-219): a configured image prints the configuration fingerprint at boot (config: fingerprint ...), and the signature is optional and off by default. A modified ISO has no official digest; the fingerprint is the way to see what is in it.

The builder on aldo.tech

What urmqtt-config pack --iso does, the web page does too, with the same code: the tool library compiled to WebAssembly (1.06 MB, no imports and no unsafe block). The page is at aldo.tech/en/products/urmqtt-builder/ and exists in all 15 languages of the site (for example hr).

• Nothing leaves the browser. Certificates, private keys and the finished ISO stay on your computer; the page downloads only the release files from this site, with no tracking and no third-party scripts. In the live test no request carried key material.
• The release mirror (ADR-227): hour by hour (a systemd timer, low priority) it copies releases from GitHub, checks each one before publishing, and the page in the browser verifies the SHA256SUMS signature with a key embedded in the page, and only then writes the configuration into the template. A failed check blocks the download.
• Simple by default, advanced when needed (ADR-232): the first screen is three steps (where it runs, your certificates, the image name). A single drop field sorts the certificate, key and CA by content. The advanced view opens each of the 70 fields, search, and TOML import and export; both views edit the same state. The template (63 MB) is downloaded while the user fills in the form. In a live trial, with a test mirror and a test key, building the ISO in the browser took 5 to 18 s.

The emulator, optional. Next to "Download ISO" there is "Try in the emulator": QEMU compiled to WebAssembly runs the image you just built in a browser tab. Nothing of it is downloaded before the click (about 3 MB the first time). The emulator is BIOS-only, has no network (so that the user's traffic does not pass through some intermediary and break "nothing leaves the browser"), needs about 1.5 GB of memory and a desktop browser (Chrome tested; Firefox and Safari not). In it urMQTT reaches URMQTT READY in about 8 s (with the countdown skipped) and prints the same configuration fingerprint as the page, which is proof that exactly that configuration was booted. Boot is about 2 times slower than native QEMU on the same Mac (0.82 s against 1.9 s), and cryptography about 58 times slower because the hardware instructions are emulated. That is why we draw no claims about speed from the emulator, and the page says so before it starts. During the build four bytes in the .wasm had to be patched because of a bug in the translation of the POPCNT instruction in the upstream emulator (we have not reported it yet); the emulator itself is under the GPL, with an offer of the source code.

Plain MQTT, only for networks you trust

Plain MQTT (TCP 1883, no TLS) is very common in IoT: it is the default in Mosquitto, in most Arduino and ESP firmware, in Tasmota, Zigbee2MQTT and Home Assistant, and it is used inside segmented industrial networks, behind VPNs and on private LANs, including on devices without a clock (certificates need a date), without enough flash or CPU for TLS. Over an untrusted network it is not acceptable: whoever is on the path reads and modifies the traffic. That is why P1 (ADR-229 to ADR-231) is an explicit, limited choice, off by default, and the TLS port stays; TLS is not turned off by this, the plain port comes alongside it.

• An allowlist of networks is mandatory (allow_from, CIDRs) and is checked at the SYN: outside the list there is no SYN-ACK and no state. The whole internet (0.0.0.0/0) requires a separate explicit option.
• Three login modes: password (the default when enabled; PBKDF2-HMAC-SHA-256, salt, 10,000 iterations, constant-time comparison, lockout per address), anonymous (one configured identity with its own ACL) and by address (ip). The builder hashes the password in the browser and keeps only the digest in memory.
• Fewer privileges than mTLS: a plain client never has service, operator or management privileges, has no $SYS and no console, has its own sessions ($plain/<identity>/<client id>, clean only), and its own limits and the registry of the allowed still bind it.
• What is lost is written in SECURITY.md: confidentiality, integrity, broker authenticity and replay of a recorded login. This cannot be fixed by configuration; that is how it is without TLS.

Measured on a laptop (Apple silicon, HVF, QEMU user-mode network), 16 pairs, 64 B, three runs of 10 s: at 8,000 messages per second the TLS and plain paths deliver all messages with the same latency (p50 0.17 ms against 0.13 to 0.18 ms), new connections per second: TLS about 2,600 to 2,750, anonymous plain about 4,900 to 7,800, and password login is the expensive part: one PBKDF2 costs about 2.5 ms, so the number of new logins is limited to about 200 per data core per second (308 to 342 measured on two cores). Flooding on that machine is not a stable instrument (20k to 440k delivered per second between runs) and we draw no conclusion from it. The comparison of the plain path with Mosquitto on a VPS has not been done yet. Tests: the plaintext smoke group 7 of 7 on aarch64 HVF and on x86_64 TCG, a new fuzz target plain_input (2.3 million runs with no findings).

Bugs found by live use

Two things were not caught by tests, but by the first switch-on at the real site:

• The page looked for the release files one directory too high, so every signature check answered 404 (fix 13af301). The test used a test mirror at a different path.
• The emulator keyboard never reached the guest: the cause was not a bad translation of keys, but that the dialog did not listen for keys at all (fix in ADR-232, with a boot entry that needs no keys).

Lessons

• Look for the cause, do not blame the environment. Most of the single-flow weakness and all of the dropped connections were ours; the hypervisor is real, but it had to be measured, not assumed.
• A comparison with another product must include the losses. A table with only our wins would not be a comparison.
• The measuring instrument has limits. A shared generator with 60 to 70 % stolen time changes what the numbers mean; that is why we write "lower bound" and do not claim differences below 20 %.
• Isolation is not proof. SQLite at EL0 reduces what a fault in it can do, not what it can lie about.
• A real installation finds what tests do not ask for: a wrongly assembled path and keys that nobody listens to.

What is still not proven

• The release signing key is for now a test key; v0.2.0 and v0.2.1 are pre-releases, and the images that come with them are development images with test certificates (except the template, which has no configuration).
• The 13 translations of the builder page (and of this post) were written by AI and checked by tests and spot checks, not by native speakers.
• The comparison with Mosquitto is one session, one path and one shared generator, on a shared VPS; differences below 20 % are not claims.
• The plain path has not been measured on a VPS or against Mosquitto.
• Lighthouse on the new simple view is not confirmed (old view: accessibility 95 or more).
• Bare metal, the x86_64 worker cost on KVM, Firefox and Safari for the emulator: not measured.

What comes next

• D1: a public demo broker. We cannot make the browser emulator public: it lives in one visitor's tab, is 20 to 100 times slower than KVM, has no inbound network and disappears when the tab is closed; routing the internet into someone else's browser would also be abuse. A real demo broker is possible: urMQTT on a VPS behind a name on aldo.tech, with a published test CA, a strict ACL and small limits. For now this is only an idea, not a build, and it needs explicit consent for public exposure.
• The second round for the single flow (branch perf-single-flow-wip): only once it is measured on a VPS and merged.
• The plain path against Mosquitto on a VPS, login and account revocation for plain MQTT over the signed management channel.
• After the release: bare metal (X12 to X16), IOMMU and Secure Boot, disk encryption.

The code is at github.com/zeljan-alduk/urMQTT, and docs/PROGRESS.md is still the only source of truth about what actually works, docs/PERFORMANCE.md about what has been measured. Pre-alpha, as before; bug reports and security reports following the instructions in SECURITY.md are welcome.

One note up front: the project is still pre-alpha. The numbers in this post measure code and tests, not maturity in production, and the first release is deliberately limited to VPSs with KVM. Bare metal is left for later.

What it looked like over time

The first commit was made on 28 October 2026 at 23:48 (M0). By 31 October at 23:00 the repository has 119 commits, distributed as 3, 39, 38 and 39 per calendar day. Let us say right away how that is possible: the project is built with Claude Code, and the commits carry Co-Authored-By: Claude trailers. Four calendar days of history do not mean that one person wrote 90,000 lines in four days; they mean that planning, implementation, tests and ADRs are written in short, closed loops, and every step ends with the same verification ritual as before. The numbers below describe the amount of code and tests, not how much of it has been exercised in real-world work.

The graph shows three curves that grow almost together, which is the intention: tests and recorded decisions keep pace with the code instead of lagging behind it. The slope is steepest on 29 October, when the whole path from the kernel to the broker comes into being. After that, lines are added more slowly, but tests grow faster than code (27 tests on 1.6 thousand lines at the start, 739 on 89.8 thousand today).

Lines are all .rs files outside vendor/ and fuzz/, and tests are #[test] functions. What each part added, by date from the git history:

Promise 1: one ISO for multiple platforms

What we planned. A single file that boots on aarch64 UEFI, x86_64 UEFI and x86_64 BIOS. What we built. The x86_64 port (X0 to X10) and a universal ISO with the Limine bootloader: a single hybrid ISO that boots in all of those combinations; the smoke tests boot it on four virtual machines. The menu has three entries: urMQTT (live mode, runs from RAM, writes nothing to disk), install to the disk of the machine it was started on (the installer writes a GPT and the data, then powers the machine off) and Diagnostics (hardware detection and measurements).

The first real target was a VPS rather than a physical computer: the decision is that the first release supports only VPSs (KVM), with bare metal coming after it. We tested on a Contabo VPS with an AMD EPYC processor (family 23, Zen 1), 4 vCPUs and 8 GiB of RAM, with a virtio-net network and a virtio-scsi disk. The first attempt surprised us: the ISO attached as a virtual CD booted the default entry, the production image refused to serve from read-only media and reset, and we got an endless loop of menu, kernel, reset with nothing legible. Today a fatal error in the production image leaves a message on the console for 60 seconds before the reset, and from optical media it prints that it was booted from read-only media and stops. Live mode in RAM (O3) later worked around that problem for the usual case.

The Diagnostics entry grew out of that experience: a machine we cannot see must be something we can ask what it has. It prints the CPU, RAM, network, disks and PCI devices, measures memory throughput, disk and cryptography, and prints the same measurements as JSON on the serial port. Writing to disk is allowed only in unallocated space and only after you type WIPE; in the screenshot below that was skipped.

What is still open. Real hardware. After the first release comes bare metal (USB boot, NVMe and AHCI, the network card of a real computer), together with an IOMMU and Secure Boot, because without an IOMMU a malicious or faulty device can write over all of memory. In this version we claim nothing about that.

The handshake: from 1.2 ms to 0.24 ms

The first diagnostics measurement on the VPS gave an awkward number: the mTLS handshake, server-side cryptography only, took 1.2 ms. On aarch64 we were already using hardware AES and SHA instructions, but x86_64 had nothing like that for public-key operations: RustCrypto is portable and constant-time, and therefore slow as well. Three changes brought it down.

• For P-256 we took over the nistz256 assembly from OpenSSL and ring (ADR-130), with fixed-base tables for signing and for verifying the CA signature (ADR-110, ADR-111).
• X25519 was rewritten to four 64-bit limbs with MULX/ADX instructions (ADR-131).
• After that we profiled the rest (ADR-132, ADR-200).

The numbers vary by about 10% between runs, so we recorded a range rather than a single value. Overall that is roughly five times less CPU time per handshake.

What we learned about memcpy. Once the cryptography dropped, the profile showed something we did not expect: the mem* functions (memcpy, memmove, memset) took 7% of the handshake, 13 to 39% of the TLS record path (39% for 64 B, 13% for 4 KiB) and 17 to 34% of PUBLISH fan-out. We replaced them with SSE2 assembly for x86_64 (ADR-200), with differential tests against the reference implementation for all sizes, alignments and overlaps. Per call, that is 55 to 75 cycles versus 7 to 12 for 1 to 64 bytes.

Reading the graph: the smaller the record, the larger the share of time that goes to copying, and so the larger the gain. For small messages, which make up most MQTT traffic, that is what counts.

What this does not mean. On native x86_64 we measured OpenSSL 3.5.5 only as a reference for the assembly (ADR-130), not as a product comparison. The comparison of Linux plus Mosquitto on the same VPS (task X11) has not been done yet. So we do not claim that urMQTT is faster than Linux: it has not been measured. The numbers above compare urMQTT with itself.

Overload was the second result. On the VPS, from a single Mac (RTT 35 ms), 128 clients produced 618 handshakes per second, 256 clients 393, and 512 clients 67, with thousands of rejected connections. That was not a plateau but a collapse. We found the causes in the code (two were confirmed in QEMU/HVF and could not be reproduced on the VPS), several at once: a rejected connection sent the client over the next segment carrying a valid SYN cookie, the rejection told the client by RST to retry immediately, and receive was O(queue). The fix (ADR-080 to ADR-082) drops a SYN that would not be accepted, remembers rejected flows, keeps receive at O(1) and counts handshakes per core. It is possible (a hypothesis, not verified) that part of the cause was on the client as well: the load tool closed first, so a single Mac with 16,384 ephemeral ports and a TIME_WAIT of 30 seconds is limited to about 550 connections per second. That is why we have not yet measured the server's true ceiling.

Promise 2: a raw disk, but a careful one

What we planned. To keep our own layout without a filesystem and add everything a filesystem normally provides: alignment, growth, discard, backups and checks. Why not a filesystem. We debated that and the decision was confirmed: urMQTT writes one database, one telemetry log and its own metadata, and needs no directories, permissions or file names, and every additional layer would be more code in the trusted part and one more way for a write to silently disappear. The disk is a GPT with an ESP partition and one urMQTT data partition, on which sits our own layout.

What we built (S2 to S6).

• Layout v2 (S2). Disk topology, alignment to 1 MiB or the optimal I/O size, whole physical blocks, an A/B pair of superblocks with a generation (ADR-140). The kernel rejects layout v1 with a clear message, and the offline tool urmqtt-disk migrate migrates it (ADR-141): a dry run is the default, it copies and only then switches over, with rollback and fault injection on every write.
• Growth (S3). At boot the disk can grow (it never shrinks), and on virtio-blk also at runtime, through the configuration-change interrupt (ADR-142, ADR-143).
• Discard and write-zeroes (S4). Thin-provisioned disks give space back to the host. In the test, a qcow2 file on the host dropped from 62 to 16 MiB after the telemetry segments expired, and recovery never trusts the contents of a discarded range.
• On-disk database backup (S5). An A/B copy using the SQLite online backup API in steps of 16 pages: the longest step is 4.6 ms versus 445 ms when copying in a single call. Commits during the copy had a maximum of 2.4 ms versus 2.2 ms without it. If quick_check fails at boot, the database is restored from the copy.
• Scrubbing and bounded boot (S6). The scrubber reads the disk within a bounded I/O budget and reports corrupted records, and boot reads only the newest segment of each partition, so boot time does not grow with the size of the telemetry region. That closed a problem we had calculated in advance: on a large disk, full recovery would take anywhere from minutes to tens of minutes.

The whole group of storage smoke tests passes on both architectures: 79 passed, 0 failed, on aarch64 under HVF and on x86_64 under TCG. We also measured that S4 did not slow the database down: 53.3 thousand inserts per second in batches before and after, with no measurable difference.

What surprised us. How much of a "simple" layout comes down to edge cases: virtio-scsi and virtio-blk report topology in different ways, and a partial write in the middle of a migration must always leave at least one valid version. That is why every write on the migration and growth path went through fault injection.

What is still open. SQLite pages have no checksum, so a flipped bit inside a value in a row will not be detected by the database; telemetry records and the contents of the copies have a CRC and are detected. Disk encryption (S7) is not done.

Promise 3: provably secure, as far as we can manage

Hardening (H4). The allocator wipes freed memory and checks it on reuse, detects double frees, links free-list pointers with XOR and zeroes keys when they are freed (ADR-170). Stack protection (-Zstack-protector=strong, 1315 functions, and SQLite with -fstack-protector-strong, ADR-171). Error reports in production contain no registers or addresses (ADR-172). KASLR on both architectures: the kernel is a PIE and Limine shifts it by about 19 bits, and the heap, stacks and MMIO window are placed randomly (ADR-173). On aarch64, MTE tags the heap (ADR-174), but only under TCG: HVF and x86 do not have it. We evaluated kCFI and did not adopt it because the compiler on the pinned nightly crashes with an internal error; the IOMMU is deferred along with bare metal (ADR-175, ADR-176).

The cost is measured, not assumed. On the handshake path (dp-bench, aarch64 HVF, full TLS 1.3 + mTLS + CONNECT handshakes):

Overall, then, the cost on the handshake path is about 5% for the parts measured cleanly, and with PIE and tags it comes to about 7% on one core and about 11% on four, though that measurement is noisy. We consider it acceptable, but we have not repeated it on an x86_64 VPS.

It is worth comparing with the figure from the first post (862 handshakes per second on one core). The same tool today, with hardware cryptography enabled on aarch64 (X0b) and a series of speedups that have arrived since, gives 1941 before hardening. We have not isolated the contribution of each change, so we claim nothing about the cause.

How it was verified, and what was not. H5 added Miri on a subset of host crates, Kani proofs, sanitizers, a model fuzzer for the ACL and coverage reports. Kani proved record_len in TLS, parse in TCP and two timer-wheel arithmetics. Not proven (timed out): both MQTT decoders, both configuration parsers and the behavior of the timer wheel; the CI job for Kani therefore does not block yet. Miri cannot execute assembly. Supply chain (H9): a CycloneDX SBOM with 215 components, cargo-vet passes with 39 fully audited and 140 exemptions, and cargo-deny is configured to treat vulnerability advisories as a CI failure (those CI jobs have so far been verified only locally, not on GitHub, and the negative test has not been run), together with VULNERABILITY-HANDLING.md. The honest state: of 67 kernel-linked third-party crate versions, 53 are exemptions rather than audits. Vet proves that the set of dependencies cannot change without a visible difference, not that the TCB has been reviewed. We have not written our own audit of any crate.

SNTP (T1) is a small security story in that sense: it is optional, only toward private IPv4 servers, with a maximum step and the condition that the sources agree (median, majority within one second). A clock that anyone on the path can move is not a clock we may trust for certificate validity.

On the Mac: aarch64 natively and x86_64 by emulation

We develop on a Mac, which has two ways of running the same code, and they are very different.

The rule is simple: numbers under TCG are emulation and we do not cite them as performance. We measure x86 performance only on a real VPS. That is why the cryptography numbers above come from the Contabo VPS and not from the Mac.

Lessons

• Measure on a real machine before optimizing. The first approach was to optimize what we thought was slow; only diagnostics on the VPS showed 1.2 ms and exactly where the time goes.
• Overload is a separate feature. A system that is fast under load can collapse above it; rejecting must be cheaper than accepting.
• The profile surprises. The biggest single gain for small messages was not in the cryptography but in memory copying.
• A disk without a filesystem takes more work, not less. What a filesystem normally provides (alignment, growth, discard, copies, checks) we have to write and verify ourselves, every step with fault injection.
• Honesty in numbers. Memory measurements on a shared VPS vary up to 10 times between runs and say nothing about the code, so we do not cite them. Crypto numbers are stable to about 10%.
• Development speed is not maturity. Tests and ADRs grow alongside the code, but four days is not real production.

What comes next

• H6: SQLite isolated at EL0, so that a bug in the database does not open up the kernel.
• H7: the rest of time security (NTS and step handling); SNTP is only the first step.
• X11: a comparison with Linux and Mosquitto on the same VPS; only then does it make sense to talk about relative speed.
• Release: the first public release for VPSs (KVM), with signed artifacts and instructions.
• After the release: bare metal (X12 to X16), IOMMU and Secure Boot, and disk encryption.

The code is at github.com/zeljan-alduk/urMQTT, and docs/PROGRESS.md is still the single source of truth about what actually works. Pre-alpha, as before; bug reports and security reports following the instructions in SECURITY.md are welcome.

That is how urMQTT came about: a unikernel that is an MQTT broker. The kernel, TCP/IP, TLS 1.3, MQTT and an embedded database in a single Rust binary booted from a single ISO. There are no processes, no system calls, no shell. The first platform it is built and tested on is aarch64, and the same code is heading for x86_64, by UEFI and classic BIOS, and for real hardware: the goal is one ISO that boots on all of them. The project is open source under the MIT or Apache 2.0 license, and in this post we want to record where we have got to, how it is put together inside, and what we learned along the way.

What "unikernel" means exactly

A conventional broker is a program that runs on an operating system. urMQTT has no operating system beneath it: it boots itself through UEFI firmware and the Limine bootloader, sets up its own page tables, programs the interrupt controller and the timer, finds virtio devices over PCIe, obtains an address via DHCP and listens on a single port. Everything that would normally be done by the kernel, the drivers, the network stack and the application is here one ELF running at EL1.

A comparison with a typical installation speaks for itself:

The priorities are, in this order: security, scalability, multicore performance, small footprint and an immutable installation. Whenever two goals conflict, the one higher on that list wins.

Where we are now

The project is in the pre-alpha phase, but seventeen development milestones have been closed, from M0 to M17. Today urMQTT serves MQTT 3.1.1 over TLS 1.3 with mandatory client certificates and keeps its state in SQLite on a raw disk. In short, here is what it already does:

• Kernel: its own exception vectors with decoded reports, its own page tables with W^X enforced, guard pages, PAC and BTI where the CPU has them, GICv3 and GICv2, the generic timer, SMP up to 64 cores.
• Hardware discovery from the DTB or ACPI into a single platform description, with identical results from both sources.
• Drivers: PCIe ECAM, MSI-X, virtio-net, virtio-blk, virtio-scsi and virtio-rng, and a CSPRNG seeded with hardware entropy.
• Networking: its own DHCPv4, ARP, ICMP and strict IPv4, then its own server-side-only TCP: SYN cookies, a hierarchical timer wheel, NewReno, ACK coalescing, no TIME_WAIT state.
• TLS: TLS 1.3 only, mandatory mTLS, device identity from the CN or a SAN URI; the handshake is done by rustls, followed by our own record layer with AES-128-GCM and ChaCha20-Poly1305. Session resumption with PSK tickets, a revocation list, admission control already at the handshake.
• MQTT: a zero-copy codec, fuzzed and tested with real mosquitto clients; QoS 0 and 1, wildcards, retained messages, last will (LWT), persistent sessions, an ACL restricted to devices/{id}/#, per-device rate limits.
• Storage: SQLite inside the kernel through a VFS over the raw disk, in WAL mode, with an integrity check at boot; a device registry, revocations, retained messages, persistent sessions with message queues for offline devices; a separate per-core telemetry log with a CRC on every record and recovery after a crash.
• Operations and monitoring: a watchdog timer for handlers, a health check, rate-limited logging, syslog and statsd over UDP, admission that survives reconnect storms.
• Delivery: bit-for-bit identical rebuilds verified in CI, an ISO and a raw GPT disk image, minisign signatures, instructions for KVM, Proxmox and Hetzner.

Each step is labeled with the milestone that delivered it, and the file docs/PROGRESS.md in the repository is the single source of truth about what actually works.

Architecture in three pictures

The whole design is described in docs/ARCHITECTURE.md, and the reasons for every major decision are recorded as ADRs in docs/DECISIONS.md. Here is an overview.

Layers

From top to bottom: the broker (sessions, topic tree, QoS, retained messages, wills, ACL, limits), the MQTT codec, TLS, TCP, the network layer, and then the drivers. Alongside that main path sits storage (SQLite and the telemetry log), and underneath everything are memory management, platform discovery and the configuration, which is compiled into constants at build time, together with the embedded CA certificate, the server certificate and the key. The hypervisor with its virtio devices is the only thing outside the image.

One thread per core, no shared state

Each data core runs its own loop with its own TCP stack, TLS states, a shard of the broker (sessions and the topic tree), a heap cache and a telemetry partition. The cores share no mutable state, only bounded lock-free rings, one for each ordered pair of cores. The highest core runs SQLite and nothing else.

The network card currently has a single pair of queues, held by core 0. It steers every MQTT flow to one of the worker cores according to the Toeplitz hash of its four-tuple, and sends out whatever the worker cores return. A session lives on its home core, determined by the hash of the client identifier, so takeover, wills and durability never cross cores. A connection that lands on another core keeps its TCP and TLS there, and forwards plain MQTT to the home core with credit-based flow control. Every routed message is also fanned out to the other shards of the broker, each of which keeps a replica of the retained messages.

Three rules hold this model together. Run to completion: a received packet is processed entirely on one core, from TCP through in-place TLS decryption, MQTT parsing and routing to the encrypted reply, with no task queues and no copying into application buffers. Adaptive polling: under load the receive queues are actively polled, and at idle the core falls back to interrupts and WFI; interrupt handlers only acknowledge the interrupt and mark the work. Watchdog timer: a 10 ms tick on each core catches handlers that exceed their budget.

The life of a PUBLISH

A device sends a TLS record. The network card hands it to the core's loop by descriptor, without copying. The loop checks TCP, decrypts the TLS in place and parses the MQTT from the borrowed buffer, then checks the ACL, the rate limit and the path through the topic tree in memory. If another core is interested in that topic, it receives a routing message with a reference to the payload and the topic identifier; the database receives a telemetry record or a retained-message update, in a batch. The PUBACK reply is encrypted directly into the outgoing buffer and merged with the TCP ACK, so a single segment goes out to the device. For the other core, the message costs one encryption per subscriber at send time.

What a connection costs

The target is at most 512 bytes per idle TLS connection, aiming for 256. While a connection is idle, nothing is held in a buffer for it: received data is processed directly from the network frame, and the send, record and MQTT buffers exist only while something is unfinished or unacknowledged. The kernel measures this itself: exact heap counters in requested bytes and in the blocks they occupy, written every ten seconds as heap above baseline per open connection.

Measured under HVF with four virtual cores, three of them workers, with every device holding an mTLS connection and one QoS 1 subscription, including allocator rounding:

The network frame pool is a fixed ~2 MiB, which is roughly 200 B per connection at ten thousand connections and about 2 B at a million. We could not measure beyond that: QEMU's user-mode networking on macOS stopped accepting connections at 10,197, roughly at the limit of 10,240 descriptors of the QEMU process. The figure falls as the fixed parts are spread over more connections, so it does not grow with their number.

Security model

Security is the first priority and wins every conflict with the other goals. The model has several layers:

• One port. TCP 8883, TLS 1.3 only, mandatory client certificates signed by the CA embedded in the image. The MQTT client identifier must equal the device identifier from the certificate.
• No entry points. No shell, no console input, no SSH, no administration API, no dynamic code, no SQL reachable over the network and no filesystem.
• Memory safety. Rust, with forbid(unsafe_code) in the protocol crates; every unsafe block has a // SAFETY: comment that a lint checks, and external input never causes a panic: clippy in the crates on the data path forbids unwrap, expect, indexing and panic!.
• Exploit mitigation. W^X is enforced through descriptor encoding and the SCTLR_EL1.WXN bit and audited on every boot. After boot the page tables are read-only, with no writable alias. The lower half of the address space is disabled. Every stack has guard pages, and exceptions run on a separate stack. PAC and BTI are enabled when the CPU supports them, and smoke tests verify this.
• Abuse resistance. SYN cookies; admission rejects new connections, before any cryptography, when the heap is low, when the database is lagging, when too many handshakes are in flight per core or per source address, or when the handshake rate is too high. Per-device rate limits, bounded queues, a TCP window tied to broker backpressure, rate-limited logging, a watchdog timer and a health check.
• Immutability. The configuration, the CA and the server key are embedded in the image. Changing anything means a new image.

The threat model is recorded in docs/SECURITY.md.

Storage without a filesystem

The data lives on a separate disk, virtio-blk or virtio-scsi, with no filesystem on it. The layout is minimal and our own:

LBA 0      [ superblock: magic · layout version · uuid · region table · CRC-32C ]
LBA 8, 16  [ file size table, slots A/B (the newest valid one wins) ]
LBA 64     [ SQLite main database ½ ][ SQLite WAL ⅛ ][ journal 1 MiB ][ telemetry log: 8 partitions … ]
last 4 KiB [ scratch space for the block device self-test at boot ]

SQLite (vendored 3.54.1, with a digest check at build time) is compiled into the kernel with clang, without the standard library, and uses a fixed memsys5 heap. Its VFS maps three fixed files onto disk regions. It runs in WAL mode with locking_mode=EXCLUSIVE, so it needs no shared-memory index, and with synchronous=FULL; if quick_check finds an error at boot, the system halts safely. The database holds the device registry, revocations, retained messages and persistent sessions with QoS 1 message queues for devices that are not connected.

The database core, the one with the highest number, runs SQLite and nothing else. The data plane sends it state changes over a lock-free SPSC ring; it gathers them into one transaction every 5 ms or every 512 operations and returns the acknowledgments over a second ring. An empty disk is formatted on first boot; anything else is never formatted over.

Telemetry takes a different path. Every accepted PUBLISH on topics from storage.telemetry_topics goes into its core's partition of the telemetry log: a ring of segments with a CRC on every record, salted with the segment sequence so that old passes of the ring never reappear. The cores hand records to the database over their own ring, without copying, and it writes them out every 5 ms or every 256 KiB. At boot, recovery keeps every record that was written out, cuts off the unfinished tail and continues in a new segment. Retention is by size, through the ring, and by age, through retention_days. We verified this with a million records and with kill -9 in the middle of a write.

The numbers

The load tool in tools/loadgen does three things: a storm of TLS 1.3 handshakes with mTLS and CONNECT, long-lived connections with PINGREQ that publish telemetry every second, and reconnect storms. make bench on an Apple M4 Pro under HVF gives:

The in-guest figure, where the client and the server are inside the same virtual machine and each worker core works on its own, shows that the data plane scales linearly. The network figure is limited by QEMU's single-threaded user networking, not by the guest; this is recorded as ADR-032.

We test reconnect storms by having all devices connect at once, retry rejected attempts with a backoff, and then all drop without a DISCONNECT. The heap_limit_mb= parameter in development mode makes admission treat the heap as smaller than it is, so a storm larger than the heap can be tried with few sockets. In one such round of 3,000 devices, 1,368 connected in 36 seconds and 5,619 attempts were cleanly rejected, with a limit of 8 MiB; the memory returned to baseline after the storm.

How we work

The project is driven by a written specification, PROMPT.md, with a milestone plan, and every milestone ends with the same ritual: make check and make smoke green, a single commit tagged M<n>, an entry in docs/PROGRESS.md and an ADR for every decision that is not obvious. The rules are short and we do not negotiate them: no panics on external input, unsafe only in hardware and FFI crates and always with a comment, no allocations on the hot path, a comment next to every atomic ordering, every limit comes from configuration, and deliberate temporary pieces fail loudly.

CI does more than we could manage by hand: fmt, clippy for the host and for aarch64-unknown-none, typos, actionlint and shellcheck; tests on Linux and macOS on arm64; a kernel build for the development and production configurations with the ELF size in the summary; a boot under QEMU with all smoke-test scenarios and saved serial logs; cargo-deny for advisories, licenses and sources; one minute of fuzzing on each of eleven targets, and ten minutes per target at night with a cached corpus. On top of that come CodeQL, OpenSSF Scorecard, Dependabot and a release with SLSA provenance. Every third-party action is pinned to a full commit SHA.

We take reproducibility literally: make repro builds HEAD twice in fresh worktrees and compares the ELF, the ISO and the raw image byte by byte. Releases are signed with minisign, and the artifacts can be verified with gh attestation verify. The production image contains the server's private key, so the ISO itself is a secret; the images attached to GitHub releases are development ones, with temporary keys, and are meant only for trying things out.

What we learned

A few lessons that changed the plan:

• Our own TCP paid off after all. The prototype ran on smoltcp and served us well up to M13. But a server holding hundreds of thousands of connections does not need a client TCP; it needs SYN cookies, a timer wheel and 96 bytes of state. In M14 we replaced smoltcp with our own server-side-only stack, with simulated-loss tests and segment fuzzing.
• Measure memory from the inside. Per-connection estimates were useless until the kernel started counting the heap in allocator blocks rather than in requested bytes. Only then did we see where the 512 bytes really go, and we brought it down to 494.
• The test tooling is part of the product. The biggest limit was not the kernel but QEMU's user-mode networking and the number of ephemeral ports on the development machine. For a hundred thousand connections and more you need vmnet-shared on macOS, or Linux with KVM, a tap interface and multiple client addresses.
• SQLite without an operating system is possible, but only under strict conditions: a fixed heap, exclusive locking without shared memory, a VFS that knows nothing about files, and admission that does not let the database lag.
• Reconnect storms are the normal state, not the exception. When a mobile network drops and comes back, thousands of devices knock at the same time. That is why admission rejects a connection before any cryptography, and every rejected connection costs almost nothing.

What comes next

Work after M17 is planned in three directions, and the order among them is recorded in docs/PLAN.md. None of this has been built yet.

• One ISO for multiple platforms. The same file urmqtt-<env>-<version>.iso and one raw GPT image that boot on aarch64 UEFI, x86_64 UEFI and x86_64 BIOS, in virtual machines and on real hardware, starting with a single Intel PC. Still an appliance with everything inside, reproducible and signed.
• A raw disk, no filesystem. A byte-level format specification, a host tool urmqtt-disk that shares its parsers with the kernel, region alignment to 1 MiB, growing the disk with the data, discard for thin-provisioned disks, A/B database snapshots, bit-rot checking and, optionally, AES-256-XTS encryption with hardware AES.
• Provably secure. Runtime revocation and a device allowlist over a signed control channel, without a new image; differential and conformance testing of our own TLS and TCP; KASLR, stack protection, kCFI, MTE and a hardened allocator; SQLite isolated at EL0; Secure Boot; SBOM and cargo-vet; and finally a package for an external audit.

Platforms that offer only non-virtio devices, such as AWS ENA or Azure MANA, are not supported for now. Raspberry Pi 4 and 5 as a KVM host, Hetzner CAX and Oracle Ampere A1 are documented and supported in code, but have not yet been tried on the real service.

Try it

The repository is at github.com/zeljan-alduk/urMQTT. With Rust on the pinned nightly toolchain and QEMU, make certs, make run and a single mosquitto client are enough for a first PUBLISH over mTLS on your own machine. The warning from the beginning holds at the end too: this is pre-alpha, and everything that works is written in docs/PROGRESS.md. Bug reports, design questions and security reports following the instructions in SECURITY.md are welcome. We will report back here on each next milestone.

This blog is where we write down what we have learned along the way. Not marketing copy, but concrete notes: what went wrong, how we fixed it and what we would do differently today.

What we will write about

• Fare collection in public transport. Contactless cards, SAM modules, working offline and everything a terminal has to survive in a vehicle that shakes for twelve hours a day.
• Mobile apps in the field. Android and Flutter on devices that are not ordinary phones: with card readers, printers and batteries that have to last a whole shift.
• Backend and fleet monitoring. How to know a device in a vehicle has a problem before the driver reports it.
• Hardware. From a prototype on the bench to a board that goes into series production, and all the mistakes in between.
• Local AI models. Where models on your own hardware make sense, and where the cloud and an API are simply the better choice.

How we will write

Short and concrete, with code and numbers where we have them. We will publish when we have something to say, not on a schedule. Posts are written in Croatian and translated into every language of this site, and you can follow along by RSS.

If there is a topic you would like us to cover, or you are working on a similar problem, write to us at info@aldo.tech.