urMQTT on a real VPS: the ISO, a 0.24 ms handshake, our own disk and a hardened kernel

In the first post about urMQTT we described a unikernel that is an MQTT broker, up to milestone M17, and we ended with three promises under the heading "What comes next": a single ISO that boots on multiple platforms, a raw disk without a filesystem, and security that can be proven rather than merely claimed. None of that had been built at the time. This post follows what happened to each promise, in the same order as before: what we planned, what we built, what surprised us and what is still open. Along the way we will also show how long it took, because that is part of the story.

One note up front: the project is still pre-alpha. The numbers in this post measure code and tests, not maturity in production, and the first release is deliberately limited to VPSs with KVM. Bare metal is left for later.

What it looked like over time

The first commit was made on 28 October 2026 at 23:48 (M0). By 31 October at 23:00 the repository has 119 commits, distributed as 3, 39, 38 and 39 per calendar day. Let us say right away how that is possible: the project is built with Claude Code, and the commits carry Co-Authored-By: Claude trailers. Four calendar days of history do not mean that one person wrote 90,000 lines in four days; they mean that planning, implementation, tests and ADRs are written in short, closed loops, and every step ends with the same verification ritual as before. The numbers below describe the amount of code and tests, not how much of it has been exercised in real-world work.

The graph shows three curves that grow almost together, which is the intention: tests and recorded decisions keep pace with the code instead of lagging behind it. The slope is steepest on 29 October, when the whole path from the kernel to the broker comes into being. After that, lines are added more slowly, but tests grow faster than code (27 tests on 1.6 thousand lines at the start, 739 on 89.8 thousand today).

Milestone	Commit	Time	Rust lines	Tests	ADRs
M0	9412a9c	28 Oct 23:48	1,592	27	13
M5	a61083b	29 Oct 01:32	9,314	88	24
M9	4be8cc9	29 Oct 02:21	15,729	149	28
M11	4800904	29 Oct 03:43	22,737	180	30
M13	a558d5e	29 Oct 05:22	26,556	193	32
M17	7df4f98	29 Oct 08:48	33,504	232	36
S1	ecf72ec	30 Oct 00:46	41,910	299	45
X8	fb1504a	30 Oct 19:08	56,341	456	59
PERF	76f7c36	31 Oct 14:19	72,511	618	76
today	HEAD	31 Oct 23:00	89,790	739	104

Lines are all .rs files outside vendor/ and fuzz/, and tests are #[test] functions. What each part added, by date from the git history:

Date	Milestones	What was added
29 Oct	M0 to M17	kernel, networking, TLS, broker, SQLite, multicore, our own TCP, hardening, reproducible release
29 Oct evening and 30 Oct	X0 to X10, S1, H2, H3 (X0 starts on 29 Oct at 20:54)	x86_64 port, universal ISO, a single disk and a self-installing ISO, TLS checks, attacker clients
31 Oct	O1 to O4, T1, PERF, H4, H5, H9, S2 to S6	insight and a console over MQTT, live mode and diagnostics, SNTP, speedups, hardening, verification, supply chain, storage

Promise 1: one ISO for multiple platforms

What we planned. A single file that boots on aarch64 UEFI, x86_64 UEFI and x86_64 BIOS. What we built. The x86_64 port (X0 to X10) and a universal ISO with the Limine bootloader: a single hybrid ISO that boots in all of those combinations; the smoke tests boot it on four virtual machines. The menu has three entries: urMQTT (live mode, runs from RAM, writes nothing to disk), install to the disk of the machine it was started on (the installer writes a GPT and the data, then powers the machine off) and Diagnostics (hardware detection and measurements).

The first real target was a VPS rather than a physical computer: the decision is that the first release supports only VPSs (KVM), with bare metal coming after it. We tested on a Contabo VPS with an AMD EPYC processor (family 23, Zen 1), 4 vCPUs and 8 GiB of RAM, with a virtio-net network and a virtio-scsi disk. The first attempt surprised us: the ISO attached as a virtual CD booted the default entry, the production image refused to serve from read-only media and reset, and we got an endless loop of menu, kernel, reset with nothing legible. Today a fatal error in the production image leaves a message on the console for 60 seconds before the reset, and from optical media it prints that it was booted from read-only media and stops. Live mode in RAM (O3) later worked around that problem for the usual case.

The Diagnostics entry grew out of that experience: a machine we cannot see must be something we can ask what it has. It prints the CPU, RAM, network, disks and PCI devices, measures memory throughput, disk and cryptography, and prints the same measurements as JSON on the serial port. Writing to disk is allowed only in unallocated space and only after you type WIPE; in the screenshot below that was skipped.

What is still open. Real hardware. After the first release comes bare metal (USB boot, NVMe and AHCI, the network card of a real computer), together with an IOMMU and Secure Boot, because without an IOMMU a malicious or faulty device can write over all of memory. In this version we claim nothing about that.

The handshake: from 1.2 ms to 0.24 ms

The first diagnostics measurement on the VPS gave an awkward number: the mTLS handshake, server-side cryptography only, took 1.2 ms. On aarch64 we were already using hardware AES and SHA instructions, but x86_64 had nothing like that for public-key operations: RustCrypto is portable and constant-time, and therefore slow as well. Three changes brought it down.

• For P-256 we took over the nistz256 assembly from OpenSSL and ring (ADR-130), with fixed-base tables for signing and for verifying the CA signature (ADR-110, ADR-111).
• X25519 was rewritten to four 64-bit limbs with MULX/ADX instructions (ADR-131).
• After that we profiled the rest (ADR-132, ADR-200).

Measurement on the VPS (server, microseconds)	Before	After
mTLS handshake, all cryptography	1200	about 240 (237 in the graph measurement, 244 in the screenshot)
X25519	146	48
ECDSA signing	673	34
ECDSA verification	309	90 (107 in the screenshot)
handshake, old memcpy, same boot	278.8	229.4 (-17.7%)

The numbers vary by about 10% between runs, so we recorded a range rather than a single value. Overall that is roughly five times less CPU time per handshake.

What we learned about memcpy. Once the cryptography dropped, the profile showed something we did not expect: the mem* functions (memcpy, memmove, memset) took 7% of the handshake, 13 to 39% of the TLS record path (39% for 64 B, 13% for 4 KiB) and 17 to 34% of PUBLISH fan-out. We replaced them with SSE2 assembly for x86_64 (ADR-200), with differential tests against the reference implementation for all sizes, alignments and overlaps. Per call, that is 55 to 75 cycles versus 7 to 12 for 1 to 64 bytes.

Path	mem* share before	Faster after
handshake	7%	17.7%
TLS record 64 B	39%	26.4%
TLS record 256 B	37%	19.6%
TLS record 1 KiB	24%	16.1%
TLS record 4 KiB	13%	6.6%
message fan-out	17 to 34%	7 to 34%

Reading the graph: the smaller the record, the larger the share of time that goes to copying, and so the larger the gain. For small messages, which make up most MQTT traffic, that is what counts.

What this does not mean. On native x86_64 we measured OpenSSL 3.5.5 only as a reference for the assembly (ADR-130), not as a product comparison. The comparison of Linux plus Mosquitto on the same VPS (task X11) has not been done yet. So we do not claim that urMQTT is faster than Linux: it has not been measured. The numbers above compare urMQTT with itself.

Overload was the second result. On the VPS, from a single Mac (RTT 35 ms), 128 clients produced 618 handshakes per second, 256 clients 393, and 512 clients 67, with thousands of rejected connections. That was not a plateau but a collapse. We found the causes in the code (two were confirmed in QEMU/HVF and could not be reproduced on the VPS), several at once: a rejected connection sent the client over the next segment carrying a valid SYN cookie, the rejection told the client by RST to retry immediately, and receive was O(queue). The fix (ADR-080 to ADR-082) drops a SYN that would not be accepted, remembers rejected flows, keeps receive at O(1) and counts handshakes per core. It is possible (a hypothesis, not verified) that part of the cause was on the client as well: the load tool closed first, so a single Mac with 16,384 ephemeral ports and a TIME_WAIT of 30 seconds is limited to about 550 connections per second. That is why we have not yet measured the server's true ceiling.

Promise 2: a raw disk, but a careful one

What we planned. To keep our own layout without a filesystem and add everything a filesystem normally provides: alignment, growth, discard, backups and checks. Why not a filesystem. We debated that and the decision was confirmed: urMQTT writes one database, one telemetry log and its own metadata, and needs no directories, permissions or file names, and every additional layer would be more code in the trusted part and one more way for a write to silently disappear. The disk is a GPT with an ESP partition and one urMQTT data partition, on which sits our own layout.

What we built (S2 to S6).

• Layout v2 (S2). Disk topology, alignment to 1 MiB or the optimal I/O size, whole physical blocks, an A/B pair of superblocks with a generation (ADR-140). The kernel rejects layout v1 with a clear message, and the offline tool urmqtt-disk migrate migrates it (ADR-141): a dry run is the default, it copies and only then switches over, with rollback and fault injection on every write.
• Growth (S3). At boot the disk can grow (it never shrinks), and on virtio-blk also at runtime, through the configuration-change interrupt (ADR-142, ADR-143).
• Discard and write-zeroes (S4). Thin-provisioned disks give space back to the host. In the test, a qcow2 file on the host dropped from 62 to 16 MiB after the telemetry segments expired, and recovery never trusts the contents of a discarded range.
• On-disk database backup (S5). An A/B copy using the SQLite online backup API in steps of 16 pages: the longest step is 4.6 ms versus 445 ms when copying in a single call. Commits during the copy had a maximum of 2.4 ms versus 2.2 ms without it. If quick_check fails at boot, the database is restored from the copy.
• Scrubbing and bounded boot (S6). The scrubber reads the disk within a bounded I/O budget and reports corrupted records, and boot reads only the newest segment of each partition, so boot time does not grow with the size of the telemetry region. That closed a problem we had calculated in advance: on a large disk, full recovery would take anywhere from minutes to tens of minutes.

The whole group of storage smoke tests passes on both architectures: 79 passed, 0 failed, on aarch64 under HVF and on x86_64 under TCG. We also measured that S4 did not slow the database down: 53.3 thousand inserts per second in batches before and after, with no measurable difference.

What surprised us. How much of a "simple" layout comes down to edge cases: virtio-scsi and virtio-blk report topology in different ways, and a partial write in the middle of a migration must always leave at least one valid version. That is why every write on the migration and growth path went through fault injection.

What is still open. SQLite pages have no checksum, so a flipped bit inside a value in a row will not be detected by the database; telemetry records and the contents of the copies have a CRC and are detected. Disk encryption (S7) is not done.

Promise 3: provably secure, as far as we can manage

Hardening (H4). The allocator wipes freed memory and checks it on reuse, detects double frees, links free-list pointers with XOR and zeroes keys when they are freed (ADR-170). Stack protection (-Zstack-protector=strong, 1315 functions, and SQLite with -fstack-protector-strong, ADR-171). Error reports in production contain no registers or addresses (ADR-172). KASLR on both architectures: the kernel is a PIE and Limine shifts it by about 19 bits, and the heap, stacks and MMIO window are placed randomly (ADR-173). On aarch64, MTE tags the heap (ADR-174), but only under TCG: HVF and x86 do not have it. We evaluated kCFI and did not adopt it because the compiler on the pinned nightly crashes with an internal error; the IOMMU is deferred along with bare metal (ADR-175, ADR-176).

The cost is measured, not assumed. On the handshake path (dp-bench, aarch64 HVF, full TLS 1.3 + mTLS + CONNECT handshakes):

Step	1 core	4 cores	Change
no hardening	1941	7620
+ allocator	1898	7429	-2.2% to -2.5%
+ stack protection	1845	7243	together -4.9% to -5%
+ PIE and tags	about 1800	about 6750	on a loaded machine, noisy

Overall, then, the cost on the handshake path is about 5% for the parts measured cleanly, and with PIE and tags it comes to about 7% on one core and about 11% on four, though that measurement is noisy. We consider it acceptable, but we have not repeated it on an x86_64 VPS.

It is worth comparing with the figure from the first post (862 handshakes per second on one core). The same tool today, with hardware cryptography enabled on aarch64 (X0b) and a series of speedups that have arrived since, gives 1941 before hardening. We have not isolated the contribution of each change, so we claim nothing about the cause.

How it was verified, and what was not. H5 added Miri on a subset of host crates, Kani proofs, sanitizers, a model fuzzer for the ACL and coverage reports. Kani proved record_len in TLS, parse in TCP and two timer-wheel arithmetics. Not proven (timed out): both MQTT decoders, both configuration parsers and the behavior of the timer wheel; the CI job for Kani therefore does not block yet. Miri cannot execute assembly. Supply chain (H9): a CycloneDX SBOM with 215 components, cargo-vet passes with 39 fully audited and 140 exemptions, and cargo-deny is configured to treat vulnerability advisories as a CI failure (those CI jobs have so far been verified only locally, not on GitHub, and the negative test has not been run), together with VULNERABILITY-HANDLING.md. The honest state: of 67 kernel-linked third-party crate versions, 53 are exemptions rather than audits. Vet proves that the set of dependencies cannot change without a visible difference, not that the TCB has been reviewed. We have not written our own audit of any crate.

SNTP (T1) is a small security story in that sense: it is optional, only toward private IPv4 servers, with a maximum step and the condition that the sources agree (median, majority within one second). A clock that anyone on the path can move is not a clock we may trust for certificate validity.

On the Mac: aarch64 natively and x86_64 by emulation

We develop on a Mac, which has two ways of running the same code, and they are very different.

	aarch64 under HVF	x86_64 under TCG
What it is	native execution, speed close to a real processor	QEMU emulates the instructions
Handshake in the guest	about 0.58 ms (first measured around M13); dp-bench today 1941/s per core	about 3.3 ms
TLS smoke scenarios	fast	about 15 minutes per scenario
MTE	no (not available on HVF)	x86 does not have it; heap tagging works only for an aarch64 guest under TCG
What it is for	real measurements	correctness: the x86 path is verified, the numbers are not performance

The rule is simple: numbers under TCG are emulation and we do not cite them as performance. We measure x86 performance only on a real VPS. That is why the cryptography numbers above come from the Contabo VPS and not from the Mac.

Lessons

• Measure on a real machine before optimizing. The first approach was to optimize what we thought was slow; only diagnostics on the VPS showed 1.2 ms and exactly where the time goes.
• Overload is a separate feature. A system that is fast under load can collapse above it; rejecting must be cheaper than accepting.
• The profile surprises. The biggest single gain for small messages was not in the cryptography but in memory copying.
• A disk without a filesystem takes more work, not less. What a filesystem normally provides (alignment, growth, discard, copies, checks) we have to write and verify ourselves, every step with fault injection.
• Honesty in numbers. Memory measurements on a shared VPS vary up to 10 times between runs and say nothing about the code, so we do not cite them. Crypto numbers are stable to about 10%.
• Development speed is not maturity. Tests and ADRs grow alongside the code, but four days is not real production.

What comes next

• H6: SQLite isolated at EL0, so that a bug in the database does not open up the kernel.
• H7: the rest of time security (NTS and step handling); SNTP is only the first step.
• X11: a comparison with Linux and Mosquitto on the same VPS; only then does it make sense to talk about relative speed.
• Release: the first public release for VPSs (KVM), with signed artifacts and instructions.
• After the release: bare metal (X12 to X16), IOMMU and Secure Boot, and disk encryption.

The code is at github.com/zeljan-alduk/urMQTT, and docs/PROGRESS.md is still the single source of truth about what actually works. Pre-alpha, as before; bug reports and security reports following the instructions in SECURITY.md are welcome.