urMQTT v0.3.0: disk encryption, a boot that refuses a modified image, and what we broke ourselves

The third text, urMQTT after the VPS, ended with a "What comes next" list: a public demo broker (D1), a second round on the slow single flow, the plain path against Mosquitto on the VPS and, "after the release", bare metal, IOMMU, Secure Boot and disk encryption. This text says which of that was done, which was not, and what we found on the way. Disk encryption and part of Secure Boot did make it in before the release, because they were on the way to the audit. The second round on the single flow and the plain path against Mosquitto were not done.

The note up front stays the same: the project is pre-alpha, the release targets VPSs with KVM, and the numbers measure code, tests and specific experiments, not maturity in production.

How it looked over time

From the last text (1 November, 20:41) to today (3 November, 02:43) the main branch has 37 new commits, 215 in all. As before, the project is built with Claude Code (commits carry Co-Authored-By: Claude), and this time three branches ran in parallel (D1, H8, release preparation), so the order of commits is not strictly the order of work. The numbers are computed with the same method as in the last text (all .rs files outside vendor/ and fuzz/, #[test] functions, ## ADR- headings in docs/DECISIONS.md) and checked on the old commit, which gives the same numbers as then.

Moment	Commit	Time	Lines of Rust	Tests	ADRs	Commits
last text	d21e3ec	1 Nov, 20:41	112,408	926	126	178
today	f1a7fbc	3 Nov, 02:43	115,148	962	133	215

The growth in code is modest (+2,740 lines, +36 tests), because this time there was not much new code: most of the work is tests, scripts, documentation and checks. Scripts and JavaScript are not in the line count.

Encryption of the disk at rest (S7)

What we planned. Optional encryption of the data partition, off by default, with a measured cost. What we built (ADR-240):

• A layer under the disk. CryptDev wraps the block device below Disk, so the layout, the size table, SQLite, the telemetry and the backups work on plaintext and know nothing about the encryption. The algorithm is AES-256-XTS (IEEE 1619), one 512-byte sector per data unit, not a 4 KiB block as in the plan: storage writes in sectors, and a 4 KiB unit would turn every 512-byte write into a read-modify-write that a power cut can tear in half.
• The superblock stays in the clear. The two superblock slots carry a flag and a key check (HMAC-SHA-256 of a label and the disk's UUID), never the key. Everything else in the partition is encrypted. A sector that is all zero on the medium is read as zero and is not decrypted, so discard, space never written and growing the disk work as before.
• Configuration. storage.encryption = "aes-256-xts" and storage.disk_key (a file with 64 bytes of key). The blob format becomes 4 only when encryption is on, so every other image still boots on older kernels. An older reader refuses the unknown flag in the superblock.
• No hardware AES: refusal. The kernel refuses storage.encryption on a processor without AES instructions (or when [cpu] disable turns them off), instead of quietly running a slow software version.
• The conversion is offline. The kernel never encrypts or decrypts a disk at start-up. urmqtt-disk encrypt, decrypt and rekey do it with the VM switched off: a conversion record is written first and cleared last (while it exists the kernel refuses the disk), chunks of 1 MiB are first saved to a journal on the host, then the converted copy is written, and the superblock is switched last. Tested with a power cut at every single write for all three procedures, clean and half-written, with the cache dropped and kept: always a valid disk with all its data after --resume.

Measured. XTS throughput per core on Apple silicon (one MiB of sectors in a loop): 3,109 MB/s encrypting and 3,103 MB/s decrypting with hardware AES, against 252 and 258 MB/s in software. That is why we do not allow software. db-bench in a VM (aarch64, HVF, 5,000 inserts in 50 transactions, five runs of each variant, interleaved):

	ops/s, five runs	median
encryption off	49,239, 47,401, 50,802, 53,417, 46,531	49,239
encryption on	57,790, 46,173, 45,939, 52,787, 49,632	49,632

The difference of the medians is +0.8 %, and the spread inside one variant (46,000 to 58,000) is larger than that, so the cost is not measurable in this test. That is also what we expect: the whole db-bench run writes about 6 MB, which is about 2 ms of AES, and a commit waits for the flush, not for the cipher.

The Diagnostic entry now also shows AES-XTS next to AES-GCM (on HVF the kernel measures about 800 MiB/s for XTS against about 560 MiB/s for AES-GCM) and shows whether hardware AES is available.

What XTS does not give. There is no authentication: a modified encrypted sector decrypts to noise in blocks of 16 bytes (the CRCs and the scrubber find that), and someone who can write to the volume can swap or roll back whole sectors between two snapshots. It protects a stolen volume or snapshot, not a stolen ISO, because the key is in the image's configuration. What stays visible: the GPT, the superblock, the UUID and which sectors were ever written.

What we found in our own work on the way. Writing the cryptography chapter, we found two things. The step that multiplies the XTS tweak had a branch that depended on a bit derived from the key; now it is branch-free. And the claim "checked against IEEE 1619 vector 10" covered only the first encrypted block; now there is also a naive reference implementation (u128 arithmetic, block by block) that checks whole sectors in both directions on 200 random keys and sectors, and the documentation says what was really checked.

A boot that refuses a modified image (H8)

The requirement from the audit list was simple and until now unverified: no test showed that a modified kernel does not start. The hash pin in limine.conf existed, but without proof.

• A test that flips a byte. scripts/tamper.py flips one bit in every copy of the kernel (or of the configuration module) inside a built ISO or raw image, and SMOKE_ONLY=integrity shows the loader refusing: 24 checks passed, on aarch64 UEFI, x86_64 UEFI and x86_64 BIOS, with controls that do boot. The refusal message is read from the serial port through a switch for tests only. On BIOS the message reaches only the screen, so that test claims only that the machine does not boot.
• The loader's configuration enrolled (ADR-250). limine enroll-config works on BOOTAA64.EFI, BOOTX64.EFI and the BIOS stage, so a modified limine.conf (a menu entry or an argument added) is refused too. urmqtt-config pack --iso enrolls again.
• The Secure Boot path (ADR-251). scripts/sign-loaders.sh (test keys, signing, verifying, an OVMF variable store), mkiso.sh --sb-key and the secureboot group on x86_64: a signed ISO boots and builds reproducibly, while an unsigned one, one signed with the wrong key and one changed after signing get Access Denied (10 checks passed). Limine still refuses a modified kernel under Secure Boot.
• The rules of the installer entry (ADR-253). The plan spoke of one word on the command line, install. The production kernel in fact accepts three (install, live, diag), and only install writes. The test (9 checks) runs it on three machines with a blank disk whose hash must not change.
• Secrecy of the image (ADR-252). A loud warning when a production image is built outside CI, and a scan of the image, the build logs and the serial logs for key bytes in many encodings (11 checks).
• TPM, only ADR-254. What it would mean to seal the server key to a vTPM or TPM 2.0. Recommendation: do not build it for a VPS-first release. We did not check which providers offer a vTPM at all.

What this does NOT mean (written down as H8-W1 to H8-W6 in AUDIT.md): an installed disk does not boot under Secure Boot because its configuration module has no hash (ADR-217); a signed ISO cannot be configured afterwards (configure, then sign); the BIOS path has no signature; there is no protection against rolling back to an older version; on aarch64 the firmware here does not enforce Secure Boot, so only the loader's signature was verified; there is no project key, so release images are unsigned for Secure Boot; and no physical machine was tried.

The audit package (H10)

docs/AUDIT.md is now a package an external auditor can use without questions: the system on one page, the base we trust (89,612 lines of our own Rust in the image, 67 linked third-party packages), the inputs and their parsers, the matrix of protections with their tests (and the protections without tests), the weaknesses with their status, a table of every network-reachable loop and allocation with its bound and test, instructions for building and testing, a proposed scope for an external audit in ten areas, and a red-team list with 31 attacks and their results. SECURITY.md has a STRIDE table per component.

• A correction to the first text. The text of 29 October says that every unsafe block has a // SAFETY: comment checked by a lint. That was not true: the lint (deny(unsafe_code)) only limits where unsafe may be, and the comments have been tracked since H0 by a generated inventory, which even then found about 30 sites without a justification and, at the audit, 108 (because new kinds of sites had come in the meantime, for example the attributes #[unsafe(no_mangle)] and link_section). That is why the inventory, not the lint, is what now holds them at zero.
• 889 unsafe sites, each with a written justification. The 108 that had none (mostly #[unsafe(no_mangle)] and link_section, a few unsafe fn) got a comment (an AI agent wrote it, comments only, and I reviewed a sample); the generated inventory in CI records every new site without a justification. A justification is a claim to be reviewed, not a proof, and that is item 5 of the proposed scope.
• Five open findings stay open, written down: no fuzzing of the PCI and virtio parsers, no test that writes to the page tables after they are locked, the run time of P-256, X25519 and Ed25519 was not measured, no fuzzing of deliberately crafted SQLite files (contained by H6), and 53 of the 67 linked packages are cargo-vet exemptions, not audits.

Mistakes we found ourselves

• TLS traffic secrets were not wiped (H0-W2, open since the first day of the audit): the table of captured secrets was a Vec that remove and swap_remove moved around and left copies. Now the table is slot-based: a slot is wiped in place when taken, dropped or evicted, and nothing is moved.
• A branch dependent on the key in XTS (above).
• log-slow-flood on x86_64 was failing ("log ring full"), also on the v0.2.1 source. The cause was not a defect but the sizing of the test: the scenario drove every core at 25 ms, while ADR-090 had measured it at 50 ms. On x86_64 COM1 is trapped port I/O per byte, one core that writes to the sinks needs about 9 ms per line, and three cores at 25 ms ask for more than a second of work per second, so the rings fill (the intended overflow). The test now runs at 50 ms on x86_64 (it passed, 30 of 30 in the run that includes the baseline scenarios); on aarch64 it stays at 25 ms.
• Merging the branches broke the old templates. The re-enrollment in H8 required an end marker in limine.conf, which templates from before H8 do not have, so urmqtt-config pack --iso and the builder page refused every release before H8, including the v0.2.x that the page offers. The builder tests in the browser caught it (4 of 100 failed), not H8's own test. Fixed (a template without the marker is configured as before) and a test added that fails with the original error when the fix is taken out.
• Our documentation wrongly claimed that x86_64 was verified only in an emulator. X11 ran the production image on a real Contabo KVM VPS (4 vCPU AMD EPYC) with mutual TLS under load. Corrected everywhere: x86_64 ran on a real KVM VPS at the H7 commit and in an emulator, but the code added after that (encryption, boot integrity) has not run on KVM, and it has not been tried on physical hardware.

D1 and the offer "your service as a unikernel"

The public demo broker is built, but not published: the configuration (live mode, ACL demo/{id}/#, plain MQTT only for the bridge), a script that makes a published test CA and demo certificates, a WebSocket-to-TCP bridge (only the Python standard library, limits on connections, rate and size, an origin check), a "Try urMQTT live" page with an MQTT client in plain JavaScript, a runbook and a threat model. Tested on loopback and in QEMU (18 scenarios, 26 bridge tests). Not tested: a real VPS, DNS, Caddy, the container image. It needs a separate VPS and explicit approval for public exposure.

In addition, the site now has an offer, Unikernel (15 languages, no prices): adapting your service so that it runs as a unikernel (a feasibility assessment, adaptation, a pilot, support), with urMQTT as the reference. The text states the limits of today's foundation explicitly: IPv4, TCP only as a server, KVM, Rust the natural choice, C only as an isolated worker.

The v0.3.0 release

The tag v0.3.0 was set on 3 November on commit f1a7fbc, and the release workflow succeeded. The release is marked as a pre-release and contains the development aarch64 kernel, ISO and image, the template ISO (kernels for aarch64 and x86_64 and an empty configuration, which urmqtt-config pack --iso and the builder page turn into your image), a CycloneDX SBOM, SHA256SUMS and its minisign signature, and SLSA provenance attestations. Verified after publishing, in an empty directory: the signature of SHA256SUMS is valid, all five files match the checksums, both attestations (the kernel, and the ISO with the CycloneDX predicate) pass, the SBOM has 221 components, and the template configures (pack --iso re-enrolls six copies of the loader). The signing key is still a test key.

The order mattered and is worth writing down: the builder page was updated before the tag (it knows format 4 and all the older templates), and only then did the mirror on aldo.tech pick up the release. The mirror listed it signed and verified (ok, configuration format 4, the template's sha256 equal to the one in SHA256SUMS), and the live page offers it as the default release and builds an ISO in the browser in about 20 seconds. The release notes also say what is rarely written: CI on GitHub is not green (see below), support is only for the latest version (v0.2.1 and older stop), and the project did the audit itself.

On the site we almost did ourselves harm on the way: the site's publishing script (publish.sh) runs rsync --delete, and the release mirror belongs to another user, so every publish ended with error 23 (and would have broken the "Publish" button in the blog admin). Fixed with an exclusion for /urmqtt/releases/. A similar finding: deploy.sh with --delete removes the builder page from the generator, so it has to be put back before publishing.

What is still not proven

• CI on GitHub is red. Not only today: the last run of the full CI before the tag had seven failed jobs, and the runs going back to 31 October are mostly red. The ones that failed: lint (shellcheck warnings in scripts), TLS interop (all wolfSSL cases), the supply chain (the exemption list vet-exemptions.md is not up to date on Linux), fuzz smoke (the development certificates are missing in that job), the sanitizers (LeakSanitizer reports a leak in tests), tests on Linux aarch64 (one test of the tool fails with "Broken pipe") and one Kani proof for the timer wheel does not pass. We have not yet sorted out which of these are environment and which are real defects; until we do, we do not claim that the release passed CI.
• The signing key is a test key, and the release images are development builds with test certificates (except the template).
• The translations of the pages and of this text into the other 13 languages were written by AI and not checked by native speakers.
• Disk encryption has not been measured on x86_64 (neither on KVM nor on physical hardware), and neither the telemetry throughput nor the start-up time with encryption has been measured. It is not on the builder page (it needs a key dialog).
• There is no independent security audit. The scope is written; we have no auditor.
• Secure Boot is partial (above), without a physical machine.
• Bare metal, IOMMU, the public demo broker: not done or not published.

What comes next

1. Sort out and fix CI (environment first, then the Kani proof and the test that fails on aarch64 Linux), because we do not regard a release without a green CI as finished.
2. A decision on D1 (a separate VPS, DNS, approval for public exposure).
3. Fuzzing of the PCI and virtio parsers and a test for the locked page tables (H0-W9, H0-W10).
4. A key dialog on the builder page, so that encryption becomes available without the command line.
5. After the release: bare metal (X12 to X16) and IOMMU, in an order that depends on the hardware inventory of the test PC.

The code is at github.com/zeljan-alduk/urMQTT; docs/PROGRESS.md remains the only source of truth about what really works, docs/PERFORMANCE.md about what was measured, and docs/AUDIT.md about what is still not proven. Pre-alpha, as before; bug reports and security reports following the instructions in SECURITY.md are welcome.