urMQTT after the VPS: SQLite in user mode, a comparison with Mosquitto, configuration as data and a builder in the browser

The second post, urMQTT on a real VPS, ended with a "What comes next" list: H6 (SQLite outside the kernel), H7 (time security), X11 (a comparison with Mosquitto on the same VPS) and the first release. This post follows that list in the same order and then adds two things that were not on it, because they appeared along the way: configuration as data, with a web ISO builder, and optional plain MQTT without TLS. As before: what we planned, what we built, what surprised us and what is open. This post also contains several results that do not favour urMQTT; we kept them because they are part of the truth.

The note up front stays the same: the project is pre-alpha, the first release is only for VPSs with KVM, and the numbers measure code, tests and specific experiments, not maturity in production.

What it looked like over time

From the previous post (31.10. at 23:00) to today (1.11. at 20:41) the repository has 59 new commits, 178 in total. All of them were made within a single calendar day, so the same warning applies as before: the project is built with Claude Code (commits carry Co-Authored-By: Claude), the plan, code, tests and ADRs are produced in short closed loops, and the numbers below say something about the amount of code and checks, not about how much human work is in them.

The numbers were computed with the same method as in the previous post: all .rs files outside vendor/ and fuzz/, #[test] functions, ## ADR- headings in docs/DECISIONS.md.

Moment	Commit	Time	Rust lines	Tests	ADRs
previous post	794f607	31.10. 23:00	89,790	739	104
H6 done	a0423cc	1.11. 03:59	95,078	754	107
X11	a29f49b	1.11. 07:56	95,579	754	108
web builder and release mirror	582783f	1.11. 16:49	105,645	848	122
plain MQTT, documentation	8f959f1	1.11. 19:05	112,408	926	125
today	d21e3ec	1.11. 20:41	112,408	926	126

Work went on in parallel on several branches, so the order of commits is not strictly the order of work. ADRs are numbered up to 232, but the numbering has gaps (some were renamed when branches were merged); we count headings, not the highest number. Releases: v0.2.0 (1.11., 18:45 UTC) and v0.2.1 (1.11., 20:22 UTC), both marked as pre-release.

H6: SQLite in user mode

What we planned. SQLite is 254,268 lines of C that parse the database, the WAL and the journal from disk, and the disk may be crafted or modified. Until then it ran in the kernel, with kernel privileges. What we built (ADR-210 to ADR-212): SQLite, its libc parts and its heap were moved into a separate program, dbworker, which runs at EL0 (aarch64) or ring 3 (x86_64), in its own address space. The kernel no longer links SQLite. The worker talks to the kernel through a shared memory window and twelve system calls, and every length and offset that comes back is checked before use. A fault in the worker (illegal access, undefined instruction, an exceeded time budget of 30 s) ends the call, the worker is restarted and the database is opened from the WAL as at boot; after 5 restarts in a minute the storage is declared faulty rather than restarted endlessly.

The cost, measured on aarch64 under HVF, the same VM, a kernel with built-in SQLite versus a kernel with the worker, six alternating runs of each:

Measurement	Before (in kernel)	After (worker)	Difference
entering and leaving user mode	-	1.1 us per round trip
db-bench, 5,000 inserts in 50 transactions, best / median	61.9k / 58.4k op/s	54.7k / 53.3k op/s	-11.6 % / -8.7 %
one insert per transaction, best / median	2,319 / 2,148 op/s	2,307 / 2,126 op/s	-0.5 % / -1 %
whole backup in one call (23 MiB)	514 ms	635 ms	+23 %

Disk writes (1,572) and flushes (319) are identical, as they must be. What is not isolated: the kernel still trusts what the worker returns, after checking its shape. A compromised worker can lie (for example, return an empty list of revoked certificates and thereby "un-revoke" them), but it cannot read kernel memory, TLS keys or ticket keys. There is no KPTI, so Spectre/Meltdown-type leaks are not prevented and we claim nothing about them; the PAC key is shared by EL0 and EL1. On x86_64 we did not measure the switching cost on KVM (under TCG the number would be meaningless), and fuzzing of the worker and the protocol itself is not finished. The storage test suite (79 scenarios, each of which runs SQLite in the worker) passes on aarch64 HVF and x86_64 TCG, and 76 on aarch64 TCG (one is skipped because TCG has no MSI-X).

H7 is closed, but not the way we announced. The lower bound on time (ADR-050) and SNTP with consensus (T1) cover the scope of H7; NTS (RFC 8915) remains out of scope and is recorded as a residual risk, not as something that works. We do not consider time security "solved", only its provable part done.

A new network option. Contabo does not give DHCP to a custom ISO, so until now the VPS needed a different image from the lab VM. Now network.mode = "auto" (ADR-214) asks DHCP first, and if there is no lease within the set time (dhcp_timeout_s, default 15 s) or it loses the lease, it applies the configured static address. Tried manually on QEMU (with DHCP it uses the lease, without a network it falls back to the static address after 5 s); there is no smoke scenario for it yet.

X11: a comparison with Mosquitto on the same VPS

The same kind of machine as in the previous post (Contabo, 4 vCPU AMD EPYC, 8 GiB), load from a Linux host 15 ms away, mutual TLS 1.3 with the same certificates on both sides, Mosquitto 2.0.21 on Debian 13 without persistence. The method and decisions are in ADR-213, the raw data in docs/perf-results/. Median of three runs.

	urMQTT	Mosquitto 2.0.21
handshakes/s, 32 / 64 / 128 clients (bench profile)	169 / 278 / 413	82 / 140 / 184
same, with the limits as shipped	185 / 262 / 333 (at 128: 202 to 286 errors, handshake resets)	-
memory per idle connection (20,000 connections)	0.50 KB of kernel heap (0.66 KB at 5,000)	26.8 KB of process growth
latency p50, QoS 0 / QoS 1, 800 messages/s, 64 B	14.0 / 14.8 ms	21.4 / 39.5 ms
QoS 1, 4,800 messages/s, delivered/s (p50)	4,209 (77 ms)	4,235 (197 ms)
100 subscribers, 50 messages/s: p50	16.6 ms	99 ms
single flow, QoS 0, 64 B, published/s	2,962 (0 to 7,733)	22,749

What this means and what it does not. The handshakes are lower bounds for both servers: the load generator is a shared VM with 60 to 70 % stolen CPU time and at 128 threads it was close to its own limit, while urMQTT's worker cores were busy only 12 to 13 %. The ratio at 128 clients (about 2.2 times) is the most meaningful; at 32 and 64 it mostly measures clients waiting. The memory numbers do not count the same thing (kernel heap versus process RSS), so "about 50 times less" is an order of magnitude, not a precise ratio. A latency of 14 ms is one network round trip (about 15 ms), so "no measurable broker time"; the tails (p99 around 100 to 200 ms) are dominated by the shared hosts for both brokers. We do not claim differences below about 20 %.

What did not go in our favour and stayed in the tables: a single TCP flow through urMQTT carried 2 to 8 times less than through Mosquitto, with swings from zero to 7.7 thousand messages per second between runs, and the 20 ms handler watchdog cut live connections when the hypervisor stopped a vCPU (in the production setup action = "drop", 6 to 8 connections per run under QoS 0 flooding). Both are described in the next section because they had causes in our code.

Why the single flow was slow: causes, not excuses

The easiest thing would have been to say "shared VPS". We did not; with the same VPS and the same method we looked for causes. Five were found, four in the data path and one in the watchdog (ADR-221 to ADR-224):

Symptom in X11	Cause in urMQTT	Change
single flow 1.9k messages/s, often 0 delivered	Lost wakeup: the output loop returned the list of ready connections over the one just written; a subscriber whose output buffer had filled once was never served again until a PINGRESP woke it	fixed (ADR-223)
a few percent of loss turns 48k into 14k messages/s	every segment after a lost one was dropped (in-order receive only), so recovery cost one network round trip per segment	out-of-order segment queue, SACK (ADR-222)
ceiling of 4.4 MB/s at 15 ms	16-bit window, a 64 KiB send queue and slow start that counted ACKs, not bytes	window scaling, timestamps, byte counting (ADR-222)
every round of a flow that fits waits 0 to 10 ms	the dispatcher slept until the next 10 ms tick (virtio-net has no receive interrupt)	polling the NIC every 200 us while connections exist (ADR-224)
1 to 8 dropped connections per run	the watchdog charged the handler for elapsed time, and the VPS stops vCPUs often and for long	the budget is the time the core actually ran (ADR-221)

The stops are real: on this VPS cores 0 to 2 had a median of 36 to 80 late ticks per 5 seconds, and the worst stop in the window had a median of about 70 ms and went up to 0.9 s (core 3, database and journal: up to 1.55 s), while the host reports no stolen time (host steal 0 us). That is why the watchdog needed a second clock, not a larger budget.

	X11 code (7f9517e)	now (86a71ff)	Mosquitto
single flow, QoS 0, 64 B, published / delivered per s	1.9k / 0	23.7k / 12.6k	29.3k / 29.2k
single flow, QoS 0, 1 KiB	0 / 0	3.9k / 2.3k	5.4k / 5.4k
16 pairs, flooding, QoS 0, 64 B	24.7k / 20.0k	43.2k / 37.4k	31.4k / 30.7k
16 pairs, flooding, QoS 0, 1 KiB	3.7k / 2.9k	5.0k / 4.5k	5.9k / 5.7k
lost connections per run (16 pairs, 64 B)	13 (7 to 16)	0 (0 to 0)	0

Watchdog: 148 dropped connections in 30 loaded runs before, 0 in 24 runs after; on a local rig that delays and drops frames, at 0.1 % loss a single 64 B flow of 14.3k messages/s (the X11 kernel) now reaches 41.0k, and at 0.5 % from 5.0k to 17.7k.

What this does not say. urMQTT is still not faster than Mosquitto on a single flow: at 64 B it publishes about 80 % of its speed and delivers about 40 %, because it drops QoS 0 when the subscriber's flow is slower than the publisher's (Mosquitto queues without a limit). At 1 KiB in the 16-pair flooding Mosquitto is ahead by about 15 %, which is within noise. The generator was again a shared VM with 58 to 77 % stolen time. The second round (heap locking moved out of the data path, send-side SACK and more) is on the unintegrated branch perf-single-flow-wip; its numbers are not claims yet and we do not give them here.

A promise we did not make, but kept: configuration is data

Until now every configuration was a new build: keys and limits were built into the image. C1 turns that around (ADR-215 to ADR-219): the kernel is built once, and the configuration is data.

• The urmqtt-cfg blob (C1a): a versioned, canonical record (fields in ascending tag order, a SHA-256 fingerprint, an optional detached Ed25519 signature), 70 fields with defaults and bounds, and a decoder that rejects anything malformed.
• The kernel reads it as a Limine module (C1b), before anything else, on aarch64 and x86_64. There is one released kernel for every ENV except dev (the aarch64 ELFs for release, prod and bench have the same SHA-256; verified only on aarch64). The released kernel rejects a development configuration, and without a configuration it prints "refusing to serve" and serves nothing.
• The installed disk carries the configuration and an update keeps it (C1c).
• urmqtt-config and the ISO template (C1d): pack, show, verify, and pack --iso replaces both copies of the configuration slot and the 24 module digests in a finished ISO. The result is byte for byte the same as the ISO that mkiso.sh --config builds. The v0.2.0 release template is 63,111,168 bytes.
• The trust statement (ADR-219): a configured image prints the configuration fingerprint at boot (config: fingerprint ...), and the signature is optional and off by default. A modified ISO has no official digest; the fingerprint is the way to see what is in it.

The builder on aldo.tech

What urmqtt-config pack --iso does, the web page does too, with the same code: the tool library compiled to WebAssembly (1.06 MB, no imports and no unsafe block). The page is at aldo.tech/en/products/urmqtt-builder/ and exists in all 15 languages of the site (for example hr).

• Nothing leaves the browser. Certificates, private keys and the finished ISO stay on your computer; the page downloads only the release files from this site, with no tracking and no third-party scripts. In the live test no request carried key material.
• The release mirror (ADR-227): hour by hour (a systemd timer, low priority) it copies releases from GitHub, checks each one before publishing, and the page in the browser verifies the SHA256SUMS signature with a key embedded in the page, and only then writes the configuration into the template. A failed check blocks the download.
• Simple by default, advanced when needed (ADR-232): the first screen is three steps (where it runs, your certificates, the image name). A single drop field sorts the certificate, key and CA by content. The advanced view opens each of the 70 fields, search, and TOML import and export; both views edit the same state. The template (63 MB) is downloaded while the user fills in the form. In a live trial, with a test mirror and a test key, building the ISO in the browser took 5 to 18 s.

The emulator, optional. Next to "Download ISO" there is "Try in the emulator": QEMU compiled to WebAssembly runs the image you just built in a browser tab. Nothing of it is downloaded before the click (about 3 MB the first time). The emulator is BIOS-only, has no network (so that the user's traffic does not pass through some intermediary and break "nothing leaves the browser"), needs about 1.5 GB of memory and a desktop browser (Chrome tested; Firefox and Safari not). In it urMQTT reaches URMQTT READY in about 8 s (with the countdown skipped) and prints the same configuration fingerprint as the page, which is proof that exactly that configuration was booted. Boot is about 2 times slower than native QEMU on the same Mac (0.82 s against 1.9 s), and cryptography about 58 times slower because the hardware instructions are emulated. That is why we draw no claims about speed from the emulator, and the page says so before it starts. During the build four bytes in the .wasm had to be patched because of a bug in the translation of the POPCNT instruction in the upstream emulator (we have not reported it yet); the emulator itself is under the GPL, with an offer of the source code.

Plain MQTT, only for networks you trust

Plain MQTT (TCP 1883, no TLS) is very common in IoT: it is the default in Mosquitto, in most Arduino and ESP firmware, in Tasmota, Zigbee2MQTT and Home Assistant, and it is used inside segmented industrial networks, behind VPNs and on private LANs, including on devices without a clock (certificates need a date), without enough flash or CPU for TLS. Over an untrusted network it is not acceptable: whoever is on the path reads and modifies the traffic. That is why P1 (ADR-229 to ADR-231) is an explicit, limited choice, off by default, and the TLS port stays; TLS is not turned off by this, the plain port comes alongside it.

• An allowlist of networks is mandatory (allow_from, CIDRs) and is checked at the SYN: outside the list there is no SYN-ACK and no state. The whole internet (0.0.0.0/0) requires a separate explicit option.
• Three login modes: password (the default when enabled; PBKDF2-HMAC-SHA-256, salt, 10,000 iterations, constant-time comparison, lockout per address), anonymous (one configured identity with its own ACL) and by address (ip). The builder hashes the password in the browser and keeps only the digest in memory.
• Fewer privileges than mTLS: a plain client never has service, operator or management privileges, has no $SYS and no console, has its own sessions ($plain/<identity>/<client id>, clean only), and its own limits and the registry of the allowed still bind it.
• What is lost is written in SECURITY.md: confidentiality, integrity, broker authenticity and replay of a recorded login. This cannot be fixed by configuration; that is how it is without TLS.

Measured on a laptop (Apple silicon, HVF, QEMU user-mode network), 16 pairs, 64 B, three runs of 10 s: at 8,000 messages per second the TLS and plain paths deliver all messages with the same latency (p50 0.17 ms against 0.13 to 0.18 ms), new connections per second: TLS about 2,600 to 2,750, anonymous plain about 4,900 to 7,800, and password login is the expensive part: one PBKDF2 costs about 2.5 ms, so the number of new logins is limited to about 200 per data core per second (308 to 342 measured on two cores). Flooding on that machine is not a stable instrument (20k to 440k delivered per second between runs) and we draw no conclusion from it. The comparison of the plain path with Mosquitto on a VPS has not been done yet. Tests: the plaintext smoke group 7 of 7 on aarch64 HVF and on x86_64 TCG, a new fuzz target plain_input (2.3 million runs with no findings).

Bugs found by live use

Two things were not caught by tests, but by the first switch-on at the real site:

• The page looked for the release files one directory too high, so every signature check answered 404 (fix 13af301). The test used a test mirror at a different path.
• The emulator keyboard never reached the guest: the cause was not a bad translation of keys, but that the dialog did not listen for keys at all (fix in ADR-232, with a boot entry that needs no keys).

Lessons

• Look for the cause, do not blame the environment. Most of the single-flow weakness and all of the dropped connections were ours; the hypervisor is real, but it had to be measured, not assumed.
• A comparison with another product must include the losses. A table with only our wins would not be a comparison.
• The measuring instrument has limits. A shared generator with 60 to 70 % stolen time changes what the numbers mean; that is why we write "lower bound" and do not claim differences below 20 %.
• Isolation is not proof. SQLite at EL0 reduces what a fault in it can do, not what it can lie about.
• A real installation finds what tests do not ask for: a wrongly assembled path and keys that nobody listens to.

What is still not proven

• The release signing key is for now a test key; v0.2.0 and v0.2.1 are pre-releases, and the images that come with them are development images with test certificates (except the template, which has no configuration).
• The 13 translations of the builder page (and of this post) were written by AI and checked by tests and spot checks, not by native speakers.
• The comparison with Mosquitto is one session, one path and one shared generator, on a shared VPS; differences below 20 % are not claims.
• The plain path has not been measured on a VPS or against Mosquitto.
• Lighthouse on the new simple view is not confirmed (old view: accessibility 95 or more).
• Bare metal, the x86_64 worker cost on KVM, Firefox and Safari for the emulator: not measured.

What comes next

• D1: a public demo broker. We cannot make the browser emulator public: it lives in one visitor's tab, is 20 to 100 times slower than KVM, has no inbound network and disappears when the tab is closed; routing the internet into someone else's browser would also be abuse. A real demo broker is possible: urMQTT on a VPS behind a name on aldo.tech, with a published test CA, a strict ACL and small limits. For now this is only an idea, not a build, and it needs explicit consent for public exposure.
• The second round for the single flow (branch perf-single-flow-wip): only once it is measured on a VPS and merged.
• The plain path against Mosquitto on a VPS, login and account revocation for plain MQTT over the signed management channel.
• After the release: bare metal (X12 to X16), IOMMU and Secure Boot, disk encryption.

The code is at github.com/zeljan-alduk/urMQTT, and docs/PROGRESS.md is still the only source of truth about what actually works, docs/PERFORMANCE.md about what has been measured. Pre-alpha, as before; bug reports and security reports following the instructions in SECURITY.md are welcome.