perf(firecracker): UFFD-probe default snapshot restore + faster socket poll + full-copy warn#28
Merged
Merged
Conversation
…t allows it MICROVM_MEM_BACKEND unset now resolves via a host probe instead of hard-defaulting to File: if /proc/sys/vm/unprivileged_userfaultfd is 1, from_env picks the Uffd backend (VM resumes immediately, pages fault in lazily) instead of File (FC reads the entire guest-RAM file before resume — ~150ms for a 4GiB guest per the uffd module's estimate). The sysctl is the probe — not a userfaultfd(2) attempt in this process — because the uffd is created by the Firecracker process itself; a jailed, uid-dropped FC can be weaker than this process, so a syscall probe here would over-approximate. Sysctl off → File plus a one-line stderr warning naming the sysctl and the explicit override. Behavior default change (from_env only): capable hosts silently gain lazy restores. An explicit MICROVM_MEM_BACKEND always wins in both directions, and MemBackend::default() stays File so programmatically built configs are unchanged.
wait_for_socket_ready slept a flat 100 ms between readiness probes, but the FC API socket is typically ready well under 20 ms after exec — so nearly every spawn ate 50–90 ms of pure idle wait. Poll at 2 ms first, doubling to the old 100 ms cap (2, 4, 8, …, 100), so fast hosts are detected within single-digit ms and slow hosts converge to exactly the previous cadence. Timeout semantics unchanged.
…e no bitmap consumer /machine-config was sent track_dirty_pages=true whenever the spec left it unset, but this adapter only ever issues snapshot_type=Full creates and enable_diff_snapshots=false loads — the dirty-page bitmap costs a write-protect fault on every first-touch guest write and nothing reads it. Default it to false; VmSpec.track_dirty_pages=Some(true) still re-enables it per-VM for externally driven diff snapshots. Behavior default change: specs that relied on the implicit true (none in-tree; diff snapshots were never issuable through this crate) must now set Some(true) explicitly.
… back to full byte copy clone_file_with_mode now reports which rung of the reflink → hardlink → copy ladder a clone landed on, and the first clone that reaches the full streaming copy emits a one-time [microvm-rootfs] warning naming the remediation (btrfs / XFS reflink=1). Previously a writable rootfs on ext4 silently byte-copied the whole multi-GB image for every VM with nothing in the logs pointing at the slow path. No behavior change to the fallback order itself; the strategy enum is crate-internal.
tangletools
approved these changes
Jul 17, 2026
tangletools
left a comment
Contributor
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — d71aaee6
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-07-17T23:20:37Z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three restore/spawn-path wins from the firecracker-core audit, each in its own commit, plus the alpha bump. All perf numbers below are code-comment estimates, not bench measurements on a real host — labeled per item.
1. UFFD memory backend auto-default for snapshot restore
MICROVM_MEM_BACKENDunset now resolves through a host probe instead of hard-defaulting toFile:/proc/sys/vm/unprivileged_userfaultfdreads1→Uffd(VM resumes immediately, guest RAM pages in lazily).Fileplus a one-line stderr warning naming the sysctl and the explicit override.MICROVM_MEM_BACKEND=file|uffdalways wins, both directions.MemBackend::default()staysFile, so programmatically built configs are unchanged — onlyfrom_env()auto-detects.Why the sysctl and not a
userfaultfd(2)attempt in-process: the uffd is created by the Firecracker process (it hands the fd to our handler over the UDS). A jailed FC runs uid-dropped and can be weaker than this process, so a syscall probe here would say yes while the jailed restore fails. The sysctl is the one signal valid for every FC we spawn. Cost of that conservatism: a privileged, non-jailed FC on a sysctl-off host doesn't get the auto-win — those operators setMEM_BACKEND=uffdexplicitly (README documents this).src/uffd.rsmodule docs, not benched).File.vm.unprivileged_userfaultfd=1will start using UFFD restores after the consumer bumps. Roll back per-host withMICROVM_MEM_BACKEND=file.2. Socket-ready poll: 2ms → 100ms exponential backoff
wait_for_socket_readyslept a flat 100ms between probes; the FC API socket is typically ready in <20ms, so nearly every spawn idled 50–90ms. Now 2, 4, 8, …, capped at the old 100ms; timeout semantics unchanged.3.
track_dirty_pagesdefault offVerified in this tree: the adapter only ever issues
snapshot_type: "Full"creates (firecracker.rscreate_snapshot) andenable_diff_snapshots: falseloads — nothing consumes the dirty bitmap, which costs a write-protect fault per first-touch guest write. Unset specs now sendfalse;VmSpec.track_dirty_pages = Some(true)still re-enables per-VM.Some(true).4. Rootfs clone: warn once on full-copy fallback
The reflink → hardlink → copy ladder now reports its landing rung, and the first clone that reaches the full byte copy emits a one-time
[microvm-rootfs]warning with the remediation (btrfs / XFSreflink=1). Previously a writable rootfs on ext4 silently byte-copied multi-GB images per VM.Tests
Defaultstays File; backoff schedule2,4,8,16,32,64,100,100+ sticky cap; machine-config body defaultstrack_dirty_pages=falseand stays settable; SharedOk clone never lands on FullCopy within one fs; Independent clone never shares the source inode.cargo fmt --check✓,cargo clippy --all-targets --all-features -D warnings✓,cargo test --all-features219 passed / 0 failed / 7 pre-existing ignores (+1 doctest) ✓,cargo test --no-default-features✓,cargo doc --all-features -D warnings✓.Release (maintainer step — nothing here publishes)
No publish workflow exists (
.github/workflows/has onlyci.yml), so the release is manual.Cargo.tomlis already bumped to0.4.0-alpha.4in this branch. After merge:Consumer bump (sandbox-runtime →
microvm-runtime = "0.4.0-alpha.4") is a separate follow-up.