Skip to content

perf(firecracker): UFFD-probe default snapshot restore + faster socket poll + full-copy warn#28

Merged
drewstone merged 5 commits into
mainfrom
feat/uffd-default-fc-spawn-perf
Jul 17, 2026
Merged

perf(firecracker): UFFD-probe default snapshot restore + faster socket poll + full-copy warn#28
drewstone merged 5 commits into
mainfrom
feat/uffd-default-fc-spawn-perf

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Three restore/spawn-path wins from the firecracker-core audit, each in its own commit, plus the alpha bump. All perf numbers below are code-comment estimates, not bench measurements on a real host — labeled per item.

1. UFFD memory backend auto-default for snapshot restore

MICROVM_MEM_BACKEND unset now resolves through a host probe instead of hard-defaulting to File:

  • /proc/sys/vm/unprivileged_userfaultfd reads 1Uffd (VM resumes immediately, guest RAM pages in lazily).
  • Otherwise → File plus a one-line stderr warning naming the sysctl and the explicit override.
  • An explicit MICROVM_MEM_BACKEND=file|uffd always wins, both directions.
  • MemBackend::default() stays File, so programmatically built configs are unchanged — only from_env() auto-detects.

Why the sysctl and not a userfaultfd(2) attempt in-process: the uffd is created by the Firecracker process (it hands the fd to our handler over the UDS). A jailed FC runs uid-dropped and can be weaker than this process, so a syscall probe here would say yes while the jailed restore fails. The sysctl is the one signal valid for every FC we spawn. Cost of that conservatism: a privileged, non-jailed FC on a sysctl-off host doesn't get the auto-win — those operators set MEM_BACKEND=uffd explicitly (README documents this).

  • Win: ~150ms → ~5ms per warm restore for a 4GiB guest (estimate from src/uffd.rs module docs, not benched).
  • Risk: capable hosts silently change restore backend on upgrade. The UFFD handler path itself is unchanged and was already the supported explicit setting; incapable hosts keep File.
  • Behavior-default change (loud): operator boxes with vm.unprivileged_userfaultfd=1 will start using UFFD restores after the consumer bumps. Roll back per-host with MICROVM_MEM_BACKEND=file.

2. Socket-ready poll: 2ms → 100ms exponential backoff

wait_for_socket_ready slept a flat 100ms between probes; the FC API socket is typically ready in <20ms, so nearly every spawn idled 50–90ms. Now 2, 4, 8, …, capped at the old 100ms; timeout semantics unchanged.

  • Win: ~50–90ms off every FC spawn (inferred from the old flat interval, not benched).
  • Risk: negligible — slow hosts converge to exactly the old cadence.

3. track_dirty_pages default off

Verified in this tree: the adapter only ever issues snapshot_type: "Full" creates (firecracker.rs create_snapshot) and enable_diff_snapshots: false loads — nothing consumes the dirty bitmap, which costs a write-protect fault per first-touch guest write. Unset specs now send false; VmSpec.track_dirty_pages = Some(true) still re-enables per-VM.

  • Win: removes per-write KVM dirty-log overhead on every VM (unbenched).
  • Behavior-default change (loud): anyone driving diff snapshots against the FC socket out-of-band (nothing in-tree does) must set Some(true).

4. Rootfs clone: warn once on full-copy fallback

The reflink → hardlink → copy ladder now reports its landing rung, and the first clone that reaches the full byte copy emits a one-time [microvm-rootfs] warning with the remediation (btrfs / XFS reflink=1). Previously a writable rootfs on ext4 silently byte-copied multi-GB images per VM.

  • Win: observability only; no fallback-order change.

Tests

  • New: probe-absent→Uffd / probe-absent-no→File / explicit-env-wins / Default stays File; backoff schedule 2,4,8,16,32,64,100,100 + sticky cap; machine-config body defaults track_dirty_pages=false and stays settable; SharedOk clone never lands on FullCopy within one fs; Independent clone never shares the source inode.
  • Gates run locally (mirrors ci.yml): cargo fmt --check ✓, cargo clippy --all-targets --all-features -D warnings ✓, cargo test --all-features 219 passed / 0 failed / 7 pre-existing ignores (+1 doctest) ✓, cargo test --no-default-features ✓, cargo doc --all-features -D warnings ✓.

Release (maintainer step — nothing here publishes)

No publish workflow exists (.github/workflows/ has only ci.yml), so the release is manual. Cargo.toml is already bumped to 0.4.0-alpha.4 in this branch. After merge:

git checkout main && git pull
cargo publish --dry-run
cargo publish
git tag v0.4.0-alpha.4 && git push origin v0.4.0-alpha.4

Consumer bump (sandbox-runtime → microvm-runtime = "0.4.0-alpha.4") is a separate follow-up.

…t allows it

MICROVM_MEM_BACKEND unset now resolves via a host probe instead of
hard-defaulting to File: if /proc/sys/vm/unprivileged_userfaultfd is 1,
from_env picks the Uffd backend (VM resumes immediately, pages fault in
lazily) instead of File (FC reads the entire guest-RAM file before
resume — ~150ms for a 4GiB guest per the uffd module's estimate).

The sysctl is the probe — not a userfaultfd(2) attempt in this process —
because the uffd is created by the Firecracker process itself; a jailed,
uid-dropped FC can be weaker than this process, so a syscall probe here
would over-approximate. Sysctl off → File plus a one-line stderr warning
naming the sysctl and the explicit override.

Behavior default change (from_env only): capable hosts silently gain
lazy restores. An explicit MICROVM_MEM_BACKEND always wins in both
directions, and MemBackend::default() stays File so programmatically
built configs are unchanged.
wait_for_socket_ready slept a flat 100 ms between readiness probes, but
the FC API socket is typically ready well under 20 ms after exec — so
nearly every spawn ate 50–90 ms of pure idle wait. Poll at 2 ms first,
doubling to the old 100 ms cap (2, 4, 8, …, 100), so fast hosts are
detected within single-digit ms and slow hosts converge to exactly the
previous cadence. Timeout semantics unchanged.
…e no bitmap consumer

/machine-config was sent track_dirty_pages=true whenever the spec left
it unset, but this adapter only ever issues snapshot_type=Full creates
and enable_diff_snapshots=false loads — the dirty-page bitmap costs a
write-protect fault on every first-touch guest write and nothing reads
it. Default it to false; VmSpec.track_dirty_pages=Some(true) still
re-enables it per-VM for externally driven diff snapshots.

Behavior default change: specs that relied on the implicit true (none
in-tree; diff snapshots were never issuable through this crate) must
now set Some(true) explicitly.
… back to full byte copy

clone_file_with_mode now reports which rung of the reflink → hardlink →
copy ladder a clone landed on, and the first clone that reaches the full
streaming copy emits a one-time [microvm-rootfs] warning naming the
remediation (btrfs / XFS reflink=1). Previously a writable rootfs on
ext4 silently byte-copied the whole multi-GB image for every VM with
nothing in the logs pointing at the slow path.

No behavior change to the fallback order itself; the strategy enum is
crate-internal.

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — d71aaee6

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-17T23:20:37Z

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants