snapshot: add file-backed guest RAM and save/restore snapshot support#2960
Merged
jstarks merged 20 commits intomicrosoft:mainfrom Mar 19, 2026
Merged
snapshot: add file-backed guest RAM and save/restore snapshot support#2960jstarks merged 20 commits intomicrosoft:mainfrom
jstarks merged 20 commits intomicrosoft:mainfrom
Conversation
|
This PR modifies files containing For more on why we check whole files, instead of just diffs, check out the Rustonomicon |
jstarks
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
Adds snapshot save/restore plumbing to OpenVMM by introducing file-backed guest RAM, snapshot manifest/state I/O helpers, and integration points in openvmm_entry and petri/vmm_tests.
Changes:
- Add file-backed guest RAM support (CLI + worker parameters) and propagate the shared-memory backing through OpenVMM + petri.
- Add snapshot manifest format + disk I/O helpers and wire up
save-snapshotREPL +--restore-snapshot. - Add guide documentation and tests covering file-backed memory and snapshot save-to-disk paths.
Reviewed changes
Copilot reviewed 23 out of 24 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| vmm_tests/vmm_tests/tests/tests/x86_64.rs | Adds integration tests for file-backed RAM boot and snapshot save-to-disk roundtrip. |
| vmm_tests/vmm_tests/Cargo.toml | Adds openvmm_helpers dependency for snapshot/shared-memory helpers in tests. |
| petri/src/worker.rs | Extends worker launch to accept shared memory backing; adds pause/save RPC helpers. |
| petri/src/vm/openvmm/start.rs | Opens/sizes memory backing file and passes shared memory handle into worker launch. |
| petri/src/vm/openvmm/runtime.rs | Exposes pause and save_state APIs on the petri OpenVMM VM wrapper. |
| petri/src/vm/openvmm/modify.rs | Adds builder API to configure file-backed memory path. |
| petri/src/vm/openvmm/mod.rs | Adds memory_backing_file field to OpenVMM petri config struct. |
| petri/src/vm/openvmm/construct.rs | Initializes new memory_backing_file config field. |
| openvmm/openvmm_helpers/src/snapshot.rs | Introduces snapshot manifest type plus read/write/validate helpers and unit tests. |
| openvmm/openvmm_helpers/src/shared_memory.rs | Adds shared-memory helpers for opening/sizing backing files and platform handle conversion. |
| openvmm/openvmm_helpers/src/lib.rs | Exposes new shared_memory and snapshot helper modules. |
| openvmm/openvmm_helpers/Cargo.toml | Adds deps needed for snapshot/shared-memory helpers and unit tests. |
| openvmm/openvmm_entry/src/ttrpc/mod.rs | Extends worker parameters with shared_memory field (defaults to None). |
| openvmm/openvmm_entry/src/snapshot.rs | Re-exports snapshot APIs from openvmm_helpers for entry crate use. |
| openvmm/openvmm_entry/src/lib.rs | Wires snapshot restore into launch; adds save-snapshot interactive command and restore preparation. |
| openvmm/openvmm_entry/src/cli_args.rs | Adds --memory-backing-file and --restore-snapshot flags and conflict rules. |
| openvmm/openvmm_defs/src/worker.rs | Adds SharedMemoryFd alias and threads it into VmWorkerParameters. |
| openvmm/openvmm_defs/Cargo.toml | Adds Windows dependency needed for shared-memory handle support. |
| openvmm/openvmm_core/src/worker/dispatch.rs | Plumbs optional shared memory backing into VM initialization. |
| openvmm/membacking/src/memory_manager/mod.rs | Adds constructor to build SharedMemoryBacking from an existing mappable handle. |
| Guide/src/user_guide/openvmm/snapshots.md | New user-facing documentation for snapshot save/restore workflow. |
| Guide/src/dev_guide/snapshot_format.md | New developer documentation for on-disk snapshot format and validation. |
| Guide/src/SUMMARY.md | Adds new snapshot pages to the guide navigation. |
| Cargo.lock | Updates lockfile for added crates/dependencies. |
You can also share your feedback on Copilot code review. Take the survey.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds snapshot save/restore support to OpenVMM by introducing file-backed guest RAM, shared helper utilities for snapshot I/O + shared-memory handles, and new integration/unit tests and documentation covering the snapshot workflow and on-disk format.
Changes:
- Add file-backed guest RAM plumbing (CLI + worker parameters + petri support) using a shared-memory backing handle.
- Implement snapshot I/O helpers (
manifest.bin/state.bin+memory.binhardlink) and wire snapshot restore/save intoopenvmm_entry. - Add petri/vmm_tests coverage and Guide pages documenting snapshot usage and format.
Reviewed changes
Copilot reviewed 22 out of 23 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| vmm_tests/vmm_tests/tests/tests/x86_64.rs | Adds integration tests for file-backed memory boot and snapshot save-to-disk roundtrip. |
| vmm_tests/vmm_tests/Cargo.toml | Adds dependency on openvmm_helpers for snapshot/shared-memory helpers in tests. |
| petri/src/worker.rs | Extends worker launch to accept optional shared-memory backing; adds pause/save RPC helpers. |
| petri/src/vm/openvmm/start.rs | Opens/sizes memory backing file (if configured) and passes shared memory handle to worker. |
| petri/src/vm/openvmm/runtime.rs | Exposes pause/save_state/resume APIs in the petri OpenVMM runtime. |
| petri/src/vm/openvmm/modify.rs | Adds builder helper with_memory_backing_file(...). |
| petri/src/vm/openvmm/mod.rs | Stores optional memory_backing_file in config struct. |
| petri/src/vm/openvmm/construct.rs | Initializes memory_backing_file to None in defaults. |
| openvmm/openvmm_helpers/src/snapshot.rs | New snapshot manifest + read/write/validate helpers and unit tests. |
| openvmm/openvmm_helpers/src/shared_memory.rs | New helpers to open/size memory backing files and convert to OS shared-memory handles. |
| openvmm/openvmm_helpers/src/lib.rs | Exposes new shared_memory and snapshot modules. |
| openvmm/openvmm_helpers/Cargo.toml | Adds dependencies needed for snapshot/shared-memory helpers (and dev-dep tests). |
| openvmm/openvmm_entry/src/ttrpc/mod.rs | Initializes new VmWorkerParameters.shared_memory field to None. |
| openvmm/openvmm_entry/src/snapshot.rs | Re-exports snapshot helpers from openvmm_helpers. |
| openvmm/openvmm_entry/src/lib.rs | Implements snapshot save command + restore path (manifest/state read + memory.bin mapping). |
| openvmm/openvmm_entry/src/cli_args.rs | Adds --memory-backing-file and --restore-snapshot CLI flags. |
| openvmm/openvmm_defs/src/worker.rs | Adds SharedMemoryFd alias and shared_memory field to VM worker parameters. |
| openvmm/openvmm_core/src/worker/dispatch.rs | Wires shared_memory backing into InitializedVm::new(..., shared_memory). |
| openvmm/membacking/src/memory_manager/mod.rs | Adds constructor SharedMemoryBacking::from_mappable(...). |
| Guide/src/user_guide/openvmm/snapshots.md | New user-facing snapshot workflow documentation. |
| Guide/src/dev_guide/snapshot_format.md | New developer doc describing on-disk snapshot format and validation. |
| Guide/src/SUMMARY.md | Adds the new Guide pages to the navigation. |
| Cargo.lock | Lockfile updates for new dependencies. |
You can also share your feedback on Copilot code review. Take the survey.
mattkur
reviewed
Mar 17, 2026
Contributor
mattkur
left a comment
mattkur
left a comment
There was a problem hiding this comment.
Some minor doc nits. Thanks for adding a Guide page!
Contributor
There was a problem hiding this comment.
Pull request overview
Adds file-backed guest RAM support and implements snapshot save/restore using a disk-backed memory file plus persisted device/CPU state.
Changes:
- Add
--memory-backing-fileand--restore-snapshotCLI support and worker plumbing for file-backed guest memory. - Implement snapshot directory format (
manifest.bin,state.bin,memory.bin) and shared snapshot I/O helpers inopenvmm_helpers. - Add petri + vmm integration tests and documentation for snapshot workflows and on-disk format.
Reviewed changes
Copilot reviewed 22 out of 23 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| vmm_tests/vmm_tests/tests/tests/x86_64.rs | Adds integration tests for file-backed memory boot and snapshot save-to-disk. |
| vmm_tests/vmm_tests/Cargo.toml | Adds openvmm_helpers dependency for snapshot/shared-memory helpers in tests. |
| petri/src/worker.rs | Adds worker launch parameter for shared memory handle; exposes pause/save RPC helpers. |
| petri/src/vm/openvmm/start.rs | Opens/sizes requested memory backing file and passes shared memory handle to worker. |
| petri/src/vm/openvmm/runtime.rs | Adds pause/save_state APIs and serializes saved state to bytes. |
| petri/src/vm/openvmm/modify.rs | Adds builder method with_memory_backing_file(). |
| petri/src/vm/openvmm/mod.rs | Stores optional memory backing path in config. |
| petri/src/vm/openvmm/construct.rs | Initializes memory_backing_file config field. |
| openvmm/openvmm_helpers/src/snapshot.rs | New snapshot manifest type + read/write/validate helpers with unit tests. |
| openvmm/openvmm_helpers/src/shared_memory.rs | New helpers to open/size backing file and convert to shared-memory handle. |
| openvmm/openvmm_helpers/src/lib.rs | Exposes new snapshot and shared_memory modules. |
| openvmm/openvmm_helpers/Cargo.toml | Adds deps (fs-err, sparse_mmap, tempfile) for snapshot/shared-memory helpers. |
| openvmm/openvmm_entry/src/ttrpc/mod.rs | Initializes new shared_memory worker parameter to None. |
| openvmm/openvmm_entry/src/snapshot.rs | Re-exports snapshot helpers for entry crate usage. |
| openvmm/openvmm_entry/src/lib.rs | Implements snapshot save command + restore flow wiring into worker parameters. |
| openvmm/openvmm_entry/src/cli_args.rs | Adds CLI flags and conflict rules for snapshot restore + memory backing file. |
| openvmm/openvmm_defs/src/worker.rs | Adds SharedMemoryFd type + new worker parameter shared_memory. |
| openvmm/openvmm_core/src/worker/dispatch.rs | Wires shared memory backing into VM initialization via SharedMemoryBacking. |
| openvmm/membacking/src/memory_manager/mod.rs | Adds constructor SharedMemoryBacking::from_mappable(). |
| Guide/src/user_guide/openvmm/snapshots.md | New user-facing snapshot save/restore guide. |
| Guide/src/dev_guide/snapshot_format.md | New developer documentation for snapshot on-disk format. |
| Guide/src/SUMMARY.md | Adds new snapshot docs pages to the guide TOC. |
You can also share your feedback on Copilot code review. Take the survey.
smalis-msft
reviewed
Mar 17, 2026
| |-----------------|---------------------------------------------| | ||
| | `manifest.bin` | Protobuf-encoded snapshot metadata | | ||
| | `state.bin` | Serialized device state | | ||
| | `memory.bin` | Hard link to the memory backing file | |
Contributor
There was a problem hiding this comment.
Suggested change
| | `memory.bin` | Hard link to the memory backing file | | |
| | `memory.bin` | Memory backing file | |
Member
There was a problem hiding this comment.
no i think it really is a hard link, right? look at snapshot_format
Contributor
There was a problem hiding this comment.
It is a hard link, I just don't think that's detail that needs to be in the guide overview
smalis-msft
reviewed
Mar 17, 2026
smalis-msft
reviewed
Mar 17, 2026
|
|
||
| ## Saving a snapshot | ||
|
|
||
| Start a VM with file-backed memory, then use the interactive console to save: |
Contributor
There was a problem hiding this comment.
Suggested change
| Start a VM with file-backed memory, then use the interactive console to save: | |
| Start a VM with file-backed memory: |
smalis-msft
reviewed
Mar 17, 2026
| ```admonish warning | ||
| After saving, the VM remains **paused**. Do **not** resume the VM — resuming | ||
| would mutate guest RAM through the hard-linked `memory.bin`, corrupting the | ||
| snapshot. Use `shutdown` to exit OpenVMM after saving. |
Contributor
There was a problem hiding this comment.
Should save-snapshot automatically shutdown afterwards?
Member
There was a problem hiding this comment.
it sure seems like we should block users from resuming in this state, right? or do you mean that once you resume, you must pause and save again? should there be some run epoch to enforce this?
smalis-msft
reviewed
Mar 17, 2026
smalis-msft
reviewed
Mar 17, 2026
| each restore. | ||
| - VMs using VPCI or PCIe devices do not currently support save/restore | ||
| - OpenHCL-based VMs do not currently support this snapshot mechanism | ||
| - PCAT firmware does not support save/restore |
Contributor
There was a problem hiding this comment.
Suggested change
| - PCAT firmware does not support save/restore | |
| - VMs using PCAT firmware does not support save/restore |
smalis-msft
reviewed
Mar 17, 2026
smalis-msft
reviewed
Mar 17, 2026
smalis-msft
reviewed
Mar 17, 2026
smalis-msft
reviewed
Mar 17, 2026
Add 'move' to closures capturing mem_path in file_backed_memory_boot and snapshot_save_to_disk tests. Clone mem_path before the closure so it remains available for assertions after the VM is launched.
- Restructure write_snapshot to use needs_link flag instead of early return
- Fix docs to match actual CLI names (save-snapshot, --restore-snapshot)
- Only resize memory backing file when newly created; error on size mismatch
- Use system page size via SparseMapping::page_size() instead of hardcoded 4096
- Change created_at from String to mesh::payload::Timestamp
- Use fs_err consistently in snapshot.rs and callers
- Add manifest version check on restore
- Add exit guidance ('shutdown') after save-snapshot
- Extract open_memory_backing_file/file_to_shared_memory_fd into openvmm_defs
to remove duplication between openvmm_entry and petri
- Remove unused pub use Mappable from membacking
- Move Snapshot Format doc next to Save State in Guide SUMMARY
Move shared_memory helpers (open_memory_backing_file, file_to_shared_memory_fd) from openvmm_defs::worker to openvmm_helpers::shared_memory. The SharedMemoryFd type alias stays in openvmm_defs since it's part of VmWorkerParameters. Move snapshot module (SnapshotManifest, write_snapshot, read_snapshot, validate_manifest) from openvmm_entry to openvmm_helpers::snapshot so petri tests can reuse the same code. Remove CLI-specific error message referencing --memory-backing-file. openvmm_entry::snapshot re-exports from openvmm_helpers. Extract SaveSnapshot interactive command logic into a dedicated save_snapshot() function so errors are caught and printed instead of propagating via ? and terminating the VM. Remove redundant raw_os_error EXDEV check in write_snapshot (CrossesDevices already covers it). Update the snapshot_save_to_disk VMM test to use the shared snapshot module instead of reimplementing the write logic.
- Replace std::env::consts::ARCH with a GUEST_ARCH const derived from cfg!(guest_arch) in snapshot save/restore code - Use call_failable(VmRpc::Save, ()) instead of manual result mapping - Update docs to reference guest architecture instead of host - Hardcode "x86_64" in x86_64-specific test instead of using ARCH
- Link SnapshotManifest to generated rustdoc - Cross-reference Save State rules in Device state and Extending sections
smalis-msft
reviewed
Mar 19, 2026
| - VMs using PCAT firmware do not support save/restore | ||
| - `--memory` and `--processors` must be specified on restore and match the | ||
| snapshot manifest values. A future version may read these from the snapshot | ||
| automatically. |
Contributor
There was a problem hiding this comment.
Should mention here that extra device configuration needs to be specified on the command line during restore too, and will not be loaded from the snapshot without it
smalis-msft
reviewed
Mar 19, 2026
smalis-msft
reviewed
Mar 19, 2026
smalis-msft
reviewed
Mar 19, 2026
| `SaveError::NotSupported`. | ||
| - **NVMe** — `NvmeController` returns `SaveError::NotSupported`. | ||
| - **Pass-through PCI** — `AssignedPciDevice`, `RelayedVpciDevice`. | ||
| - **VGA / GDMA** — marked `todo!()` (will panic on save). |
Contributor
There was a problem hiding this comment.
Is this the only thing blocking PCAT save/restore?
Member
Author
There was a problem hiding this comment.
Probably--we support PCAT save/restore with OpenHCL, so this is probably the only thing left.
smalis-msft
reviewed
Mar 19, 2026
| | Scenario | Result | | ||
| |---|---| | ||
| | Device set matches exactly | Restore succeeds | | ||
| | Snapshot contains a device not in current config | **Restore fails** — unknown unit name | |
Contributor
There was a problem hiding this comment.
So presumably someday we'll have to add something for hot-remove?
smalis-msft
reviewed
Mar 19, 2026
| The following table summarises support for the device types relevant to | ||
| OpenVMM snapshots: | ||
|
|
||
| | Device | Bus | Save/Restore | |
Contributor
There was a problem hiding this comment.
I feel like this will get out of date quickly
Contributor
There was a problem hiding this comment.
Pull request overview
Adds snapshot save/restore support using file-backed guest RAM, wiring it through OpenVMM CLI/REPL, worker initialization, shared helpers, and Petri/vmm_tests coverage.
Changes:
- Introduces file-backed guest RAM plumbing (CLI flags, worker parameter, Petri support) via a reusable shared-memory helper.
- Implements snapshot directory I/O (manifest/state + hardlinked memory file) and restore validation/boot path.
- Adds integration/unit tests plus user/dev documentation for snapshot workflows and on-disk format.
Reviewed changes
Copilot reviewed 21 out of 22 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| vmm_tests/vmm_tests/tests/tests/x86_64.rs | Adds Petri integration tests for file-backed RAM boot and save-to-disk snapshot roundtrip. |
| vmm_tests/vmm_tests/Cargo.toml | Pulls in openvmm_helpers for snapshot/shared-memory helpers in tests. |
| petri/src/worker.rs | Adds pause/save RPC wrappers and plumbs shared-memory handle into worker launch. |
| petri/src/vm/openvmm/start.rs | Opens/creates memory backing file and passes shared memory handle when launching worker. |
| petri/src/vm/openvmm/runtime.rs | Exposes pause/save_state VM APIs and encodes saved state for disk persistence. |
| petri/src/vm/openvmm/modify.rs | Adds builder method to enable file-backed guest memory in Petri. |
| petri/src/vm/openvmm/mod.rs | Stores optional memory backing file path in Petri VM config. |
| petri/src/vm/openvmm/construct.rs | Initializes new Petri config field (memory_backing_file). |
| openvmm/openvmm_helpers/src/snapshot.rs | Implements snapshot manifest type, write/read helpers, and manifest validation (+ unit tests). |
| openvmm/openvmm_helpers/src/shared_memory.rs | Adds helpers to open/size backing files and convert into platform shared-memory handles. |
| openvmm/openvmm_helpers/src/lib.rs | Exports new shared_memory and snapshot helper modules. |
| openvmm/openvmm_helpers/Cargo.toml | Adds dependencies for snapshot I/O and Windows shared-mem mapping. |
| openvmm/openvmm_entry/src/ttrpc/mod.rs | Initializes new worker parameter field (shared_memory) for ttrpc usage. |
| openvmm/openvmm_entry/src/lib.rs | Adds restore preparation, save-snapshot command, and worker launch plumbing for shared memory + saved state. |
| openvmm/openvmm_entry/src/cli_args.rs | Adds --memory-backing-file / --restore-snapshot and enforces mutual exclusivity. |
| openvmm/openvmm_defs/src/worker.rs | Defines portable shared-memory handle type and adds it to worker launch parameters. |
| openvmm/openvmm_core/src/worker/dispatch.rs | Passes shared-memory backing into VM initialization. |
| openvmm/membacking/src/memory_manager/mod.rs | Adds constructor to build SharedMemoryBacking from a Mappable. |
| Guide/src/user_guide/openvmm/snapshots.md | Documents snapshot save/restore usage and limitations. |
| Guide/src/dev_guide/snapshot_format.md | Documents snapshot directory layout and format details for developers. |
| Guide/src/SUMMARY.md | Links new snapshot docs into the guide summary. |
You can also share your feedback on Copilot code review. Take the survey.
| let saved_state_bytes = mesh::payload::encode(saved_state_msg); | ||
|
|
||
| // Fsync the memory backing file. | ||
| let memory_file = fs_err::File::open(memory_file_path)?; |
smalis-msft
reviewed
Mar 19, 2026
| | virtio-net | Virtio (PCI/MMIO) | Yes | | ||
| | virtio-pmem | Virtio (PCI/MMIO) | Yes | | ||
| | virtio-rng | Virtio (PCI/MMIO) | Yes | | ||
| | NVMe | PCI | **No** | |
Contributor
There was a problem hiding this comment.
Do we need to go add any of these for this particular sprint?
smalis-msft
previously approved these changes
Mar 19, 2026
benhillis
approved these changes
Mar 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add snapshot save/restore support to OpenVMM, built on file-backed guest RAM.
File-backed guest memory
Add
--memory-backing-file <path>CLI flag to back guest RAM with a regular file instead of anonymous memory. The file is created (or opened) and sized to match the configured guest RAM. On Unix the fd is directly mmappable; on Windows a section handle is created. This reuses the existingSharedMemoryBacking→GuestMemoryBuilder::existing_backing()code path.--memory-backing-fileand--private-memoryare made mutually exclusive at the CLI level.Snapshot save
Add a
save-snapshot <dir>REPL command (alias:snap) that:VmRpc::Savemanifest.bin,state.bin, and a hardlink ofmemory.binto the snapshot directoryThe on-disk format uses a
SnapshotManifestprotobuf containing version, architecture, memory size, VP count, and page size.Snapshot restore
Add
--restore-snapshot <dir>CLI flag that restores a VM from a previously saved snapshot directory:manifest.binandstate.binmemory.binas the shared memory backingLoadMode::None)Shared helpers (
openvmm_helpers)Extract reusable snapshot I/O (
write_snapshot,read_snapshot,validate_manifest) and shared memory helpers (open_memory_backing_file,file_to_shared_memory_fd) intoopenvmm_helpersso bothopenvmm_entryand petri tests can use them.Tests
file_backed_memory_boot: boots a VM with file-backed RAM and verifies the backing file existssnapshot_save_to_disk: full save-to-disk integration test — boot, pause, save state, write to disk, verify roundtrip, resume, confirm VM is still functionalDocumentation
Guide/src/user_guide/openvmm/snapshots.md— save/restore workflowGuide/src/dev_guide/snapshot_format.md— on-disk format specification