Content hash based resources#4148
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a persistent resource storage system for Graphite, replacing the practice of embedding raw image data directly within the node graph. It implements a ResourceStorage trait with specialized backends: IndexedDbResourceStorage for web environments and MmapResourceStorage for desktop, utilizing Blake3 for content hashing. Key architectural changes include a preprocessor that automatically swaps resource hashes for dedicated resource nodes and updates to the document save/load flow to handle externalized assets. Feedback focuses on scalability and stability concerns, specifically the memory-intensive hydration of the entire IndexedDB store on startup, the use of expect() and panic! which could lead to application crashes, and the inefficient re-initialization of HTTP clients during resource loading.
| self.resize(); | ||
|
|
||
| self.desktop_wrapper.init(self.wgpu_context.clone()); | ||
| let resource_storage = MmapResourceStorage::new(dirs::app_resources_dir()).expect("Failed to initialize on-disk resource storage"); |
There was a problem hiding this comment.
Using .expect() here will cause the application to crash immediately if the resources directory cannot be created or accessed (e.g., due to permission issues). While this occurs at startup, it would be better to handle this error gracefully and show a message to the user, especially since this directory is critical for the new resource system.
| async fn load_resource<'a: 'n>(_: impl Ctx, _primary: (), #[scope("editor-api")] _editor: &'a PlatformEditorApi, #[name("URL")] url: String) -> Arc<[u8]> { | ||
| let placeholder = || -> Arc<[u8]> { Arc::from(Vec::<u8>::new()) }; | ||
|
|
||
| let response = match reqwest::Client::new().get(&url).send().await { |
There was a problem hiding this comment.
Creating a new reqwest::Client for every execution of the load_resource node is inefficient. reqwest clients are intended to be reused to benefit from internal connection pooling and to avoid the overhead of re-initializing the client state for every request.
| let response = match reqwest::Client::new().get(&url).send().await { | |
| let response = match _editor.application_io.as_ref().map(|io| io.http_client()).unwrap_or_else(reqwest::Client::new).get(&url).send().await { |
| application_io.load_resource(&hash).unwrap_or_else(|| { | ||
| panic!("Resource {hash} not found"); | ||
| }) |
There was a problem hiding this comment.
Using panic! inside a node is generally discouraged as it will crash the entire node graph executor and potentially the application. If a resource is missing (e.g., if it was manually deleted from the disk), it would be better to return an empty Resource or a placeholder image and log a warning instead of crashing.
application_io.load_resource(&hash).unwrap_or_else(|| {
log::error!("Resource {hash} not found");
Resource::new(Vec::new())
})c902384 to
31b7b33
Compare
4b7a823 to
847b8e9
Compare
15fcaac to
d5f0140
Compare
058da30 to
8200b15
Compare
35eec4d to
3c27e65
Compare
1ff1dbd to
1b08a3a
Compare
There was a problem hiding this comment.
19 issues found across 44 files
Confidence score: 2/5
- Merge risk is high because several high-confidence 8/10 issues are user-impacting: panic paths in
node-graph/nodes/gstd/src/platform_application_io.rsandnode-graph/graph-craft/src/application_io.rscan crash resource loading/render flows when resources or hashes are missing. - There is concrete document integrity risk across
node-graph/graph-craft/src/application_io/resource/indexed_db.rsandeditor/src/messages/resource/resource_message_handler.rs: reads/exports depend on in-memory cache and can omit resources that still exist only in IndexedDB, leading to broken saves. - Migration and persistence paths also look fragile in
editor/src/messages/portfolio/document_migration.rsandeditor/src/messages/portfolio/portfolio_message_handler.rs, where legacy shapes/state cleanup can be skipped and leave incompatible or stale persisted data. - Pay close attention to
node-graph/nodes/gstd/src/platform_application_io.rs,node-graph/graph-craft/src/application_io/resource/indexed_db.rs, andeditor/src/messages/resource/resource_message_handler.rs- crash and data-loss behaviors are the main merge blockers.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="node-graph/graph-craft/src/application_io/wasm.rs">
<violation number="1" location="node-graph/graph-craft/src/application_io/wasm.rs:1">
P1: Custom agent: **PR title enforcement**
PR title is not in imperative mood and lacks a leading action verb</violation>
</file>
<file name="editor/src/messages/portfolio/document/utility_types/embedded_resources.rs">
<violation number="1" location="editor/src/messages/portfolio/document/utility_types/embedded_resources.rs:47">
P2: Serializing the `HashMap` directly makes saved document output nondeterministic. Sort the resource entries before writing them so identical documents serialize identically.</violation>
</file>
<file name="node-graph/libraries/application-io/src/lib.rs">
<violation number="1" location="node-graph/libraries/application-io/src/lib.rs:155">
P2: Hash the shared `Io` allocation here, not the `Arc` handle's address. As written, `Arc::clone` of the same `application_io` produces different `EditorApi` hashes and can cause avoidable cache misses.
(Based on your team's feedback about avoiding misleading or inconsistent hashing implementations.) [FEEDBACK_USED]</violation>
</file>
<file name="node-graph/graph-craft/src/application_io.rs">
<violation number="1" location="node-graph/graph-craft/src/application_io.rs:68">
P1: Don't `expect` on `resources` here. `new()`/`default()` leave it unset, and some call sites (for example the CLI) construct `PlatformApplicationIo` without calling `inject_resources()`, so loading a content-hash resource can abort the whole process.
(Based on your team's feedback about avoiding panics in application code.) [FEEDBACK_USED]</violation>
</file>
<file name="editor/src/dispatcher.rs">
<violation number="1" location="editor/src/dispatcher.rs:42">
P2: `with_executor()` now leaves the resource subsystem uninitialized, so non-test callers built through this constructor will drop `ResourceMessage`s and can panic when a resource handle is requested.
(Based on your team's feedback about making invalid states unrepresentable in data models.) [FEEDBACK_USED]</violation>
</file>
<file name="node-graph/graph-craft/src/application_io/resource/mmap.rs">
<violation number="1" location="node-graph/graph-craft/src/application_io/resource/mmap.rs:56">
P2: Verify the mapped bytes still hash to the requested `ResourceHash` before returning them.</violation>
<violation number="2" location="node-graph/graph-craft/src/application_io/resource/mmap.rs:107">
P1: Don't report a successful resource hash after the filesystem write failed. Otherwise callers can store a hash that this backend can never load.</violation>
</file>
<file name="node-graph/nodes/gstd/src/platform_application_io.rs">
<violation number="1" location="node-graph/nodes/gstd/src/platform_application_io.rs:140">
P2: Use an actual transparent image placeholder here; empty bytes make failed loads disappear instead of preserving the transparent fallback promised by this node.</violation>
<violation number="2" location="node-graph/nodes/gstd/src/platform_application_io.rs:142">
P2: This now treats every resource as an HTTP URL, so local asset paths documented for this node will always fail.</violation>
<violation number="3" location="node-graph/nodes/gstd/src/platform_application_io.rs:265">
P1: Don't panic when the resource backend or hash is missing; this node can crash the whole render path on ordinary load failures.
(Based on your team's feedback about avoiding panics in application code.) [FEEDBACK_USED]</violation>
</file>
<file name="node-graph/graph-craft/src/application_io/resource.rs">
<violation number="1" location="node-graph/graph-craft/src/application_io/resource.rs:12">
P2: This `Mutex` is unnecessary for the in-memory store; it only adds contention and panic-on-poison paths to code that could use a plain `HashMap`.
(Based on your team's feedback about reusing existing abstractions instead of adding mutexes.) [FEEDBACK_USED]</violation>
</file>
<file name="editor/src/application.rs">
<violation number="1" location="editor/src/application.rs:12">
P2: `Editor::new()` now returns a partially initialized editor and depends on a separate `replace_application_io()` call before graph execution is safe.
(Based on your team's feedback about placing integrity-enforcing initialization where the state is created.) [FEEDBACK_USED]</violation>
</file>
<file name="editor/src/messages/portfolio/document_migration.rs">
<violation number="1" location="editor/src/messages/portfolio/document_migration.rs:1594">
P1: Handle the legacy 1-input image node shape here too; after the alias remap, older embedded-image documents skip this migration and keep an incompatible input layout.</violation>
</file>
<file name="editor/src/messages/resource/resource_message_handler.rs">
<violation number="1" location="editor/src/messages/resource/resource_message_handler.rs:35">
P1: `export()` silently drops resources that are missing from the in-memory cache, which can save broken documents on the IndexedDB backend.</violation>
<violation number="2" location="editor/src/messages/resource/resource_message_handler.rs:54">
P1: The non-test default creates an unusable `ResourceMessageHandler`: it can panic in `resources()` or silently export no resources during save.
(Based on your team's feedback about avoiding panics and making invalid states unrepresentable.) [FEEDBACK_USED]</violation>
</file>
<file name="editor/src/messages/portfolio/document/graph_operation/utility_types.rs">
<violation number="1" location="editor/src/messages/portfolio/document/graph_operation/utility_types.rs:304">
P1: This rewires the bitmap layer to a hash before the resource write can fail safely. If `ResourceMessage::Write` hits an I/O error, the node still keeps the hash-only reference and later resource loading panics when that hash is missing.
(Based on your team's feedback about avoiding panics in application code.) [FEEDBACK_USED]</violation>
</file>
<file name="node-graph/graph-craft/src/application_io/resource/indexed_db.rs">
<violation number="1" location="node-graph/graph-craft/src/application_io/resource/indexed_db.rs:177">
P1: `read()` only checks the cache, so saving a reopened autosaved document can omit resources that still live only in IndexedDB.</violation>
</file>
<file name="editor/src/messages/portfolio/portfolio_message_handler.rs">
<violation number="1" location="editor/src/messages/portfolio/portfolio_message_handler.rs:452">
P1: This GC fallback loads and selects another document, then aborts the cleanup pass for legacy persisted state entries.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| @@ -1,105 +0,0 @@ | |||
| use dyn_any::StaticType; | |||
There was a problem hiding this comment.
P1: Custom agent: PR title enforcement
PR title is not in imperative mood and lacks a leading action verb
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node-graph/graph-craft/src/application_io/wasm.rs:
<comment>PR title is not in imperative mood and lacks a leading action verb</comment>
<file context>
@@ -1,105 +0,0 @@
-use dyn_any::StaticType;
-use graphene_application_io::{ApplicationError, ApplicationIo, ResourceFuture};
-use std::collections::HashMap;
-use std::sync::Arc;
-#[cfg(feature = "tokio")]
-use tokio::io::AsyncReadExt;
-#[cfg(target_family = "wasm")]
-use wasm_bindgen::JsCast;
-#[cfg(feature = "wgpu")]
</file context>
| } | ||
|
|
||
| fn load_resource(&self, hash: ResourceHash) -> graphene_application_io::ResourceFuture { | ||
| self.resources.as_ref().expect("Resource storage not initialized").load(hash) |
There was a problem hiding this comment.
P1: Don't expect on resources here. new()/default() leave it unset, and some call sites (for example the CLI) construct PlatformApplicationIo without calling inject_resources(), so loading a content-hash resource can abort the whole process.
(Based on your team's feedback about avoiding panics in application code.)
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node-graph/graph-craft/src/application_io.rs, line 68:
<comment>Don't `expect` on `resources` here. `new()`/`default()` leave it unset, and some call sites (for example the CLI) construct `PlatformApplicationIo` without calling `inject_resources()`, so loading a content-hash resource can abort the whole process.
(Based on your team's feedback about avoiding panics in application code.) </comment>
<file context>
@@ -1,14 +1,93 @@
+ }
+
+ fn load_resource(&self, hash: ResourceHash) -> graphene_application_io::ResourceFuture {
+ self.resources.as_ref().expect("Resource storage not initialized").load(hash)
+ }
+}
</file context>
| })(); | ||
|
|
||
| if let Err(error) = write_result { | ||
| log::error!("Failed to write resource to {path:?}: {error}"); |
There was a problem hiding this comment.
P1: Don't report a successful resource hash after the filesystem write failed. Otherwise callers can store a hash that this backend can never load.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node-graph/graph-craft/src/application_io/resource/mmap.rs, line 107:
<comment>Don't report a successful resource hash after the filesystem write failed. Otherwise callers can store a hash that this backend can never load.</comment>
<file context>
@@ -0,0 +1,160 @@
+ })();
+
+ if let Err(error) = write_result {
+ log::error!("Failed to write resource to {path:?}: {error}");
+ let _ = fs::remove_file(&tmp);
+ }
</file context>
|
|
||
| #[node_macro::node(category(""))] | ||
| pub async fn resource<'a: 'n>(_: impl Ctx, _primary: (), #[scope("editor-api")] editor_api: &'a PlatformEditorApi, hash: ResourceHash) -> Resource { | ||
| let application_io = editor_api.application_io.as_ref().expect("ApplicationIo must be available when using resources"); |
There was a problem hiding this comment.
P1: Don't panic when the resource backend or hash is missing; this node can crash the whole render path on ordinary load failures.
(Based on your team's feedback about avoiding panics in application code.)
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node-graph/nodes/gstd/src/platform_application_io.rs, line 265:
<comment>Don't panic when the resource backend or hash is missing; this node can crash the whole render path on ordinary load failures.
(Based on your team's feedback about avoiding panics in application code.) </comment>
<file context>
@@ -254,3 +259,34 @@ where
+
+#[node_macro::node(category(""))]
+pub async fn resource<'a: 'n>(_: impl Ctx, _primary: (), #[scope("editor-api")] editor_api: &'a PlatformEditorApi, hash: ResourceHash) -> Resource {
+ let application_io = editor_api.application_io.as_ref().expect("ApplicationIo must be available when using resources");
+ application_io.load_resource(hash).await.unwrap_or_else(|| {
+ panic!("Resource {hash} not found");
</file context>
| document.network_interface.replace_implementation(node_id, network_path, &mut node_template); | ||
| // Upgrade `image` nodes that stored image data as `Image<Color>`` to `image` nodes that store a reference to the image data as a resource. | ||
| // Encodes the image data as PNG and stores it in the document's embedded resources and rewires the node to reference that resource. | ||
| if reference == DefinitionIdentifier::ProtoNode(graphene_std::platform_application_io::image::IDENTIFIER) && inputs_count == 2 { |
There was a problem hiding this comment.
P1: Handle the legacy 1-input image node shape here too; after the alias remap, older embedded-image documents skip this migration and keep an incompatible input layout.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At editor/src/messages/portfolio/document_migration.rs, line 1594:
<comment>Handle the legacy 1-input image node shape here too; after the alias remap, older embedded-image documents skip this migration and keep an incompatible input layout.</comment>
<file context>
@@ -1588,12 +1589,24 @@ fn migrate_node(node_id: &NodeId, node: &DocumentNode, network_path: &[NodeId],
- document.network_interface.replace_implementation(node_id, network_path, &mut node_template);
+ // Upgrade `image` nodes that stored image data as `Image<Color>`` to `image` nodes that store a reference to the image data as a resource.
+ // Encodes the image data as PNG and stores it in the document's embedded resources and rewires the node to reference that resource.
+ if reference == DefinitionIdentifier::ProtoNode(graphene_std::platform_application_io::image::IDENTIFIER) && inputs_count == 2 {
+ let image = node.inputs.iter().find_map(|input| match input.as_value()? {
+ TaggedValue::ImageData(image) => Some(image.clone()),
</file context>
|
|
||
| let path = self.path_for(hash); | ||
| let mmap = Self::open_mmap(&path)?; | ||
| let resource = Resource::new(MmappedBytes(mmap)); |
There was a problem hiding this comment.
P2: Verify the mapped bytes still hash to the requested ResourceHash before returning them.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node-graph/graph-craft/src/application_io/resource/mmap.rs, line 56:
<comment>Verify the mapped bytes still hash to the requested `ResourceHash` before returning them.</comment>
<file context>
@@ -0,0 +1,160 @@
+
+ let path = self.path_for(hash);
+ let mmap = Self::open_mmap(&path)?;
+ let resource = Resource::new(MmappedBytes(mmap));
+
+ self.cache.write().unwrap_or_else(|poisoned| poisoned.into_inner()).insert(*hash, resource.clone());
</file context>
| async fn load_resource<'a: 'n>(_: impl Ctx, _primary: (), #[scope("editor-api")] _editor: &'a PlatformEditorApi, #[name("URL")] url: String) -> Arc<[u8]> { | ||
| let placeholder = || -> Arc<[u8]> { Arc::from(Vec::<u8>::new()) }; | ||
|
|
||
| let response = match reqwest::Client::new().get(&url).send().await { |
There was a problem hiding this comment.
P2: This now treats every resource as an HTTP URL, so local asset paths documented for this node will always fail.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node-graph/nodes/gstd/src/platform_application_io.rs, line 142:
<comment>This now treats every resource as an HTTP URL, so local asset paths documented for this node will always fail.</comment>
<file context>
@@ -137,26 +136,32 @@ fn image_to_bytes(_: impl Ctx, image: List<Raster<CPU>>) -> List<u8> {
+async fn load_resource<'a: 'n>(_: impl Ctx, _primary: (), #[scope("editor-api")] _editor: &'a PlatformEditorApi, #[name("URL")] url: String) -> Arc<[u8]> {
+ let placeholder = || -> Arc<[u8]> { Arc::from(Vec::<u8>::new()) };
+
+ let response = match reqwest::Client::new().get(&url).send().await {
+ Ok(response) => response,
+ Err(error) => {
</file context>
| let Ok(data) = data.await else { | ||
| return Arc::from(include_bytes!("../../../graph-craft/src/null.png").to_vec()); | ||
| async fn load_resource<'a: 'n>(_: impl Ctx, _primary: (), #[scope("editor-api")] _editor: &'a PlatformEditorApi, #[name("URL")] url: String) -> Arc<[u8]> { | ||
| let placeholder = || -> Arc<[u8]> { Arc::from(Vec::<u8>::new()) }; |
There was a problem hiding this comment.
P2: Use an actual transparent image placeholder here; empty bytes make failed loads disappear instead of preserving the transparent fallback promised by this node.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node-graph/nodes/gstd/src/platform_application_io.rs, line 140:
<comment>Use an actual transparent image placeholder here; empty bytes make failed loads disappear instead of preserving the transparent fallback promised by this node.</comment>
<file context>
@@ -137,26 +136,32 @@ fn image_to_bytes(_: impl Ctx, image: List<Raster<CPU>>) -> List<u8> {
- let Ok(data) = data.await else {
- return Arc::from(include_bytes!("../../../graph-craft/src/null.png").to_vec());
+async fn load_resource<'a: 'n>(_: impl Ctx, _primary: (), #[scope("editor-api")] _editor: &'a PlatformEditorApi, #[name("URL")] url: String) -> Arc<[u8]> {
+ let placeholder = || -> Arc<[u8]> { Arc::from(Vec::<u8>::new()) };
+
+ let response = match reqwest::Client::new().get(&url).send().await {
</file context>
| let placeholder = || -> Arc<[u8]> { Arc::from(Vec::<u8>::new()) }; | |
| let placeholder = || -> Arc<[u8]> { Arc::from(include_bytes!("../../../graph-craft/src/null.png").to_vec()) }; |
| @@ -0,0 +1,47 @@ | |||
| #[cfg(target_family = "wasm")] | |||
There was a problem hiding this comment.
P2: This Mutex is unnecessary for the in-memory store; it only adds contention and panic-on-poison paths to code that could use a plain HashMap.
(Based on your team's feedback about reusing existing abstractions instead of adding mutexes.)
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node-graph/graph-craft/src/application_io/resource.rs, line 12:
<comment>This `Mutex` is unnecessary for the in-memory store; it only adds contention and panic-on-poison paths to code that could use a plain `HashMap`.
(Based on your team's feedback about reusing existing abstractions instead of adding mutexes.) </comment>
<file context>
@@ -0,0 +1,47 @@
+
+#[derive(Debug, Default)]
+pub struct HashMapResourceStorage {
+ resources: Mutex<HashMap<ResourceHash, Resource>>,
+}
+
</file context>
|
|
||
| impl Editor { | ||
| pub fn new(environment: Environment, uuid_random_seed: u64) -> Self { | ||
| pub fn new(environment: Environment, uuid_random_seed: u64, resource_storage: Box<dyn ResourceStorage>) -> Self { |
There was a problem hiding this comment.
P2: Editor::new() now returns a partially initialized editor and depends on a separate replace_application_io() call before graph execution is safe.
(Based on your team's feedback about placing integrity-enforcing initialization where the state is created.)
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At editor/src/application.rs, line 12:
<comment>`Editor::new()` now returns a partially initialized editor and depends on a separate `replace_application_io()` call before graph execution is safe.
(Based on your team's feedback about placing integrity-enforcing initialization where the state is created.) </comment>
<file context>
@@ -8,23 +9,29 @@ pub struct Editor {
impl Editor {
- pub fn new(environment: Environment, uuid_random_seed: u64) -> Self {
+ pub fn new(environment: Environment, uuid_random_seed: u64, resource_storage: Box<dyn ResourceStorage>) -> Self {
ENVIRONMENT.set(environment).expect("Editor shoud only be initialized once");
graphene_std::uuid::set_uuid_seed(uuid_random_seed);
</file context>
No description provided.