-
Notifications
You must be signed in to change notification settings - Fork 632
Optimization for prover (override #1774) #1783
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
noel2004
wants to merge
32
commits into
develop
Choose a base branch
from
feat/zkvm_prover_143
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+736
−771
Open
Changes from all commits
Commits
Show all changes
32 commits
Select commit
Hold shift + click to select a range
bdf7826
rollback dep of zkvm-prover
noel2004 9839bf7
prover quit while proving process has panicked
noel2004 d262a63
Merge remote-tracking branch 'origin/develop' into feat/prover_4.7
noel2004 dc29c8c
Merge remote-tracking branch 'origin/develop' into feat/prover_4.7
noel2004 af63bc0
fix wrong configuration
noel2004 5c2803c
Merge remote-tracking branch 'origin/develop' into feat/prover_4.7
noel2004 5ae31bc
Merge remote-tracking branch 'origin/develop' into feat/prover_4.7
noel2004 98be0a0
Merge remote-tracking branch 'origin/develop' into feat/prover_4.7
noel2004 b244fa8
Merge remote-tracking branch 'origin/develop' into feat/prover_4.7
noel2004 e852915
update zkvm-prover
noel2004 b833468
add debug mode, trivial fixings
noel2004 930a12a
update zkvm-prover dep and openvm to 1.4.2
noel2004 3b174f8
add test data for mainnet galileo and prune feynman (can not be teste…
noel2004 74a3d7a
udpate zkvm-prover dep
noel2004 ede29c7
init dumper
noel2004 b270d96
dumper
noel2004 79d79ed
fix
noel2004 d306b38
fix
noel2004 03b992f
json mode
noel2004 492563f
Merge remote-tracking branch 'origin/develop' into feat/zkvm_prover_142
noel2004 6a57a2e
update dep of zkvm prover and trivial fixing
noel2004 788f6b9
chore: update da codec to v0.10.1 (#1789)
yiweichi 9b2b5e0
feat: adjustments for launching Cloak with GalileoV2 (#1793)
Thegaram 8955d4f
Merge branch 'feat/zkvm_prover_143' into develop
noel2004 58297d3
fix e2e tool
noel2004 b76539b
update test env
noel2004 93f755c
update dep
noel2004 252eea9
use galileov2 for cloak test enviroment
noel2004 a35834d
fix make file in e2e test
noel2004 e1aa90b
update ZKVM_VERSION
noel2004 f2a9ba5
upgrade dep
noel2004 649c441
pump crates version
noel2004 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| use async_trait::async_trait; | ||
| use libzkp::ProvingTaskExt; | ||
| use scroll_proving_sdk::prover::{ | ||
| proving_service::{ | ||
| GetVkRequest, GetVkResponse, ProveRequest, ProveResponse, QueryTaskRequest, | ||
| QueryTaskResponse, TaskStatus, | ||
| }, | ||
| ProvingService, | ||
| }; | ||
| use scroll_zkvm_types::ProvingTask; | ||
|
|
||
| #[derive(Default)] | ||
| pub struct Dumper { | ||
| #[allow(dead_code)] | ||
| pub target_path: String, | ||
| pub json_mode: bool, | ||
| } | ||
|
|
||
| impl Dumper { | ||
| fn dump(&self, input_string: &str) -> eyre::Result<()> { | ||
| let task: ProvingTaskExt = serde_json::from_str(input_string)?; | ||
| let task = ProvingTask::from(task); | ||
|
|
||
| if self.json_mode { | ||
| let file = std::fs::File::create("input_task.json")?; | ||
| serde_json::to_writer(std::io::BufWriter::new(file), &task)?; | ||
| } else { | ||
| // stream-encode serialized_witness to input_task.bin using bincode 2.0 | ||
| let input_file = std::fs::File::create("input_task.bin")?; | ||
| let mut input_writer = std::io::BufWriter::new(input_file); | ||
| bincode::encode_into_std_write( | ||
| &task.serialized_witness, | ||
| &mut input_writer, | ||
| bincode::config::standard(), | ||
| )?; | ||
|
|
||
| // stream-encode aggregated_proofs to agg_proofs.bin using bincode 2.0 | ||
| let agg_file = std::fs::File::create("agg_proofs.bin")?; | ||
| let mut agg_writer = std::io::BufWriter::new(agg_file); | ||
| for proof in &task.aggregated_proofs { | ||
| let sz = bincode::serde::encode_into_std_write( | ||
| &proof.proofs, | ||
| &mut agg_writer, | ||
| bincode::config::standard(), | ||
| )?; | ||
| println!("written {sz} bytes for proof"); | ||
| } | ||
| } | ||
|
|
||
| Ok(()) | ||
| } | ||
| } | ||
|
|
||
| #[async_trait] | ||
| impl ProvingService for Dumper { | ||
| fn is_local(&self) -> bool { | ||
| true | ||
| } | ||
| async fn get_vks(&self, _: GetVkRequest) -> GetVkResponse { | ||
| // get vk has been deprecated in new prover with dynamic asset loading scheme | ||
| GetVkResponse { | ||
| vks: vec![], | ||
| error: None, | ||
| } | ||
| } | ||
| async fn prove(&mut self, req: ProveRequest) -> ProveResponse { | ||
| let error = if let Err(e) = self.dump(&req.input) { | ||
| Some(format!("failed to dump: {}", e)) | ||
| } else { | ||
| None | ||
| }; | ||
|
|
||
| ProveResponse { | ||
| status: TaskStatus::Failed, | ||
| error, | ||
| ..Default::default() | ||
| } | ||
| } | ||
|
|
||
| async fn query_task(&mut self, req: QueryTaskRequest) -> QueryTaskResponse { | ||
| QueryTaskResponse { | ||
| task_id: req.task_id, | ||
| status: TaskStatus::Failed, | ||
| error: Some("dump file finished but need a fail return to exit".to_string()), | ||
| ..Default::default() | ||
| } | ||
| } | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Honor
target_pathwhen writing dump files.dump()always writes to the current working directory, so a CLI-specifiedtarget_pathis ignored. This makes the dump output location incorrect.🐛 Proposed fix
fn dump(&self, input_string: &str) -> eyre::Result<()> { let task: ProvingTaskExt = serde_json::from_str(input_string)?; let task = ProvingTask::from(task); + let base_dir = if self.target_path.is_empty() { + std::path::Path::new(".") + } else { + std::path::Path::new(&self.target_path) + }; + std::fs::create_dir_all(base_dir)?; + if self.json_mode { - let file = std::fs::File::create("input_task.json")?; + let file = std::fs::File::create(base_dir.join("input_task.json"))?; serde_json::to_writer(std::io::BufWriter::new(file), &task)?; } else { // stream-encode serialized_witness to input_task.bin using bincode 2.0 - let input_file = std::fs::File::create("input_task.bin")?; + let input_file = std::fs::File::create(base_dir.join("input_task.bin"))?; let mut input_writer = std::io::BufWriter::new(input_file); bincode::encode_into_std_write( &task.serialized_witness, &mut input_writer, bincode::config::standard(), )?; // stream-encode aggregated_proofs to agg_proofs.bin using bincode 2.0 - let agg_file = std::fs::File::create("agg_proofs.bin")?; + let agg_file = std::fs::File::create(base_dir.join("agg_proofs.bin"))?; let mut agg_writer = std::io::BufWriter::new(agg_file); for proof in &task.aggregated_proofs { let sz = bincode::serde::encode_into_std_write( &proof.proofs, &mut agg_writer, bincode::config::standard(), )?; println!("written {sz} bytes for proof"); } }Also applies to: 24-39
🤖 Prompt for AI Agents