Add s3-graphs-zip workflow for zipping CE graphs#999
Add s3-graphs-zip workflow for zipping CE graphs#999
Conversation
rsaksida
commented
Mar 6, 2026
rsaksida
commented
- Add s3-graphs-zip workflow for archiving CE graphs
- Hook up legacy CER API to Argo workflows (WIP)
- Add script to package CE graph JSON files into zip files in S3 - Streams zip files directly to S3 with multipart upload support - Calls preconfigured webhook for notifications - Add Dockerfile for Argo / container orchestration - Add docker-compose.yml with LocalStack for integration test setup - Add workflow template
db/migrate/20260306120000_add_argo_workflow_fields_to_envelope_downloads.rb
Show resolved
Hide resolved
|
Go ahead and deploy it in Sandbox. Then let @JWaltuch or @mparsons-ce know to test it. Then go to production. :) |
| resources: | ||
| requests: | ||
| cpu: "1000m" | ||
| memory: "2Gi" |
There was a problem hiding this comment.
With the streaming ZIP operation, why does it need 2GB minimum memory minimum? I would think, at most the memory consumption would be the size of the ZIP metadata header. That is about 50MB for 500K files. So it should not be 2GB minimum. Reason this is concerning is because we are allocating 2GB up front, that means Kubernetes is going to assume that it needs to auto-scale more VMs to server other apps. I recommend reducing this number to 200MB or lower to see where it breaks. Then increase it by 100MB increments until it doesn't break.
Also, add a # comment here indicating the reason for the low memory allocation.
Same question for the CPU. 1000m is probably too high for this workflow.
Note that these workflows are very light weight. So we should keep to the minimum required memory allocations to reduce costs as we start to run unrelated workflows in parallel.
2 participants
