Skip to content

Storage GC, acceptable-use enforcement, and data residency #49

@jeremymanning

Description

@jeremymanning

Description

Per spec T130-T131 and T144a:

  1. Storage cap and GC (T144a): per-donor storage cap enforcement and content GC for expired/withdrawn data
  2. Acceptable-use filter (T130): refuse unauthorized scanning, malware, illegal content, surveillance, credential cracking at job submission
  3. Shard residency enforcement (T131): per-donor shard-category allowlist enforcement in data plane placement

Requirements

  • Storage GC: track per-donor storage usage, evict expired/orphaned data, enforce configurable cap
  • Acceptable-use: content classification at submission time, reject prohibited workloads
  • Shard residency: enforce data residency constraints (EU, US, UK, JP resident data placed only on nodes in matching jurisdiction)
  • Geographic shard placement: ≥3 continents, ≤2 shards/country per the erasure coding spec

Success Criteria

  • Per-donor storage usage tracked and capped
  • Expired data garbage collected automatically
  • Prohibited workload classes rejected at submission
  • Data residency constraints enforced in shard placement
  • Geographic diversity maintained in erasure-coded placement
  • Integration tests for each enforcement mechanism
  • cargo test passes

Testing (Principle V)

  • Fill donor storage to cap → verify new data rejected/old data evicted
  • Submit malware-class workload → verify rejected
  • Submit EU-resident data → verify shards placed only on EU nodes
  • Verify geographic diversity: remove nodes → verify placement adjusts

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions