testing: WAN soak + load + chaos test harness

[TheNewAutonomy]

Goal

Prove stability/robustness under realistic WAN conditions before mainnet.

Scope

• Soak test: multi-day run with churn (restarts), variable latency/loss
• Load test: sustained tx rate + receipt polling + indexer traffic
• Chaos scenarios:
  • partial partitions (A↔B↔C), short disconnects
  • slow node, disk pressure, CPU pressure
• Report/metrics collection and pass/fail thresholds

Acceptance criteria

• Automated harness can reproduce scenarios and output a summarized report
• Clear go/no-go thresholds exist for mainnet readiness

Refs: #201 (threat model/adversarial plan), #202 (fuzz/property tests), #200/#204 (networking/sync).

[TheNewAutonomy]

Added an initial WAN soak/chaos harness under scripts/wan_chaos/.

What it provides:

• 3-node testnet in Linux network namespaces (ip netns) connected via a bridge
• WAN-like conditions via tc netem (latency/jitter/loss)
• Chaos events: optional partition (iptables) and optional node restart
• Polls catalyst-cli status via RPC and writes a run directory with logs + a simple report.txt

Run:

• sudo bash scripts/wan_chaos/run.sh

Key env vars:

• DURATION_SECS, LOSS_PCT, LATENCY_MS, JITTER_MS
• PARTITION_AT_SECS, PARTITION_DURATION_SECS
• RESTART_AT_SECS, RESTART_NODE

Suggested initial go/no-go thresholds (to refine):

• applied_cycle continues advancing on >=2/3 nodes during non-partition windows
• after a 30s partition, all 3 nodes resume advancing within 60s of heal
• no unbounded growth in P2P caches (bounded by configured limits)

Next steps:

• Extend report to parse applied_cycle explicitly and compute “stalled seconds” rather than the current heuristic.
• Add sustained tx load (txgen + receipt polling) and indexer-like RPC traffic.

[TheNewAutonomy]

Updated scripts/wan_chaos/run.sh to produce objective liveness metrics:

• parses applied_cycle: per node
• reports stalled_seconds.* and longest_stall_seconds.*
• records partition_time_to_resume_seconds after heal (when enabled)
• records restart_time_to_resume_seconds after restart (when enabled)

This replaces the earlier “status line unchanged” heuristic.