DEVOP-579: add NetworkPolicy egress rollout plan (doc only)

[srt0422]

Summary

Adds tickets/devop-579-network-policy-rollout.md — a staged plan for rolling default-deny-egress NetworkPolicies across our 13 clusters.

Why doc-first

NetworkPolicy egress hardening is a 3-engineer-week project where the bulk of effort is discovery, not deployment. default-deny-egress silently breaks every workload that has an un-enumerated outbound dependency, so rushing it is production-impacting. Capturing the plan now (Phases 0-4, rollback procedure, dependencies on DEVOP-588/589) means subsequent loop runs or human owners can pick up execution without redoing the planning.

This PR adds only the plan document. No NetworkPolicy is deployed.

Test plan

• Document compiles, internal links resolve.
• Reviewer: confirm CNI assumption (Calico/Cilium across all 13 clusters?) and adjust Phase 0 if wrong.
• Reviewer: confirm priority ordering of namespaces in Phase 1 matches actual blast-radius.

Related

• Linear: https://linear.app/alloralabs/issue/DEVOP-579
• Hard dep: DEVOP-589 (Harbor proxy-cache) must land before Phase 2.
• Soft dep: DEVOP-588 (Kyverno on all clusters) for Phase 4.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

---

Summary by cubic

Adds a phased plan to roll out default-deny-egress and parallel default-deny-ingress Kubernetes NetworkPolicies across all 13 clusters per DEVOP-579; documentation only (tickets/devop-579-network-policy-rollout.md), no policies are deployed. The plan includes 48-hour soak windows with a clean-soak gate, a suspect‑egress checklist (webhook/pastebin/ngrok/cloud‑metadata/dynamic‑DNS) for review, a documented rollback command, and SECURITY-RUNBOOK.md updates.

• Dependencies
  • Hard: DEVOP-589 (Harbor proxy-cache) must land before Phase 2.
  • Soft: DEVOP-588 (Kyverno on all clusters) for Phase 4 enforcement.

^{Written for commit 7b93a18. Summary will update on new commits. Review in cubic}

[cubic-dev-ai]

4 issues found across 1 file

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tickets/devop-579-network-policy-rollout.md">

<violation number="1" location="tickets/devop-579-network-policy-rollout.md:33">
P2: The discovery checklist omits the DEVOP-579 requirement to explicitly flag suspect egress destinations (webhook/pastebin/ngrok/169.254.169.254).</violation>

<violation number="2" location="tickets/devop-579-network-policy-rollout.md:52">
P1: Phase 3 uses 24-hour soak windows, but linked Linear issue DEVOP-579 specifies 48-hour soaks for staged rollout.</violation>

<violation number="3" location="tickets/devop-579-network-policy-rollout.md:64">
P2: Phase 4 is missing the DEVOP-579 requirement to document the rollout/policies in SECURITY-RUNBOOK.md.</violation>

<violation number="4" location="tickets/devop-579-network-policy-rollout.md:74">
P2: This plan marks ingress NetworkPolicies as out of scope, but linked Linear issue DEVOP-579 requires default-deny for both egress and ingress.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant K8s as Kubernetes Clusters (13)
    participant CNI as CNI Plugin (Calico/Cilium)
    participant Hubble as Hubble/Flow Logs
    participant Harbor as Harbor Registry
    participant Kyverno as Kyverno Policy Engine
    participant Runbook as Rollback Runbook

    Note over K8s,Runbook: Phase 0 — Pre-flight Assessment
    K8s->>CNI: Confirm NetworkPolicy support
    alt CNI supports NetworkPolicy
        CNI-->>K8s: Calico, Cilium, or Antrea confirmed
    else Flannel without --network-policy
        CNI-->>K8s: Need to migrate CNI first
    end
    K8s->>Hubble: Enable flow logs on staging cluster
    Note over Hubble: Capture 7 days baseline traffic

    Note over K8s,Rollback: Phase 1 — Discovery (per namespace)
    loop For each namespace in priority order
        K8s->>Hubble: Query egress flow logs (7 days)
        Hubble-->>K8s: Destination CIDRs, DNS, ports
        K8s->>K8s: Categorize traffic (internal/infra/vendor/registries/customer)
        K8s->>K8s: Document in network-policies/discovery/<namespace>.md
    end

    Note over K8s,Rollback: Phase 2 — Allowlist Authoring
    K8s->>K8s: Create default-deny.yaml (deny all egress except DNS)
    K8s->>K8s: Create allowlist.yaml (derived from Phase 1)
    Note over K8s: DNS to kube-dns/coredns (53/udp, 53/tcp)
    Note over K8s: NTP always allowed (123/udp)
    Note over K8s: Cluster-internal pod-to-pod allowed by default
    K8s->>Harbor: Dependency on DEVOP-589 (Harbor proxy-cache)
    alt DEVOP-589 landed
        Note over K8s: Allowlists reference Harbor proxy instead of direct registries
    else Not yet landed
        Note over K8s: Allowlists must allow direct ghcr.io, docker.io, etc.
    end

    Note over K8s,Rollback: Phase 3 — Staged Rollout
    K8s->>K8s: Day 1: Apply to 1 staging namespace, observe 24h
    K8s->>K8s: Day 2: Apply to all staging namespaces, observe 24h
    K8s->>K8s: Day 3: Apply to 1 production namespace (lowest risk), observe 24h
    K8s->>K8s: Days 4-5: Roll forward remaining namespaces (lowest blast-radius first)
    alt Egress broken for workload
        K8s->>Runbook: kubectl delete networkpolicy default-deny -n <ns>
        Runbook-->>K8s: Egress restored immediately
    end

    Note over K8s,Rollback: Phase 4 — Steady State
    alt DEVOP-588 landed (Kyverno on all clusters)
        Kyverno->>K8s: Auto-flag new namespaces without default-deny
        K8s->>K8s: Monthly review of discovery documents
    else Not yet landed
        Note over K8s: Manual enforcement only
    end

Loading

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.}