From 1b720034611867fd5c7c9bdcfb145a86f45d0299 Mon Sep 17 00:00:00 2001 From: "cortex-ai-agents[bot]" <279748396+cortex-ai-agents[bot]@users.noreply.github.com> Date: Mon, 11 May 2026 07:29:49 +0000 Subject: [PATCH] docs: update for per-AZ ProjectQuota, FlavorGroupCapacity, and usage reconciler MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reflect that ProjectQuota is now one CRD per (project × AZ) pair with naming convention quota-{projectID}-{az}, document the usage reconciler that pre-computes AssignedInstances and UsedResources into CommittedResource status, and add FlavorGroupCapacity and ProjectQuota to the apis.md CRD overview. Co-Authored-By: Claude Opus 4.7 --- docs/apis.md | 20 +++++++++++++++++++ .../committed-resource-reservations.md | 10 ++++++---- 2 files changed, 26 insertions(+), 4 deletions(-) diff --git a/docs/apis.md b/docs/apis.md index d3a2d9416..9160f8816 100644 --- a/docs/apis.md +++ b/docs/apis.md @@ -20,10 +20,14 @@ graph LR; decision(Decision CRD) reservation(Reservation CRD) committedresource(CommittedResource CRD) + projectquota(ProjectQuota CRD) + flavorgroupcapacity(FlavorGroupCapacity CRD) pipeline --> descheduling pipeline --> decision pipeline --> reservation committedresource --> reservation + committedresource --> projectquota + flavorgroupcapacity --> committedresource end prometheus(Prometheus) @@ -114,6 +118,22 @@ The status tracks the accepted amount, usage information (assigned VMs and used For more details on how committed resources interact with reservations, see [committed resource reservations](reservations/committed-resource-reservations.md). +### FlavorGroupCapacity + +```bash +kubectl get flavorgroupcapacities +``` + +FlavorGroupCapacity caches pre-computed capacity data for one flavor group in one availability zone. One CRD exists per (flavor group × AZ) pair, maintained by the capacity controller on a fixed interval. The spec identifies the flavor group and AZ; the status holds per-flavor slot counts (`PlaceableVMs`, `PlaceableHosts`, `TotalCapacityVMSlots`, `TotalCapacityHosts`), aggregate fields (`CommittedCapacity`, `TotalCapacity`, `TotalInstances`), and a `LastReconcileAt` timestamp. The capacity API reads these CRDs instead of probing the scheduler on each request. + +### ProjectQuota + +```bash +kubectl get projectquotas +``` + +ProjectQuota stores quota allocations and computed usage for one project in one availability zone. One CRD exists per (project × AZ) pair, named `quota-{projectID}-{az}`. The spec holds the project identity, availability zone, and a flat `Quota map[string]int64` mapping LIQUID resource names to their per-AZ quota value. The status holds `TotalUsage` and `PaygUsage` as flat `map[string]int64` fields tracking per-resource consumption in that AZ, maintained by the quota controller through periodic full reconciles, incremental Hypervisor diffs, and PaygUsage-only recomputes triggered by CommittedResource status changes. + ### Deschedulings ```bash diff --git a/docs/reservations/committed-resource-reservations.md b/docs/reservations/committed-resource-reservations.md index 4a8a783ae..4d96d43a6 100644 --- a/docs/reservations/committed-resource-reservations.md +++ b/docs/reservations/committed-resource-reservations.md @@ -80,11 +80,11 @@ flowchart LR ### State (CRDs) -**`CommittedResource` CRD** — primary source of truth for a commitment accepted by Cortex. One CRD per commitment UUID. Spec holds the commitment identity (project, flavor group, resource type, amount, ...). Status holds the acceptance outcome (`Ready` condition with reason `Planned`/`Reserving`/`Rejected`/`Accepted`) and the accepted amount. +**`CommittedResource` CRD** — primary source of truth for a commitment accepted by Cortex. One CRD per commitment UUID. Spec holds the commitment identity (project, flavor group, resource type, amount, ...). Status holds the acceptance outcome (`Ready` condition with reason `Planned`/`Reserving`/`Rejected`/`Accepted`), the accepted amount, and usage fields populated by the usage reconciler: `AssignedInstances` (VM UUIDs deterministically assigned to this CR), `UsedResources` (total resource consumption of assigned VMs), `LastUsageReconcileAt`, and `UsageObservedGeneration`. **`Reservation` CRD** — a single reservation slot on a hypervisor, owned by a `CommittedResource`. One `CommittedResource` may drive multiple `Reservation` CRDs (one per flavor-sized slot). Only memory commitments create Reservation CRDs; cores commitments do not. See [./failover-reservations.md](./failover-reservations.md) for the failover reservation type. -**`ProjectQuota` CRD** — per-project quota store. Written by the Quota API when Limes pushes quota; read by the Report-Usage endpoint to include per-AZ quota in usage responses. One CRD per project. +**`ProjectQuota` CRD** — per-project, per-AZ quota store. One CRD exists per (project × availability zone) pair, named `quota-{projectID}-{az}`. Written by the Quota API when Limes pushes quota (one CRD is created for each AZ in the request). The quota controller reconciles usage into the status: `TotalUsage` and `PaygUsage` are flat `map[string]int64` fields tracking per-resource consumption in that AZ. The controller watches CommittedResource and Hypervisor CRDs to maintain these values via periodic full reconciles, incremental HV diffs, and PaygUsage-only recomputes triggered by CommittedResource status changes. **`FlavorGroupCapacity` CRD** — per-flavor-group, per-AZ capacity snapshot maintained by the capacity controller (outside this subsystem). The Report-Capacity endpoint reads these to compute available capacity. @@ -296,7 +296,7 @@ The `CommittedResource` controller handles all downstream work. `AllowRejection= ### Quota API -`PUT /commitments/v1/projects/:project_id/quota` — receives the project's quota allocation from Limes and persists it as a `ProjectQuota` CRD (one per project). For flavor groups with `HandlesCommitments=true`, Limes sends per-AZ quota breakdowns; these are stored in the CRD and read back by the Report-Usage endpoint to include per-AZ quota in usage reports. Writes are idempotent; concurrent writes are resolved with retry-on-conflict. +`PUT /commitments/v1/projects/:project_id/quota` — receives the project's quota allocation from Limes and persists it as `ProjectQuota` CRDs, one per (project × availability zone) combination, named `quota-{projectID}-{az}`. For flavor groups with `HandlesCommitments=true`, Limes sends per-AZ quota breakdowns; each AZ gets its own CRD with a flat `Quota map[string]int64` holding per-resource quota values for that zone. The quota controller then reconciles usage into each CRD's status (`TotalUsage`, `PaygUsage`). Writes are idempotent; concurrent writes are resolved with retry-on-conflict. ### Report-Usage API @@ -307,7 +307,9 @@ For each flavor group `X` that accepts commitments, Cortex exposes three resourc - `hw_version_X_cores` — CPU cores (`HandlesCommitments=false`; derived from RAM via fixed ratio where applicable) - `hw_version_X_instances` — instance count (`HandlesCommitments=false`) -For flavor groups with `HandlesCommitments=true`, the response includes per-AZ quota from the `ProjectQuota` CRD (written by the Quota API). +For flavor groups with `HandlesCommitments=true`, the response includes per-AZ quota from the `ProjectQuota` CRDs (written by the Quota API). + +VM-to-commitment assignment is read from pre-computed `CommittedResource.Status` fields rather than being calculated inline at request time. A dedicated **usage reconciler** (in `internal/scheduling/reservations/commitments/usage_reconciler.go`) watches `CommittedResource` and `Hypervisor` CRDs and periodically runs the deterministic assignment algorithm, writing `AssignedInstances`, `UsedResources`, `LastUsageReconcileAt`, and `UsageObservedGeneration` into each CommittedResource's status. The Report-Usage endpoint reads these status fields to determine which VMs belong to which commitment. If a CR has not yet been reconciled, its VMs appear as PAYG until the first usage reconcile completes. For each VM, the API reports whether it accounts to a specific commitment or PAYG. This assignment is deterministic and may differ from the actual Cortex internal assignment used for scheduling.