Replace boolean feature flags with three-mode FeatureMode in placement shim

docs/architecture.md

@@ -36,15 +36,35 @@ Cortex receives the list of possible hosts and their weights from Nova. It then
 
 As part of the [CobaltCore](https://cobaltcore-dev.github.io/docs/) stack, we provide a Placement-like API shim, which translates requests from Nova and Neutron to the [Hypervisor CRD](https://github.com/cobaltcore-dev/openstack-hypervisor-operator) based on the KVM stack provided by [IronCore](https://ironcore.dev/), [Gardener](https://gardener.cloud/) and [Garden Linux](https://gardenlinux.io/). This means, instead of managing resource inventories in Placement's database, the Hypervisor CRD is used to track resource allocations and hypervisor capabilities.
 
-### Feature Flags
+### Feature Modes
 
-Each major capability of the shim is gated behind a feature flag in the Helm configuration. When a flag is disabled, the corresponding endpoints fall back to forwarding requests to upstream Placement unchanged. This allows operators to adopt CRD-backed behavior incrementally.
+Each endpoint group of the shim is controlled by a **feature mode** in the Helm configuration (`features.<endpoint>`). There are three modes:
 
-| Flag | Endpoints affected | Behavior when enabled |
-|---|---|---|
-| `features.enableResourceProviders` | `/resource_providers` and sub-resources | Serve KVM resource providers from Hypervisor CRDs; merge with upstream for non-KVM providers |
-| `features.enableRoot` | `GET /` | Return a static version discovery document from config instead of forwarding to upstream |
-| `features.enableTraits` | `/traits` | Serve traits from local ConfigMaps instead of upstream Placement |
+| Mode | Description |
+|---|---|
+| `passthrough` | Forward all requests to upstream Placement without any shim logic. This is the default for every endpoint when unset. |
+| `hybrid` | Combine upstream Placement with local CRD data. Upstream must be available; the shim keeps CRD state in sync to prepare for cutover. |
+| `crd` | Serve requests exclusively from the Hypervisor CRD and local Kubernetes resources. No upstream Placement dependency is required. |
+
+The following endpoint groups each have their own mode field:
+
+| Helm key | Endpoints affected |
+|---|---|
+| `features.resourceProviders` | `/resource_providers` and sub-resources |
+| `features.root` | `GET /` |
+| `features.traits` | `/traits` |
+| `features.resourceProviderTraits` | `/resource_providers/{uuid}/traits` |
+| `features.resourceClasses` | `/resource_classes` |
+| `features.inventories` | `/resource_providers/{uuid}/inventories` |
+| `features.aggregates` | `/resource_providers/{uuid}/aggregates` |
+| `features.allocations` | `/allocations` |
+| `features.usages` | `/usages` |
+| `features.allocationCandidates` | `/allocation_candidates` |
+| `features.reshaper` | `/reshaper` |
+
+This per-endpoint granularity allows operators to adopt CRD-backed behavior incrementally, migrating one endpoint group at a time from `passthrough` through `hybrid` to `crd`.
+
+Endpoint groups that have not yet implemented `hybrid` or `crd` logic return **501 Not Implemented** when set to those modes.
 
 ### Passthrough
 
@@ -76,21 +96,31 @@ Upstream connectivity is optional at startup: if the upstream Placement API is u
 
 ### CRD-Backed Resource Providers
 
-When `features.enableResourceProviders` is enabled, the shim serves KVM resource providers directly from Kubernetes Hypervisor CRDs rather than forwarding to upstream Placement. This is the core architectural shift: KVM hypervisor inventory lives in Kubernetes instead of in Placement's database.
+When `features.resourceProviders` is set to `hybrid` or `crd`, the shim serves KVM resource providers directly from Kubernetes Hypervisor CRDs rather than forwarding to upstream Placement. This is the core architectural shift: KVM hypervisor inventory lives in Kubernetes instead of in Placement's database.
 
 The shim supports the full CRUD surface for resource providers:
 
-- **GET /resource_providers**: Lists resource providers by merging KVM hypervisors from Kubernetes with non-KVM providers from upstream Placement. The merge is based on UUID: if a hypervisor CRD exists with the same OpenStack ID as an upstream provider, the CRD-backed version takes precedence.
-- **GET /resource_providers/{uuid}**: Looks up the UUID against indexed Hypervisor CRDs first. If found, returns the translated provider; otherwise, forwards to upstream.
-- **POST /resource_providers**: Checks the requested name and UUID against existing Hypervisor CRDs. Returns `409 Conflict` if the name or UUID collides with a KVM hypervisor, preventing shadow providers from being created in upstream Placement. If no collision, the request is forwarded to upstream.
-- **PUT /resource_providers/{uuid}**: Same collision detection as POST. Updates that would rename a KVM-managed provider are rejected with `409 Conflict`.
-- **DELETE /resource_providers/{uuid}**: Prevents deletion of CRD-backed KVM providers by returning `409 Conflict`. Non-KVM providers are forwarded to upstream.
+- **GET /resource_providers**: In `hybrid` mode, lists resource providers by merging KVM hypervisors from Kubernetes with non-KVM providers from upstream Placement. The merge is based on UUID: if a hypervisor CRD exists with the same OpenStack ID as an upstream provider, the CRD-backed version takes precedence. In `crd` mode, lists only from Kubernetes without contacting upstream.
+- **GET /resource_providers/{uuid}**: Looks up the UUID against indexed Hypervisor CRDs first. If found, returns the translated provider. In `hybrid` mode, if not found, forwards to upstream; in `crd` mode, returns 404.
+- **POST /resource_providers**: Checks the requested name and UUID against existing Hypervisor CRDs. Returns `409 Conflict` if the name or UUID collides with a KVM hypervisor, preventing shadow providers from being created in upstream Placement. In `hybrid` mode, if no collision, the request is forwarded to upstream; in `crd` mode, non-KVM providers are rejected with 404.
+- **PUT /resource_providers/{uuid}**: Same collision detection as POST. Updates that would rename a KVM-managed provider are rejected with `409 Conflict`. Non-KVM providers are forwarded to upstream in `hybrid` mode or rejected with 404 in `crd` mode.
+- **DELETE /resource_providers/{uuid}**: Prevents deletion of CRD-backed KVM providers by returning `409 Conflict`. Non-KVM providers are forwarded to upstream in `hybrid` mode or rejected with 404 in `crd` mode.
 
 For efficient lookups, the shim indexes Hypervisor CRDs on three fields: `status.hypervisorId` (the OpenStack UUID), `metadata.uid` (the Kubernetes UID), and `metadata.name`. These indexes are registered at startup via the multicluster client, enabling O(1) lookups by any of these keys.
 
+### Root Endpoint
+
+The `GET /` endpoint returns a version discovery document. The behavior depends on the mode set in `features.root`:
+
+- **passthrough**: Forwards to upstream Placement as-is.
+- **hybrid**: Fetches the version document from upstream and computes the **version intersection** with the local static configuration. The result uses the higher minimum version and the lower maximum version, yielding the narrowest compatible window. If the ranges don't overlap, the local config is returned as-is.
+- **crd**: Returns the static version discovery document from the `versioning` config section without contacting upstream.
+
+Both `hybrid` and `crd` modes require a `versioning` config block with `id`, `minVersion`, `maxVersion`, and `status`.
+
 ### Traits
 
-When `features.enableTraits` is enabled, the shim serves OpenStack Placement traits from a pair of Kubernetes ConfigMaps instead of forwarding to upstream:
+When `features.traits` is set to `hybrid` or `crd`, the shim serves OpenStack Placement traits from a pair of Kubernetes ConfigMaps instead of forwarding to upstream:
 
 - **Static ConfigMap** (Helm-managed): Contains the standard OpenStack traits deployed via Helm. Its name is set by `traits.configMapName` in the shim config.
 - **Custom ConfigMap** (shim-managed): Stores `CUSTOM_*` traits created at runtime through PUT requests. Named `{configMapName}-custom`.
@@ -103,6 +133,8 @@ The trait endpoints support the full OpenStack Placement traits API:
 
 Writes to the custom ConfigMap are serialized across replicas using a Kubernetes Lease-backed distributed lock (see `pkg/resourcelock`). This prevents concurrent writes from corrupting the ConfigMap data.
 
+In **hybrid** mode, `GET`, `PUT`, and `DELETE` trait requests are forwarded to upstream Placement (so upstream always has the latest data), and a **periodic sync loop** runs in the background (every 60 seconds with jitter) to fetch traits from upstream and write them into the static ConfigMap. This keeps the local view in sync with upstream and prepares for cutover to `crd` mode. In **crd** mode, traits are served exclusively from the local ConfigMaps with no upstream dependency.
+
 ### Authentication
 
 The shim includes an optional Keystone token validation middleware, configured via the `auth` section in the Helm values. When enabled, every incoming request is checked against a policy table before reaching the handler.

helm/bundles/cortex-placement-shim/templates/configmap-traits.yaml

@@ -1,4 +1,4 @@
-{{- if (index .Values "cortex-shim").conf.features.enableTraits }}
+{{- if ne ((index .Values "cortex-shim").conf.features.traits | default "passthrough") "passthrough" }}
 {{- $cmName := (index .Values "cortex-shim").conf.traits.configMapName }}
 apiVersion: v1
 kind: ConfigMap

helm/bundles/cortex-placement-shim/values.yaml

@@ -42,9 +42,17 @@ cortex-shim:
     osUserDomainName:
     osProjectDomainName:
     features:
-      enableResourceProviders: false
-      enableRoot: false
-      enableTraits: false
+      resourceProviders: passthrough
+      root: passthrough
+      traits: passthrough
+      resourceProviderTraits: passthrough
+      resourceClasses: passthrough
+      inventories: passthrough
+      aggregates: passthrough
+      allocations: passthrough
+      usages: passthrough
+      allocationCandidates: passthrough
+      reshaper: passthrough
     # The shim will return this as a static version discovery document for
     # GET / instead of forwarding to upstream placement.
     versioning:

internal/shim/placement/handle_allocation_candidates.go

@@ -33,5 +33,5 @@ import (
 // inventory capacity and usage for informed decision-making. Available since
 // microversion 1.10.
 func (s *Shim) HandleListAllocationCandidates(w http.ResponseWriter, r *http.Request) {
-	s.forward(w, r)
+	s.dispatchPassthroughOnly(w, r, s.config.Features.AllocationCandidates)
 }

internal/shim/placement/handle_allocation_candidates_test.go

@@ -19,3 +19,43 @@ func TestHandleListAllocationCandidates(t *testing.T) {
 		t.Fatalf("upstream path = %q, want /allocation_candidates", gotPath)
 	}
 }
+
+func TestHandleAllocationCandidates_HybridMode(t *testing.T) {
+	down, up := newTestTimers()
+	s := &Shim{
+		config: config{
+			PlacementURL: "http://should-not-be-called:1234",
+			Features:     featuresConfig{AllocationCandidates: FeatureModeHybrid},
+		},
+		maxBodyLogSize:         4096,
+		downstreamRequestTimer: down,
+		upstreamRequestTimer:   up,
+	}
+	t.Run("GET returns 501", func(t *testing.T) {
+		w := serveHandler(t, "GET", "/allocation_candidates",
+			s.HandleListAllocationCandidates, "/allocation_candidates")
+		if w.Code != http.StatusNotImplemented {
+			t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented)
+		}
+	})
+}
+
+func TestHandleAllocationCandidates_CRDMode(t *testing.T) {
+	down, up := newTestTimers()
+	s := &Shim{
+		config: config{
+			PlacementURL: "http://should-not-be-called:1234",
+			Features:     featuresConfig{AllocationCandidates: FeatureModeCRD},
+		},
+		maxBodyLogSize:         4096,
+		downstreamRequestTimer: down,
+		upstreamRequestTimer:   up,
+	}
+	t.Run("GET returns 501", func(t *testing.T) {
+		w := serveHandler(t, "GET", "/allocation_candidates",
+			s.HandleListAllocationCandidates, "/allocation_candidates")
+		if w.Code != http.StatusNotImplemented {
+			t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented)
+		}
+	})
+}

internal/shim/placement/handle_allocations.go

@@ -23,7 +23,7 @@ import (
 // success, or 409 Conflict if inventory is insufficient or a concurrent
 // update is detected (error code: placement.concurrent_update).
 func (s *Shim) HandleManageAllocations(w http.ResponseWriter, r *http.Request) {
-	s.forward(w, r)
+	s.dispatchPassthroughOnly(w, r, s.config.Features.Allocations)
 }
 
 // HandleListAllocations handles GET /allocations/{consumer_uuid} requests.
@@ -41,7 +41,7 @@ func (s *Shim) HandleListAllocations(w http.ResponseWriter, r *http.Request) {
 	if _, ok := requiredUUIDPathParam(w, r, "consumer_uuid"); !ok {
 		return
 	}
-	s.forward(w, r)
+	s.dispatchPassthroughOnly(w, r, s.config.Features.Allocations)
 }
 
 // HandleUpdateAllocations handles PUT /allocations/{consumer_uuid} requests.
@@ -59,7 +59,7 @@ func (s *Shim) HandleUpdateAllocations(w http.ResponseWriter, r *http.Request) {
 	if _, ok := requiredUUIDPathParam(w, r, "consumer_uuid"); !ok {
 		return
 	}
-	s.forward(w, r)
+	s.dispatchPassthroughOnly(w, r, s.config.Features.Allocations)
 }
 
 // HandleDeleteAllocations handles DELETE /allocations/{consumer_uuid} requests.
@@ -71,5 +71,5 @@ func (s *Shim) HandleDeleteAllocations(w http.ResponseWriter, r *http.Request) {
 	if _, ok := requiredUUIDPathParam(w, r, "consumer_uuid"); !ok {
 		return
 	}
-	s.forward(w, r)
+	s.dispatchPassthroughOnly(w, r, s.config.Features.Allocations)
 }

internal/shim/placement/handle_allocations_test.go

@@ -76,3 +76,85 @@ func TestHandleDeleteAllocations(t *testing.T) {
 		}
 	})
 }
+
+func TestHandleAllocations_HybridMode(t *testing.T) {
+	down, up := newTestTimers()
+	s := &Shim{
+		config: config{
+			PlacementURL: "http://should-not-be-called:1234",
+			Features:     featuresConfig{Allocations: FeatureModeHybrid},
+		},
+		maxBodyLogSize:         4096,
+		downstreamRequestTimer: down,
+		upstreamRequestTimer:   up,
+	}
+	t.Run("POST returns 501", func(t *testing.T) {
+		w := serveHandler(t, "POST", "/allocations",
+			s.HandleManageAllocations, "/allocations")
+		if w.Code != http.StatusNotImplemented {
+			t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented)
+		}
+	})
+	t.Run("GET returns 501", func(t *testing.T) {
+		w := serveHandler(t, "GET", "/allocations/{consumer_uuid}",
+			s.HandleListAllocations, "/allocations/"+validUUID)
+		if w.Code != http.StatusNotImplemented {
+			t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented)
+		}
+	})
+	t.Run("PUT returns 501", func(t *testing.T) {
+		w := serveHandler(t, "PUT", "/allocations/{consumer_uuid}",
+			s.HandleUpdateAllocations, "/allocations/"+validUUID)
+		if w.Code != http.StatusNotImplemented {
+			t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented)
+		}
+	})
+	t.Run("DELETE returns 501", func(t *testing.T) {
+		w := serveHandler(t, "DELETE", "/allocations/{consumer_uuid}",
+			s.HandleDeleteAllocations, "/allocations/"+validUUID)
+		if w.Code != http.StatusNotImplemented {
+			t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented)
+		}
+	})
+}
+
+func TestHandleAllocations_CRDMode(t *testing.T) {
+	down, up := newTestTimers()
+	s := &Shim{
+		config: config{
+			PlacementURL: "http://should-not-be-called:1234",
+			Features:     featuresConfig{Allocations: FeatureModeCRD},
+		},
+		maxBodyLogSize:         4096,
+		downstreamRequestTimer: down,
+		upstreamRequestTimer:   up,
+	}
+	t.Run("POST returns 501", func(t *testing.T) {
+		w := serveHandler(t, "POST", "/allocations",
+			s.HandleManageAllocations, "/allocations")
+		if w.Code != http.StatusNotImplemented {
+			t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented)
+		}
+	})
+	t.Run("GET returns 501", func(t *testing.T) {
+		w := serveHandler(t, "GET", "/allocations/{consumer_uuid}",
+			s.HandleListAllocations, "/allocations/"+validUUID)
+		if w.Code != http.StatusNotImplemented {
+			t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented)
+		}
+	})
+	t.Run("PUT returns 501", func(t *testing.T) {
+		w := serveHandler(t, "PUT", "/allocations/{consumer_uuid}",
+			s.HandleUpdateAllocations, "/allocations/"+validUUID)
+		if w.Code != http.StatusNotImplemented {
+			t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented)
+		}
+	})
+	t.Run("DELETE returns 501", func(t *testing.T) {
+		w := serveHandler(t, "DELETE", "/allocations/{consumer_uuid}",
+			s.HandleDeleteAllocations, "/allocations/"+validUUID)
+		if w.Code != http.StatusNotImplemented {
+			t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented)
+		}
+	})
+}

internal/shim/placement/handle_reshaper.go

@@ -23,5 +23,5 @@ import (
 // resource provider does not exist or if inventory/allocation constraints
 // would be violated. Available since microversion 1.30.
 func (s *Shim) HandlePostReshaper(w http.ResponseWriter, r *http.Request) {
-	s.forward(w, r)
+	s.dispatchPassthroughOnly(w, r, s.config.Features.Reshaper)
 }

internal/shim/placement/handle_reshaper_test.go

@@ -19,3 +19,43 @@ func TestHandlePostReshaper(t *testing.T) {
 		t.Fatalf("upstream path = %q, want /reshaper", gotPath)
 	}
 }
+
+func TestHandleReshaper_HybridMode(t *testing.T) {
+	down, up := newTestTimers()
+	s := &Shim{
+		config: config{
+			PlacementURL: "http://should-not-be-called:1234",
+			Features:     featuresConfig{Reshaper: FeatureModeHybrid},
+		},
+		maxBodyLogSize:         4096,
+		downstreamRequestTimer: down,
+		upstreamRequestTimer:   up,
+	}
+	t.Run("POST returns 501", func(t *testing.T) {
+		w := serveHandler(t, "POST", "/reshaper",
+			s.HandlePostReshaper, "/reshaper")
+		if w.Code != http.StatusNotImplemented {
+			t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented)
+		}
+	})
+}
+
+func TestHandleReshaper_CRDMode(t *testing.T) {
+	down, up := newTestTimers()
+	s := &Shim{
+		config: config{
+			PlacementURL: "http://should-not-be-called:1234",
+			Features:     featuresConfig{Reshaper: FeatureModeCRD},
+		},
+		maxBodyLogSize:         4096,
+		downstreamRequestTimer: down,
+		upstreamRequestTimer:   up,
+	}
+	t.Run("POST returns 501", func(t *testing.T) {
+		w := serveHandler(t, "POST", "/reshaper",
+			s.HandlePostReshaper, "/reshaper")
+		if w.Code != http.StatusNotImplemented {
+			t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented)
+		}
+	})
+}

internal/shim/placement/handle_resource_classes.go

@@ -15,7 +15,7 @@ import (
 // categorize the types of resources that resource providers can offer as
 // inventory. Available since microversion 1.2.
 func (s *Shim) HandleListResourceClasses(w http.ResponseWriter, r *http.Request) {
-	s.forward(w, r)
+	s.dispatchPassthroughOnly(w, r, s.config.Features.ResourceClasses)
 }
 
 // HandleCreateResourceClass handles POST /resource_classes requests.
@@ -26,7 +26,7 @@ func (s *Shim) HandleListResourceClasses(w http.ResponseWriter, r *http.Request)
 // is missing, and 409 Conflict if a class with the same name already exists.
 // Available since microversion 1.2.
 func (s *Shim) HandleCreateResourceClass(w http.ResponseWriter, r *http.Request) {
-	s.forward(w, r)
+	s.dispatchPassthroughOnly(w, r, s.config.Features.ResourceClasses)
 }
 
 // HandleShowResourceClass handles GET /resource_classes/{name} requests.
@@ -38,7 +38,7 @@ func (s *Shim) HandleShowResourceClass(w http.ResponseWriter, r *http.Request) {
 	if _, ok := requiredPathParam(w, r, "name"); !ok {
 		return
 	}
-	s.forward(w, r)
+	s.dispatchPassthroughOnly(w, r, s.config.Features.ResourceClasses)
 }
 
 // HandleUpdateResourceClass handles PUT /resource_classes/{name} requests.
@@ -53,7 +53,7 @@ func (s *Shim) HandleUpdateResourceClass(w http.ResponseWriter, r *http.Request)
 	if _, ok := requiredPathParam(w, r, "name"); !ok {
 		return
 	}
-	s.forward(w, r)
+	s.dispatchPassthroughOnly(w, r, s.config.Features.ResourceClasses)
 }
 
 // HandleDeleteResourceClass handles DELETE /resource_classes/{name} requests.
@@ -67,5 +67,5 @@ func (s *Shim) HandleDeleteResourceClass(w http.ResponseWriter, r *http.Request)
 	if _, ok := requiredPathParam(w, r, "name"); !ok {
 		return
 	}
-	s.forward(w, r)
+	s.dispatchPassthroughOnly(w, r, s.config.Features.ResourceClasses)
 }