diff --git a/docs/architecture.md b/docs/architecture.md index 49a0928d9..0e91385c2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -36,15 +36,35 @@ Cortex receives the list of possible hosts and their weights from Nova. It then As part of the [CobaltCore](https://cobaltcore-dev.github.io/docs/) stack, we provide a Placement-like API shim, which translates requests from Nova and Neutron to the [Hypervisor CRD](https://github.com/cobaltcore-dev/openstack-hypervisor-operator) based on the KVM stack provided by [IronCore](https://ironcore.dev/), [Gardener](https://gardener.cloud/) and [Garden Linux](https://gardenlinux.io/). This means, instead of managing resource inventories in Placement's database, the Hypervisor CRD is used to track resource allocations and hypervisor capabilities. -### Feature Flags +### Feature Modes -Each major capability of the shim is gated behind a feature flag in the Helm configuration. When a flag is disabled, the corresponding endpoints fall back to forwarding requests to upstream Placement unchanged. This allows operators to adopt CRD-backed behavior incrementally. +Each endpoint group of the shim is controlled by a **feature mode** in the Helm configuration (`features.`). There are three modes: -| Flag | Endpoints affected | Behavior when enabled | -|---|---|---| -| `features.enableResourceProviders` | `/resource_providers` and sub-resources | Serve KVM resource providers from Hypervisor CRDs; merge with upstream for non-KVM providers | -| `features.enableRoot` | `GET /` | Return a static version discovery document from config instead of forwarding to upstream | -| `features.enableTraits` | `/traits` | Serve traits from local ConfigMaps instead of upstream Placement | +| Mode | Description | +|---|---| +| `passthrough` | Forward all requests to upstream Placement without any shim logic. This is the default for every endpoint when unset. | +| `hybrid` | Combine upstream Placement with local CRD data. Upstream must be available; the shim keeps CRD state in sync to prepare for cutover. | +| `crd` | Serve requests exclusively from the Hypervisor CRD and local Kubernetes resources. No upstream Placement dependency is required. | + +The following endpoint groups each have their own mode field: + +| Helm key | Endpoints affected | +|---|---| +| `features.resourceProviders` | `/resource_providers` and sub-resources | +| `features.root` | `GET /` | +| `features.traits` | `/traits` | +| `features.resourceProviderTraits` | `/resource_providers/{uuid}/traits` | +| `features.resourceClasses` | `/resource_classes` | +| `features.inventories` | `/resource_providers/{uuid}/inventories` | +| `features.aggregates` | `/resource_providers/{uuid}/aggregates` | +| `features.allocations` | `/allocations` | +| `features.usages` | `/usages` | +| `features.allocationCandidates` | `/allocation_candidates` | +| `features.reshaper` | `/reshaper` | + +This per-endpoint granularity allows operators to adopt CRD-backed behavior incrementally, migrating one endpoint group at a time from `passthrough` through `hybrid` to `crd`. + +Endpoint groups that have not yet implemented `hybrid` or `crd` logic return **501 Not Implemented** when set to those modes. ### Passthrough @@ -76,21 +96,31 @@ Upstream connectivity is optional at startup: if the upstream Placement API is u ### CRD-Backed Resource Providers -When `features.enableResourceProviders` is enabled, the shim serves KVM resource providers directly from Kubernetes Hypervisor CRDs rather than forwarding to upstream Placement. This is the core architectural shift: KVM hypervisor inventory lives in Kubernetes instead of in Placement's database. +When `features.resourceProviders` is set to `hybrid` or `crd`, the shim serves KVM resource providers directly from Kubernetes Hypervisor CRDs rather than forwarding to upstream Placement. This is the core architectural shift: KVM hypervisor inventory lives in Kubernetes instead of in Placement's database. The shim supports the full CRUD surface for resource providers: -- **GET /resource_providers**: Lists resource providers by merging KVM hypervisors from Kubernetes with non-KVM providers from upstream Placement. The merge is based on UUID: if a hypervisor CRD exists with the same OpenStack ID as an upstream provider, the CRD-backed version takes precedence. -- **GET /resource_providers/{uuid}**: Looks up the UUID against indexed Hypervisor CRDs first. If found, returns the translated provider; otherwise, forwards to upstream. -- **POST /resource_providers**: Checks the requested name and UUID against existing Hypervisor CRDs. Returns `409 Conflict` if the name or UUID collides with a KVM hypervisor, preventing shadow providers from being created in upstream Placement. If no collision, the request is forwarded to upstream. -- **PUT /resource_providers/{uuid}**: Same collision detection as POST. Updates that would rename a KVM-managed provider are rejected with `409 Conflict`. -- **DELETE /resource_providers/{uuid}**: Prevents deletion of CRD-backed KVM providers by returning `409 Conflict`. Non-KVM providers are forwarded to upstream. +- **GET /resource_providers**: In `hybrid` mode, lists resource providers by merging KVM hypervisors from Kubernetes with non-KVM providers from upstream Placement. The merge is based on UUID: if a hypervisor CRD exists with the same OpenStack ID as an upstream provider, the CRD-backed version takes precedence. In `crd` mode, lists only from Kubernetes without contacting upstream. +- **GET /resource_providers/{uuid}**: Looks up the UUID against indexed Hypervisor CRDs first. If found, returns the translated provider. In `hybrid` mode, if not found, forwards to upstream; in `crd` mode, returns 404. +- **POST /resource_providers**: Checks the requested name and UUID against existing Hypervisor CRDs. Returns `409 Conflict` if the name or UUID collides with a KVM hypervisor, preventing shadow providers from being created in upstream Placement. In `hybrid` mode, if no collision, the request is forwarded to upstream; in `crd` mode, non-KVM providers are rejected with 404. +- **PUT /resource_providers/{uuid}**: Same collision detection as POST. Updates that would rename a KVM-managed provider are rejected with `409 Conflict`. Non-KVM providers are forwarded to upstream in `hybrid` mode or rejected with 404 in `crd` mode. +- **DELETE /resource_providers/{uuid}**: Prevents deletion of CRD-backed KVM providers by returning `409 Conflict`. Non-KVM providers are forwarded to upstream in `hybrid` mode or rejected with 404 in `crd` mode. For efficient lookups, the shim indexes Hypervisor CRDs on three fields: `status.hypervisorId` (the OpenStack UUID), `metadata.uid` (the Kubernetes UID), and `metadata.name`. These indexes are registered at startup via the multicluster client, enabling O(1) lookups by any of these keys. +### Root Endpoint + +The `GET /` endpoint returns a version discovery document. The behavior depends on the mode set in `features.root`: + +- **passthrough**: Forwards to upstream Placement as-is. +- **hybrid**: Fetches the version document from upstream and computes the **version intersection** with the local static configuration. The result uses the higher minimum version and the lower maximum version, yielding the narrowest compatible window. If the ranges don't overlap, the local config is returned as-is. +- **crd**: Returns the static version discovery document from the `versioning` config section without contacting upstream. + +Both `hybrid` and `crd` modes require a `versioning` config block with `id`, `minVersion`, `maxVersion`, and `status`. + ### Traits -When `features.enableTraits` is enabled, the shim serves OpenStack Placement traits from a pair of Kubernetes ConfigMaps instead of forwarding to upstream: +When `features.traits` is set to `hybrid` or `crd`, the shim serves OpenStack Placement traits from a pair of Kubernetes ConfigMaps instead of forwarding to upstream: - **Static ConfigMap** (Helm-managed): Contains the standard OpenStack traits deployed via Helm. Its name is set by `traits.configMapName` in the shim config. - **Custom ConfigMap** (shim-managed): Stores `CUSTOM_*` traits created at runtime through PUT requests. Named `{configMapName}-custom`. @@ -103,6 +133,8 @@ The trait endpoints support the full OpenStack Placement traits API: Writes to the custom ConfigMap are serialized across replicas using a Kubernetes Lease-backed distributed lock (see `pkg/resourcelock`). This prevents concurrent writes from corrupting the ConfigMap data. +In **hybrid** mode, `GET`, `PUT`, and `DELETE` trait requests are forwarded to upstream Placement (so upstream always has the latest data), and a **periodic sync loop** runs in the background (every 60 seconds with jitter) to fetch traits from upstream and write them into the static ConfigMap. This keeps the local view in sync with upstream and prepares for cutover to `crd` mode. In **crd** mode, traits are served exclusively from the local ConfigMaps with no upstream dependency. + ### Authentication The shim includes an optional Keystone token validation middleware, configured via the `auth` section in the Helm values. When enabled, every incoming request is checked against a policy table before reaching the handler. diff --git a/helm/bundles/cortex-placement-shim/templates/configmap-traits.yaml b/helm/bundles/cortex-placement-shim/templates/configmap-traits.yaml index f44ff37c4..b6969aaa7 100644 --- a/helm/bundles/cortex-placement-shim/templates/configmap-traits.yaml +++ b/helm/bundles/cortex-placement-shim/templates/configmap-traits.yaml @@ -1,4 +1,4 @@ -{{- if (index .Values "cortex-shim").conf.features.enableTraits }} +{{- if ne ((index .Values "cortex-shim").conf.features.traits | default "passthrough") "passthrough" }} {{- $cmName := (index .Values "cortex-shim").conf.traits.configMapName }} apiVersion: v1 kind: ConfigMap diff --git a/helm/bundles/cortex-placement-shim/values.yaml b/helm/bundles/cortex-placement-shim/values.yaml index cb36edf94..7e1818e9e 100644 --- a/helm/bundles/cortex-placement-shim/values.yaml +++ b/helm/bundles/cortex-placement-shim/values.yaml @@ -42,9 +42,17 @@ cortex-shim: osUserDomainName: osProjectDomainName: features: - enableResourceProviders: false - enableRoot: false - enableTraits: false + resourceProviders: passthrough + root: passthrough + traits: passthrough + resourceProviderTraits: passthrough + resourceClasses: passthrough + inventories: passthrough + aggregates: passthrough + allocations: passthrough + usages: passthrough + allocationCandidates: passthrough + reshaper: passthrough # The shim will return this as a static version discovery document for # GET / instead of forwarding to upstream placement. versioning: diff --git a/internal/shim/placement/handle_allocation_candidates.go b/internal/shim/placement/handle_allocation_candidates.go index 2ead71c17..575a60cce 100644 --- a/internal/shim/placement/handle_allocation_candidates.go +++ b/internal/shim/placement/handle_allocation_candidates.go @@ -33,5 +33,5 @@ import ( // inventory capacity and usage for informed decision-making. Available since // microversion 1.10. func (s *Shim) HandleListAllocationCandidates(w http.ResponseWriter, r *http.Request) { - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.AllocationCandidates) } diff --git a/internal/shim/placement/handle_allocation_candidates_test.go b/internal/shim/placement/handle_allocation_candidates_test.go index de75a96af..561411c04 100644 --- a/internal/shim/placement/handle_allocation_candidates_test.go +++ b/internal/shim/placement/handle_allocation_candidates_test.go @@ -19,3 +19,43 @@ func TestHandleListAllocationCandidates(t *testing.T) { t.Fatalf("upstream path = %q, want /allocation_candidates", gotPath) } } + +func TestHandleAllocationCandidates_HybridMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{AllocationCandidates: FeatureModeHybrid}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("GET returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/allocation_candidates", + s.HandleListAllocationCandidates, "/allocation_candidates") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} + +func TestHandleAllocationCandidates_CRDMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{AllocationCandidates: FeatureModeCRD}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("GET returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/allocation_candidates", + s.HandleListAllocationCandidates, "/allocation_candidates") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} diff --git a/internal/shim/placement/handle_allocations.go b/internal/shim/placement/handle_allocations.go index a6524fc4b..f909cc5a7 100644 --- a/internal/shim/placement/handle_allocations.go +++ b/internal/shim/placement/handle_allocations.go @@ -23,7 +23,7 @@ import ( // success, or 409 Conflict if inventory is insufficient or a concurrent // update is detected (error code: placement.concurrent_update). func (s *Shim) HandleManageAllocations(w http.ResponseWriter, r *http.Request) { - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.Allocations) } // HandleListAllocations handles GET /allocations/{consumer_uuid} requests. @@ -41,7 +41,7 @@ func (s *Shim) HandleListAllocations(w http.ResponseWriter, r *http.Request) { if _, ok := requiredUUIDPathParam(w, r, "consumer_uuid"); !ok { return } - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.Allocations) } // HandleUpdateAllocations handles PUT /allocations/{consumer_uuid} requests. @@ -59,7 +59,7 @@ func (s *Shim) HandleUpdateAllocations(w http.ResponseWriter, r *http.Request) { if _, ok := requiredUUIDPathParam(w, r, "consumer_uuid"); !ok { return } - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.Allocations) } // HandleDeleteAllocations handles DELETE /allocations/{consumer_uuid} requests. @@ -71,5 +71,5 @@ func (s *Shim) HandleDeleteAllocations(w http.ResponseWriter, r *http.Request) { if _, ok := requiredUUIDPathParam(w, r, "consumer_uuid"); !ok { return } - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.Allocations) } diff --git a/internal/shim/placement/handle_allocations_test.go b/internal/shim/placement/handle_allocations_test.go index c42cf86e0..e39d5c43a 100644 --- a/internal/shim/placement/handle_allocations_test.go +++ b/internal/shim/placement/handle_allocations_test.go @@ -76,3 +76,85 @@ func TestHandleDeleteAllocations(t *testing.T) { } }) } + +func TestHandleAllocations_HybridMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{Allocations: FeatureModeHybrid}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("POST returns 501", func(t *testing.T) { + w := serveHandler(t, "POST", "/allocations", + s.HandleManageAllocations, "/allocations") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("GET returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/allocations/{consumer_uuid}", + s.HandleListAllocations, "/allocations/"+validUUID) + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("PUT returns 501", func(t *testing.T) { + w := serveHandler(t, "PUT", "/allocations/{consumer_uuid}", + s.HandleUpdateAllocations, "/allocations/"+validUUID) + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("DELETE returns 501", func(t *testing.T) { + w := serveHandler(t, "DELETE", "/allocations/{consumer_uuid}", + s.HandleDeleteAllocations, "/allocations/"+validUUID) + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} + +func TestHandleAllocations_CRDMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{Allocations: FeatureModeCRD}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("POST returns 501", func(t *testing.T) { + w := serveHandler(t, "POST", "/allocations", + s.HandleManageAllocations, "/allocations") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("GET returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/allocations/{consumer_uuid}", + s.HandleListAllocations, "/allocations/"+validUUID) + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("PUT returns 501", func(t *testing.T) { + w := serveHandler(t, "PUT", "/allocations/{consumer_uuid}", + s.HandleUpdateAllocations, "/allocations/"+validUUID) + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("DELETE returns 501", func(t *testing.T) { + w := serveHandler(t, "DELETE", "/allocations/{consumer_uuid}", + s.HandleDeleteAllocations, "/allocations/"+validUUID) + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} diff --git a/internal/shim/placement/handle_reshaper.go b/internal/shim/placement/handle_reshaper.go index 9e02e004a..47df4d72b 100644 --- a/internal/shim/placement/handle_reshaper.go +++ b/internal/shim/placement/handle_reshaper.go @@ -23,5 +23,5 @@ import ( // resource provider does not exist or if inventory/allocation constraints // would be violated. Available since microversion 1.30. func (s *Shim) HandlePostReshaper(w http.ResponseWriter, r *http.Request) { - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.Reshaper) } diff --git a/internal/shim/placement/handle_reshaper_test.go b/internal/shim/placement/handle_reshaper_test.go index e00eff2e2..7cb83e48f 100644 --- a/internal/shim/placement/handle_reshaper_test.go +++ b/internal/shim/placement/handle_reshaper_test.go @@ -19,3 +19,43 @@ func TestHandlePostReshaper(t *testing.T) { t.Fatalf("upstream path = %q, want /reshaper", gotPath) } } + +func TestHandleReshaper_HybridMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{Reshaper: FeatureModeHybrid}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("POST returns 501", func(t *testing.T) { + w := serveHandler(t, "POST", "/reshaper", + s.HandlePostReshaper, "/reshaper") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} + +func TestHandleReshaper_CRDMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{Reshaper: FeatureModeCRD}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("POST returns 501", func(t *testing.T) { + w := serveHandler(t, "POST", "/reshaper", + s.HandlePostReshaper, "/reshaper") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} diff --git a/internal/shim/placement/handle_resource_classes.go b/internal/shim/placement/handle_resource_classes.go index 480bb0232..9067079fd 100644 --- a/internal/shim/placement/handle_resource_classes.go +++ b/internal/shim/placement/handle_resource_classes.go @@ -15,7 +15,7 @@ import ( // categorize the types of resources that resource providers can offer as // inventory. Available since microversion 1.2. func (s *Shim) HandleListResourceClasses(w http.ResponseWriter, r *http.Request) { - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.ResourceClasses) } // HandleCreateResourceClass handles POST /resource_classes requests. @@ -26,7 +26,7 @@ func (s *Shim) HandleListResourceClasses(w http.ResponseWriter, r *http.Request) // is missing, and 409 Conflict if a class with the same name already exists. // Available since microversion 1.2. func (s *Shim) HandleCreateResourceClass(w http.ResponseWriter, r *http.Request) { - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.ResourceClasses) } // HandleShowResourceClass handles GET /resource_classes/{name} requests. @@ -38,7 +38,7 @@ func (s *Shim) HandleShowResourceClass(w http.ResponseWriter, r *http.Request) { if _, ok := requiredPathParam(w, r, "name"); !ok { return } - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.ResourceClasses) } // HandleUpdateResourceClass handles PUT /resource_classes/{name} requests. @@ -53,7 +53,7 @@ func (s *Shim) HandleUpdateResourceClass(w http.ResponseWriter, r *http.Request) if _, ok := requiredPathParam(w, r, "name"); !ok { return } - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.ResourceClasses) } // HandleDeleteResourceClass handles DELETE /resource_classes/{name} requests. @@ -67,5 +67,5 @@ func (s *Shim) HandleDeleteResourceClass(w http.ResponseWriter, r *http.Request) if _, ok := requiredPathParam(w, r, "name"); !ok { return } - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.ResourceClasses) } diff --git a/internal/shim/placement/handle_resource_classes_test.go b/internal/shim/placement/handle_resource_classes_test.go index 80ffdf40e..330cae9be 100644 --- a/internal/shim/placement/handle_resource_classes_test.go +++ b/internal/shim/placement/handle_resource_classes_test.go @@ -55,3 +55,99 @@ func TestHandleDeleteResourceClass(t *testing.T) { t.Fatalf("status = %d, want %d", w.Code, http.StatusNoContent) } } + +func TestHandleResourceClasses_HybridMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{ResourceClasses: FeatureModeHybrid}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("GET list returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/resource_classes", + s.HandleListResourceClasses, "/resource_classes") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("POST returns 501", func(t *testing.T) { + w := serveHandler(t, "POST", "/resource_classes", + s.HandleCreateResourceClass, "/resource_classes") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("GET show returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/resource_classes/{name}", + s.HandleShowResourceClass, "/resource_classes/VCPU") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("PUT returns 501", func(t *testing.T) { + w := serveHandler(t, "PUT", "/resource_classes/{name}", + s.HandleUpdateResourceClass, "/resource_classes/CUSTOM_FOO") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("DELETE returns 501", func(t *testing.T) { + w := serveHandler(t, "DELETE", "/resource_classes/{name}", + s.HandleDeleteResourceClass, "/resource_classes/CUSTOM_BAR") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} + +func TestHandleResourceClasses_CRDMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{ResourceClasses: FeatureModeCRD}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("GET list returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/resource_classes", + s.HandleListResourceClasses, "/resource_classes") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("POST returns 501", func(t *testing.T) { + w := serveHandler(t, "POST", "/resource_classes", + s.HandleCreateResourceClass, "/resource_classes") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("GET show returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/resource_classes/{name}", + s.HandleShowResourceClass, "/resource_classes/VCPU") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("PUT returns 501", func(t *testing.T) { + w := serveHandler(t, "PUT", "/resource_classes/{name}", + s.HandleUpdateResourceClass, "/resource_classes/CUSTOM_FOO") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("DELETE returns 501", func(t *testing.T) { + w := serveHandler(t, "DELETE", "/resource_classes/{name}", + s.HandleDeleteResourceClass, "/resource_classes/CUSTOM_BAR") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} diff --git a/internal/shim/placement/handle_resource_provider_aggregates.go b/internal/shim/placement/handle_resource_provider_aggregates.go index e78711df0..3d7205193 100644 --- a/internal/shim/placement/handle_resource_provider_aggregates.go +++ b/internal/shim/placement/handle_resource_provider_aggregates.go @@ -24,7 +24,7 @@ func (s *Shim) HandleListResourceProviderAggregates(w http.ResponseWriter, r *ht if _, ok := requiredUUIDPathParam(w, r, "uuid"); !ok { return } - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.Aggregates) } // HandleUpdateResourceProviderAggregates handles @@ -41,5 +41,5 @@ func (s *Shim) HandleUpdateResourceProviderAggregates(w http.ResponseWriter, r * if _, ok := requiredUUIDPathParam(w, r, "uuid"); !ok { return } - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.Aggregates) } diff --git a/internal/shim/placement/handle_resource_provider_aggregates_test.go b/internal/shim/placement/handle_resource_provider_aggregates_test.go index f55b09fed..eb35d665c 100644 --- a/internal/shim/placement/handle_resource_provider_aggregates_test.go +++ b/internal/shim/placement/handle_resource_provider_aggregates_test.go @@ -49,3 +49,61 @@ func TestHandleUpdateResourceProviderAggregates(t *testing.T) { } }) } + +func TestHandleResourceProviderAggregates_HybridMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{Aggregates: FeatureModeHybrid}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("GET returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/resource_providers/{uuid}/aggregates", + s.HandleListResourceProviderAggregates, + "/resource_providers/"+validUUID+"/aggregates") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("PUT returns 501", func(t *testing.T) { + w := serveHandler(t, "PUT", "/resource_providers/{uuid}/aggregates", + s.HandleUpdateResourceProviderAggregates, + "/resource_providers/"+validUUID+"/aggregates") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} + +func TestHandleResourceProviderAggregates_CRDMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{Aggregates: FeatureModeCRD}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("GET returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/resource_providers/{uuid}/aggregates", + s.HandleListResourceProviderAggregates, + "/resource_providers/"+validUUID+"/aggregates") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("PUT returns 501", func(t *testing.T) { + w := serveHandler(t, "PUT", "/resource_providers/{uuid}/aggregates", + s.HandleUpdateResourceProviderAggregates, + "/resource_providers/"+validUUID+"/aggregates") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} diff --git a/internal/shim/placement/handle_resource_provider_allocations.go b/internal/shim/placement/handle_resource_provider_allocations.go index 2c281e384..346362476 100644 --- a/internal/shim/placement/handle_resource_provider_allocations.go +++ b/internal/shim/placement/handle_resource_provider_allocations.go @@ -19,5 +19,5 @@ func (s *Shim) HandleListResourceProviderAllocations(w http.ResponseWriter, r *h if _, ok := requiredUUIDPathParam(w, r, "uuid"); !ok { return } - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.Allocations) } diff --git a/internal/shim/placement/handle_resource_provider_allocations_test.go b/internal/shim/placement/handle_resource_provider_allocations_test.go index 98834afab..c6dced483 100644 --- a/internal/shim/placement/handle_resource_provider_allocations_test.go +++ b/internal/shim/placement/handle_resource_provider_allocations_test.go @@ -28,3 +28,45 @@ func TestHandleListResourceProviderAllocations(t *testing.T) { } }) } + +func TestHandleResourceProviderAllocations_HybridMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{Allocations: FeatureModeHybrid}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("GET returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/resource_providers/{uuid}/allocations", + s.HandleListResourceProviderAllocations, + "/resource_providers/"+validUUID+"/allocations") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} + +func TestHandleResourceProviderAllocations_CRDMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{Allocations: FeatureModeCRD}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("GET returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/resource_providers/{uuid}/allocations", + s.HandleListResourceProviderAllocations, + "/resource_providers/"+validUUID+"/allocations") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} diff --git a/internal/shim/placement/handle_resource_provider_inventories.go b/internal/shim/placement/handle_resource_provider_inventories.go index 5091b47b1..974e11ab0 100644 --- a/internal/shim/placement/handle_resource_provider_inventories.go +++ b/internal/shim/placement/handle_resource_provider_inventories.go @@ -20,7 +20,7 @@ func (s *Shim) HandleListResourceProviderInventories(w http.ResponseWriter, r *h if _, ok := requiredUUIDPathParam(w, r, "uuid"); !ok { return } - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.Inventories) } // HandleUpdateResourceProviderInventories handles @@ -37,7 +37,7 @@ func (s *Shim) HandleUpdateResourceProviderInventories(w http.ResponseWriter, r if _, ok := requiredUUIDPathParam(w, r, "uuid"); !ok { return } - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.Inventories) } // HandleDeleteResourceProviderInventories handles @@ -53,7 +53,7 @@ func (s *Shim) HandleDeleteResourceProviderInventories(w http.ResponseWriter, r if _, ok := requiredUUIDPathParam(w, r, "uuid"); !ok { return } - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.Inventories) } // HandleShowResourceProviderInventory handles @@ -70,7 +70,7 @@ func (s *Shim) HandleShowResourceProviderInventory(w http.ResponseWriter, r *htt if _, ok := requiredPathParam(w, r, "resource_class"); !ok { return } - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.Inventories) } // HandleUpdateResourceProviderInventory handles @@ -89,7 +89,7 @@ func (s *Shim) HandleUpdateResourceProviderInventory(w http.ResponseWriter, r *h if _, ok := requiredPathParam(w, r, "resource_class"); !ok { return } - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.Inventories) } // HandleDeleteResourceProviderInventory handles @@ -106,5 +106,5 @@ func (s *Shim) HandleDeleteResourceProviderInventory(w http.ResponseWriter, r *h if _, ok := requiredPathParam(w, r, "resource_class"); !ok { return } - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.Inventories) } diff --git a/internal/shim/placement/handle_resource_provider_inventories_test.go b/internal/shim/placement/handle_resource_provider_inventories_test.go index 054e48e32..203e31639 100644 --- a/internal/shim/placement/handle_resource_provider_inventories_test.go +++ b/internal/shim/placement/handle_resource_provider_inventories_test.go @@ -137,3 +137,125 @@ func TestHandleDeleteResourceProviderInventory(t *testing.T) { } }) } + +func TestHandleResourceProviderInventories_HybridMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{Inventories: FeatureModeHybrid}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("GET list returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/resource_providers/{uuid}/inventories", + s.HandleListResourceProviderInventories, + "/resource_providers/"+validUUID+"/inventories") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("PUT list returns 501", func(t *testing.T) { + w := serveHandler(t, "PUT", "/resource_providers/{uuid}/inventories", + s.HandleUpdateResourceProviderInventories, + "/resource_providers/"+validUUID+"/inventories") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("DELETE list returns 501", func(t *testing.T) { + w := serveHandler(t, "DELETE", "/resource_providers/{uuid}/inventories", + s.HandleDeleteResourceProviderInventories, + "/resource_providers/"+validUUID+"/inventories") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("GET single returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/resource_providers/{uuid}/inventories/{resource_class}", + s.HandleShowResourceProviderInventory, + "/resource_providers/"+validUUID+"/inventories/VCPU") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("PUT single returns 501", func(t *testing.T) { + w := serveHandler(t, "PUT", "/resource_providers/{uuid}/inventories/{resource_class}", + s.HandleUpdateResourceProviderInventory, + "/resource_providers/"+validUUID+"/inventories/VCPU") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("DELETE single returns 501", func(t *testing.T) { + w := serveHandler(t, "DELETE", "/resource_providers/{uuid}/inventories/{resource_class}", + s.HandleDeleteResourceProviderInventory, + "/resource_providers/"+validUUID+"/inventories/VCPU") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} + +func TestHandleResourceProviderInventories_CRDMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{Inventories: FeatureModeCRD}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("GET list returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/resource_providers/{uuid}/inventories", + s.HandleListResourceProviderInventories, + "/resource_providers/"+validUUID+"/inventories") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("PUT list returns 501", func(t *testing.T) { + w := serveHandler(t, "PUT", "/resource_providers/{uuid}/inventories", + s.HandleUpdateResourceProviderInventories, + "/resource_providers/"+validUUID+"/inventories") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("DELETE list returns 501", func(t *testing.T) { + w := serveHandler(t, "DELETE", "/resource_providers/{uuid}/inventories", + s.HandleDeleteResourceProviderInventories, + "/resource_providers/"+validUUID+"/inventories") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("GET single returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/resource_providers/{uuid}/inventories/{resource_class}", + s.HandleShowResourceProviderInventory, + "/resource_providers/"+validUUID+"/inventories/VCPU") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("PUT single returns 501", func(t *testing.T) { + w := serveHandler(t, "PUT", "/resource_providers/{uuid}/inventories/{resource_class}", + s.HandleUpdateResourceProviderInventory, + "/resource_providers/"+validUUID+"/inventories/VCPU") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("DELETE single returns 501", func(t *testing.T) { + w := serveHandler(t, "DELETE", "/resource_providers/{uuid}/inventories/{resource_class}", + s.HandleDeleteResourceProviderInventory, + "/resource_providers/"+validUUID+"/inventories/VCPU") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} diff --git a/internal/shim/placement/handle_resource_provider_traits.go b/internal/shim/placement/handle_resource_provider_traits.go index db0f5e234..16978a593 100644 --- a/internal/shim/placement/handle_resource_provider_traits.go +++ b/internal/shim/placement/handle_resource_provider_traits.go @@ -4,9 +4,23 @@ package placement import ( + "fmt" "net/http" + + hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "sigs.k8s.io/controller-runtime/pkg/client" + logf "sigs.k8s.io/controller-runtime/pkg/log" ) +// resourceProviderTraitsResponse is the JSON body returned by +// GET /resource_providers/{uuid}/traits and +// PUT /resource_providers/{uuid}/traits. +type resourceProviderTraitsResponse struct { + Traits []string `json:"traits"` + ResourceProviderGeneration int64 `json:"resource_provider_generation"` +} + // HandleListResourceProviderTraits handles // GET /resource_providers/{uuid}/traits requests. // @@ -15,10 +29,53 @@ import ( // resource_provider_generation for concurrency tracking. Returns 404 if the // provider does not exist. func (s *Shim) HandleListResourceProviderTraits(w http.ResponseWriter, r *http.Request) { - if _, ok := requiredUUIDPathParam(w, r, "uuid"); !ok { + uuid, ok := requiredUUIDPathParam(w, r, "uuid") + if !ok { + return + } + switch s.config.Features.ResourceProviderTraits.orDefault() { + case FeatureModePassthrough: + s.forward(w, r) + case FeatureModeHybrid: + s.forward(w, r) + case FeatureModeCRD: + s.listResourceProviderTraitsCRD(w, r, uuid) + default: + http.Error(w, "unknown feature mode", http.StatusInternalServerError) + } +} + +func (s *Shim) listResourceProviderTraitsCRD(w http.ResponseWriter, r *http.Request, uuid string) { + ctx := r.Context() + log := logf.FromContext(ctx) + + var hvs hv1.HypervisorList + err := s.List(ctx, &hvs, client.MatchingFields{idxHypervisorOpenStackId: uuid}) + if apierrors.IsNotFound(err) || len(hvs.Items) == 0 { + log.Info("resource provider not found in kubernetes (crd mode)", "uuid", uuid) + http.Error(w, "resource provider not found", http.StatusNotFound) + return + } + if err != nil { + log.Error(err, "failed to list hypervisors with OpenStack ID index") + http.Error(w, "Internal Server Error", http.StatusInternalServerError) + return + } + if len(hvs.Items) > 1 { + log.Error(nil, "multiple hypervisors found with the same OpenStack ID", "uuid", uuid) + http.Error(w, "Internal Server Error", http.StatusInternalServerError) return } - s.forward(w, r) + + hv := hvs.Items[0] + traits := hv.Status.Traits + if traits == nil { + traits = []string{} + } + s.writeJSON(w, http.StatusOK, resourceProviderTraitsResponse{ + Traits: traits, + ResourceProviderGeneration: hv.Generation, + }) } // HandleUpdateResourceProviderTraits handles @@ -35,7 +92,16 @@ func (s *Shim) HandleUpdateResourceProviderTraits(w http.ResponseWriter, r *http if _, ok := requiredUUIDPathParam(w, r, "uuid"); !ok { return } - s.forward(w, r) + switch s.config.Features.ResourceProviderTraits.orDefault() { + case FeatureModePassthrough: + s.forward(w, r) + case FeatureModeHybrid: + s.forward(w, r) + case FeatureModeCRD: + http.Error(w, fmt.Sprintf("%s mode is not yet implemented for resource provider trait writes", s.config.Features.ResourceProviderTraits), http.StatusNotImplemented) + default: + http.Error(w, "unknown feature mode", http.StatusInternalServerError) + } } // HandleDeleteResourceProviderTraits handles @@ -51,5 +117,14 @@ func (s *Shim) HandleDeleteResourceProviderTraits(w http.ResponseWriter, r *http if _, ok := requiredUUIDPathParam(w, r, "uuid"); !ok { return } - s.forward(w, r) + switch s.config.Features.ResourceProviderTraits.orDefault() { + case FeatureModePassthrough: + s.forward(w, r) + case FeatureModeHybrid: + s.forward(w, r) + case FeatureModeCRD: + http.Error(w, fmt.Sprintf("%s mode is not yet implemented for resource provider trait writes", s.config.Features.ResourceProviderTraits), http.StatusNotImplemented) + default: + http.Error(w, "unknown feature mode", http.StatusInternalServerError) + } } diff --git a/internal/shim/placement/handle_resource_provider_traits_test.go b/internal/shim/placement/handle_resource_provider_traits_test.go index 809f0503f..69ac8cd8c 100644 --- a/internal/shim/placement/handle_resource_provider_traits_test.go +++ b/internal/shim/placement/handle_resource_provider_traits_test.go @@ -4,6 +4,7 @@ package placement import ( + "encoding/json" "net/http" "testing" ) @@ -70,3 +71,83 @@ func TestHandleDeleteResourceProviderTraits(t *testing.T) { } }) } + +func TestHandleResourceProviderTraits_HybridMode(t *testing.T) { + s := newTestShim(t, http.StatusOK, `{"traits":["CUSTOM_HW_FPGA"],"resource_provider_generation":1}`, nil) + s.config.Features.ResourceProviderTraits = FeatureModeHybrid + t.Run("GET forwards to upstream", func(t *testing.T) { + w := serveHandler(t, "GET", "/resource_providers/{uuid}/traits", + s.HandleListResourceProviderTraits, + "/resource_providers/"+validUUID+"/traits") + if w.Code != http.StatusOK { + t.Fatalf("status = %d, want %d", w.Code, http.StatusOK) + } + }) + t.Run("PUT forwards to upstream", func(t *testing.T) { + w := serveHandler(t, "PUT", "/resource_providers/{uuid}/traits", + s.HandleUpdateResourceProviderTraits, + "/resource_providers/"+validUUID+"/traits") + if w.Code != http.StatusOK { + t.Fatalf("status = %d, want %d", w.Code, http.StatusOK) + } + }) + + sDel := newTestShim(t, http.StatusNoContent, "", nil) + sDel.config.Features.ResourceProviderTraits = FeatureModeHybrid + t.Run("DELETE forwards to upstream", func(t *testing.T) { + w := serveHandler(t, "DELETE", "/resource_providers/{uuid}/traits", + sDel.HandleDeleteResourceProviderTraits, + "/resource_providers/"+validUUID+"/traits") + if w.Code != http.StatusNoContent { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNoContent) + } + }) +} + +func TestHandleResourceProviderTraits_CRDMode(t *testing.T) { + hv := testHypervisorFull("kvm-host-1", validUUID, nil, []string{"CUSTOM_HW_FPGA", "HW_CPU_X86_SSE42"}, nil) + s := newTestShimWithHypervisors(t, http.StatusOK, "{}", &hv) + s.config.Features.ResourceProviderTraits = FeatureModeCRD + s.config.Features.ResourceProviders = FeatureModeCRD + + t.Run("GET returns traits from CRD for KVM provider", func(t *testing.T) { + w := serveHandler(t, "GET", "/resource_providers/{uuid}/traits", + s.HandleListResourceProviderTraits, + "/resource_providers/"+validUUID+"/traits") + if w.Code != http.StatusOK { + t.Fatalf("status = %d, want %d", w.Code, http.StatusOK) + } + var resp resourceProviderTraitsResponse + if err := json.NewDecoder(w.Body).Decode(&resp); err != nil { + t.Fatalf("failed to decode response: %v", err) + } + if len(resp.Traits) != 2 { + t.Fatalf("traits count = %d, want 2", len(resp.Traits)) + } + }) + t.Run("GET returns 404 for non-KVM provider", func(t *testing.T) { + nonKVMUUID := "a1b2c3d4-e5f6-7890-abcd-ef1234567890" + w := serveHandler(t, "GET", "/resource_providers/{uuid}/traits", + s.HandleListResourceProviderTraits, + "/resource_providers/"+nonKVMUUID+"/traits") + if w.Code != http.StatusNotFound { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotFound) + } + }) + t.Run("PUT returns 501", func(t *testing.T) { + w := serveHandler(t, "PUT", "/resource_providers/{uuid}/traits", + s.HandleUpdateResourceProviderTraits, + "/resource_providers/"+validUUID+"/traits") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) + t.Run("DELETE returns 501", func(t *testing.T) { + w := serveHandler(t, "DELETE", "/resource_providers/{uuid}/traits", + s.HandleDeleteResourceProviderTraits, + "/resource_providers/"+validUUID+"/traits") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} diff --git a/internal/shim/placement/handle_resource_provider_usages.go b/internal/shim/placement/handle_resource_provider_usages.go index 82262ecc5..107beaeb2 100644 --- a/internal/shim/placement/handle_resource_provider_usages.go +++ b/internal/shim/placement/handle_resource_provider_usages.go @@ -19,5 +19,5 @@ func (s *Shim) HandleListResourceProviderUsages(w http.ResponseWriter, r *http.R if _, ok := requiredUUIDPathParam(w, r, "uuid"); !ok { return } - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.Usages) } diff --git a/internal/shim/placement/handle_resource_provider_usages_test.go b/internal/shim/placement/handle_resource_provider_usages_test.go index 76541a993..3f482127f 100644 --- a/internal/shim/placement/handle_resource_provider_usages_test.go +++ b/internal/shim/placement/handle_resource_provider_usages_test.go @@ -28,3 +28,45 @@ func TestHandleListResourceProviderUsages(t *testing.T) { } }) } + +func TestHandleResourceProviderUsages_HybridMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{Usages: FeatureModeHybrid}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("GET returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/resource_providers/{uuid}/usages", + s.HandleListResourceProviderUsages, + "/resource_providers/"+validUUID+"/usages") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} + +func TestHandleResourceProviderUsages_CRDMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{Usages: FeatureModeCRD}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("GET returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/resource_providers/{uuid}/usages", + s.HandleListResourceProviderUsages, + "/resource_providers/"+validUUID+"/usages") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} diff --git a/internal/shim/placement/handle_resource_providers.go b/internal/shim/placement/handle_resource_providers.go index 18118d90a..7d4dd1fff 100644 --- a/internal/shim/placement/handle_resource_providers.go +++ b/internal/shim/placement/handle_resource_providers.go @@ -11,6 +11,7 @@ import ( "io" "math" "net/http" + "net/url" "strconv" "strings" @@ -114,9 +115,15 @@ func (s *Shim) HandleCreateResourceProvider(w http.ResponseWriter, r *http.Reque ctx := r.Context() log := logf.FromContext(ctx) - if !s.config.Features.EnableResourceProviders { + switch s.config.Features.ResourceProviders.orDefault() { + case FeatureModePassthrough: s.forward(w, r) return + case FeatureModeHybrid, FeatureModeCRD: + // Check for KVM conflicts below. + default: + http.Error(w, "unknown feature mode", http.StatusInternalServerError) + return } // Buffer the body so we can decode it and still forward the original @@ -176,7 +183,12 @@ func (s *Shim) HandleCreateResourceProvider(w http.ResponseWriter, r *http.Reque } } - // No conflict — restore the body and forward to upstream placement. + // No conflict — forward to upstream placement (hybrid) or reject (crd). + if s.config.Features.ResourceProviders.orDefault() == FeatureModeCRD { + log.Info("crd mode: non-kvm resource provider create not supported", "name", req.Name) + http.Error(w, "resource provider not found", http.StatusNotFound) + return + } log.Info("no conflict with existing kvm hypervisor, forwarding create resource provider request to upstream placement", "name", req.Name) r.Body = io.NopCloser(bytes.NewReader(bodyBytes)) @@ -197,9 +209,15 @@ func (s *Shim) HandleShowResourceProvider(w http.ResponseWriter, r *http.Request ctx := r.Context() log := logf.FromContext(ctx) - if !s.config.Features.EnableResourceProviders { + switch s.config.Features.ResourceProviders.orDefault() { + case FeatureModePassthrough: s.forward(w, r) return + case FeatureModeHybrid, FeatureModeCRD: + // Look up in Kubernetes below. + default: + http.Error(w, "unknown feature mode", http.StatusInternalServerError) + return } uuid, ok := requiredUUIDPathParam(w, r, "uuid") @@ -211,6 +229,11 @@ func (s *Shim) HandleShowResourceProvider(w http.ResponseWriter, r *http.Request var hvs hv1.HypervisorList err := s.List(ctx, &hvs, client.MatchingFields{idxHypervisorOpenStackId: uuid}) if apierrors.IsNotFound(err) || len(hvs.Items) == 0 { + if s.config.Features.ResourceProviders.orDefault() == FeatureModeCRD { + log.Info("resource provider not found in kubernetes (crd mode)", "uuid", uuid) + http.Error(w, "resource provider not found", http.StatusNotFound) + return + } // Forward the request to placement if the hypervisor doesn't exist. log.Info("resource provider not found in kubernetes, forwarding to upstream placement", "uuid", uuid) @@ -255,9 +278,15 @@ func (s *Shim) HandleUpdateResourceProvider(w http.ResponseWriter, r *http.Reque ctx := r.Context() log := logf.FromContext(ctx) - if !s.config.Features.EnableResourceProviders { + switch s.config.Features.ResourceProviders.orDefault() { + case FeatureModePassthrough: s.forward(w, r) return + case FeatureModeHybrid, FeatureModeCRD: + // Check KVM immutability below. + default: + http.Error(w, "unknown feature mode", http.StatusInternalServerError) + return } uuid, ok := requiredUUIDPathParam(w, r, "uuid") @@ -286,6 +315,11 @@ func (s *Shim) HandleUpdateResourceProvider(w http.ResponseWriter, r *http.Reque var hvs hv1.HypervisorList err = s.List(ctx, &hvs, client.MatchingFields{idxHypervisorOpenStackId: uuid}) if apierrors.IsNotFound(err) || len(hvs.Items) == 0 { + if s.config.Features.ResourceProviders.orDefault() == FeatureModeCRD { + log.Info("resource provider not found in kubernetes (crd mode)", "uuid", uuid) + http.Error(w, "resource provider not found", http.StatusNotFound) + return + } // Forward the request to placement if the hypervisor doesn't exist. log.Info("resource provider not found in kubernetes, forwarding to upstream placement", "uuid", uuid) @@ -339,9 +373,15 @@ func (s *Shim) HandleDeleteResourceProvider(w http.ResponseWriter, r *http.Reque ctx := r.Context() log := logf.FromContext(ctx) - if !s.config.Features.EnableResourceProviders { + switch s.config.Features.ResourceProviders.orDefault() { + case FeatureModePassthrough: s.forward(w, r) return + case FeatureModeHybrid, FeatureModeCRD: + // Check KVM protection below. + default: + http.Error(w, "unknown feature mode", http.StatusInternalServerError) + return } uuid, ok := requiredUUIDPathParam(w, r, "uuid") @@ -353,6 +393,11 @@ func (s *Shim) HandleDeleteResourceProvider(w http.ResponseWriter, r *http.Reque var hvs hv1.HypervisorList err := s.List(ctx, &hvs, client.MatchingFields{idxHypervisorOpenStackId: uuid}) if apierrors.IsNotFound(err) || len(hvs.Items) == 0 { + if s.config.Features.ResourceProviders.orDefault() == FeatureModeCRD { + log.Info("resource provider not found in kubernetes (crd mode)", "uuid", uuid) + http.Error(w, "resource provider not found", http.StatusNotFound) + return + } // Forward the request to placement if the hypervisor doesn't exist. log.Info("resource provider not found in kubernetes, forwarding to upstream placement", "uuid", uuid) @@ -403,13 +448,22 @@ type listResourceProvidersResponse struct { // // See: https://docs.openstack.org/api-ref/placement/#list-resource-providers func (s *Shim) HandleListResourceProviders(w http.ResponseWriter, r *http.Request) { - ctx := r.Context() - log := logf.FromContext(ctx) - - if !s.config.Features.EnableResourceProviders { + switch s.config.Features.ResourceProviders.orDefault() { + case FeatureModePassthrough: s.forward(w, r) - return + case FeatureModeHybrid: + s.listResourceProvidersHybrid(w, r) + case FeatureModeCRD: + s.listResourceProvidersCRD(w, r) + default: + http.Error(w, "unknown feature mode", http.StatusInternalServerError) } +} + +// listResourceProvidersHybrid merges upstream and K8s results. +func (s *Shim) listResourceProvidersHybrid(w http.ResponseWriter, r *http.Request) { + ctx := r.Context() + log := logf.FromContext(ctx) s.forwardWithHook(w, r, func(w http.ResponseWriter, resp *http.Response) { if resp.StatusCode != http.StatusOK { @@ -452,30 +506,11 @@ func (s *Shim) HandleListResourceProviders(w http.ResponseWriter, r *http.Reques "count", len(hvs.Items), "uuids", uuids) // Post-filter by query parameters. - filtered := hvs.Items - if v := query.Get("uuid"); v != "" { - filtered = filterHypervisorsByUUID(ctx, filtered, v) - } - if v := query.Get("name"); v != "" { - filtered = filterHypervisorsByName(ctx, filtered, v) - } - if vals := query["member_of"]; len(vals) > 0 { - filtered = filterHypervisorsByMemberOf(ctx, filtered, vals) - } - if v := query.Get("in_tree"); v != "" { - filtered = filterHypervisorsByInTree(ctx, filtered, v) - } - if vals := query["required"]; len(vals) > 0 { - filtered = filterHypervisorsByRequired(ctx, filtered, vals) - } - if v := query.Get("resources"); v != "" { - var err error - filtered, err = filterHypervisorsByResources(ctx, filtered, v) - if err != nil { - log.Info("invalid resources query parameter", "error", err) - http.Error(w, "invalid resources query parameter: "+err.Error(), http.StatusBadRequest) - return - } + filtered, err := applyHypervisorQueryFilters(ctx, hvs.Items, query) + if err != nil { + log.Info("invalid resources query parameter", "error", err) + http.Error(w, "invalid resources query parameter: "+err.Error(), http.StatusBadRequest) + return } // Build collision sets from filtered k8s hypervisors. @@ -517,6 +552,63 @@ func (s *Shim) HandleListResourceProviders(w http.ResponseWriter, r *http.Reques }) } +// listResourceProvidersCRD serves the list from Kubernetes only, no upstream. +func (s *Shim) listResourceProvidersCRD(w http.ResponseWriter, r *http.Request) { + ctx := r.Context() + log := logf.FromContext(ctx) + query := r.URL.Query() + + var hvs hv1.HypervisorList + if err := s.List(ctx, &hvs); err != nil && !apierrors.IsNotFound(err) { + log.Error(err, "failed to list hypervisors") + http.Error(w, "Internal Server Error", http.StatusInternalServerError) + return + } + + filtered, err := applyHypervisorQueryFilters(ctx, hvs.Items, query) + if err != nil { + log.Info("invalid resources query parameter", "error", err) + http.Error(w, "invalid resources query parameter: "+err.Error(), http.StatusBadRequest) + return + } + + rps := make([]resourceProvider, 0, len(filtered)) + for _, hv := range filtered { + rps = append(rps, translateToResourceProvider(hv)) + } + log.Info("listed resource providers from kubernetes (crd mode)", "count", len(rps)) + s.writeJSON(w, http.StatusOK, listResourceProvidersResponse{ResourceProviders: rps}) +} + +// applyHypervisorQueryFilters runs the placement-style query filters against +// the given hypervisor list and returns the filtered slice. +func applyHypervisorQueryFilters(ctx context.Context, hvs []hv1.Hypervisor, query url.Values) ([]hv1.Hypervisor, error) { + filtered := hvs + if v := query.Get("uuid"); v != "" { + filtered = filterHypervisorsByUUID(ctx, filtered, v) + } + if v := query.Get("name"); v != "" { + filtered = filterHypervisorsByName(ctx, filtered, v) + } + if vals := query["member_of"]; len(vals) > 0 { + filtered = filterHypervisorsByMemberOf(ctx, filtered, vals) + } + if v := query.Get("in_tree"); v != "" { + filtered = filterHypervisorsByInTree(ctx, filtered, v) + } + if vals := query["required"]; len(vals) > 0 { + filtered = filterHypervisorsByRequired(ctx, filtered, vals) + } + if v := query.Get("resources"); v != "" { + var err error + filtered, err = filterHypervisorsByResources(ctx, filtered, v) + if err != nil { + return nil, err + } + } + return filtered, nil +} + func filterHypervisorsByUUID(ctx context.Context, hvs []hv1.Hypervisor, uuid string) []hv1.Hypervisor { log := logf.FromContext(ctx) out := make([]hv1.Hypervisor, 0, 1) diff --git a/internal/shim/placement/handle_resource_providers_e2e.go b/internal/shim/placement/handle_resource_providers_e2e.go index fdd0091d2..90b850369 100644 --- a/internal/shim/placement/handle_resource_providers_e2e.go +++ b/internal/shim/placement/handle_resource_providers_e2e.go @@ -65,8 +65,8 @@ func e2eTestResourceProviders(ctx context.Context, cl client.Client) error { // ==================== Phase 2: KVM path ==================== - if !config.Features.EnableResourceProviders { - log.Info("Skipping KVM resource provider e2e tests because enableResourceProviders is false") + if config.Features.ResourceProviders.orDefault() == FeatureModePassthrough { + log.Info("Skipping KVM resource provider e2e tests because resourceProviders mode is passthrough") } else { log.Info("=== KVM path: hypervisor-backed resource provider tests ===") if err := e2eKVMResourceProviders(ctx, sc, cl); err != nil { diff --git a/internal/shim/placement/handle_resource_providers_test.go b/internal/shim/placement/handle_resource_providers_test.go index 9b43bf818..8c73db39e 100644 --- a/internal/shim/placement/handle_resource_providers_test.go +++ b/internal/shim/placement/handle_resource_providers_test.go @@ -99,7 +99,7 @@ func newTestShimWithHypervisors(t *testing.T, upstreamStatus int, upstreamBody s Client: newFakeClient(t, hvs...), config: config{ PlacementURL: upstream.URL, - Features: featuresConfig{EnableResourceProviders: true}, + Features: featuresConfig{ResourceProviders: FeatureModeHybrid}, }, httpClient: upstream.Client(), maxBodyLogSize: 4096, @@ -884,16 +884,16 @@ func TestHandleDeleteResourceProvider(t *testing.T) { } // --------------------------------------------------------------------------- -// Feature flag tests +// Passthrough mode tests // --------------------------------------------------------------------------- -func TestHandleResourceProviders_FeatureFlagOff(t *testing.T) { +func TestHandleResourceProviders_Passthrough(t *testing.T) { hv1Obj := &hv1.Hypervisor{ ObjectMeta: metav1.ObjectMeta{Name: "hv-flagtest"}, Status: hv1.HypervisorStatus{HypervisorID: validUUID}, } - newFlagOffShim := func(t *testing.T, upstreamStatus int, upstreamBody string) *Shim { + newPassthroughShim := func(t *testing.T, upstreamStatus int, upstreamBody string) *Shim { t.Helper() upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.Header().Set("Content-Type", "application/json") @@ -908,7 +908,7 @@ func TestHandleResourceProviders_FeatureFlagOff(t *testing.T) { Client: newFakeClient(t, hv1Obj), config: config{ PlacementURL: upstream.URL, - Features: featuresConfig{EnableResourceProviders: false}, + Features: featuresConfig{ResourceProviders: FeatureModePassthrough}, }, httpClient: upstream.Client(), maxBodyLogSize: 4096, @@ -918,18 +918,18 @@ func TestHandleResourceProviders_FeatureFlagOff(t *testing.T) { } t.Run("create forwards to upstream", func(t *testing.T) { - s := newFlagOffShim(t, http.StatusCreated, `{"uuid":"new","name":"hv-flagtest"}`) + s := newPassthroughShim(t, http.StatusCreated, `{"uuid":"new","name":"hv-flagtest"}`) body := `{"name":"hv-flagtest"}` req := httptest.NewRequest(http.MethodPost, "/resource_providers", strings.NewReader(body)) w := httptest.NewRecorder() s.HandleCreateResourceProvider(w, req) if w.Code != http.StatusCreated { - t.Fatalf("status = %d, want %d (flag off should forward, not 409)", w.Code, http.StatusCreated) + t.Fatalf("status = %d, want %d (passthrough should forward, not 409)", w.Code, http.StatusCreated) } }) t.Run("show forwards to upstream", func(t *testing.T) { - s := newFlagOffShim(t, http.StatusOK, `{"uuid":"`+validUUID+`","name":"upstream-rp"}`) + s := newPassthroughShim(t, http.StatusOK, `{"uuid":"`+validUUID+`","name":"upstream-rp"}`) w := serveHandler(t, http.MethodGet, "/resource_providers/{uuid}", s.HandleShowResourceProvider, "/resource_providers/"+validUUID) if w.Code != http.StatusOK { @@ -941,7 +941,7 @@ func TestHandleResourceProviders_FeatureFlagOff(t *testing.T) { }) t.Run("update forwards to upstream", func(t *testing.T) { - s := newFlagOffShim(t, http.StatusOK, `{"uuid":"`+validUUID+`","name":"different-name"}`) + s := newPassthroughShim(t, http.StatusOK, `{"uuid":"`+validUUID+`","name":"different-name"}`) body := `{"name":"different-name"}` req := httptest.NewRequest(http.MethodPut, "/resource_providers/"+validUUID, strings.NewReader(body)) mux := http.NewServeMux() @@ -949,22 +949,22 @@ func TestHandleResourceProviders_FeatureFlagOff(t *testing.T) { w := httptest.NewRecorder() mux.ServeHTTP(w, req) if w.Code != http.StatusOK { - t.Fatalf("status = %d, want %d (flag off should forward, not 409)", w.Code, http.StatusOK) + t.Fatalf("status = %d, want %d (passthrough should forward, not 409)", w.Code, http.StatusOK) } }) t.Run("delete forwards to upstream", func(t *testing.T) { - s := newFlagOffShim(t, http.StatusNoContent, "") + s := newPassthroughShim(t, http.StatusNoContent, "") w := serveHandler(t, http.MethodDelete, "/resource_providers/{uuid}", s.HandleDeleteResourceProvider, "/resource_providers/"+validUUID) if w.Code != http.StatusNoContent { - t.Fatalf("status = %d, want %d (flag off should forward, not 409)", w.Code, http.StatusNoContent) + t.Fatalf("status = %d, want %d (passthrough should forward, not 409)", w.Code, http.StatusNoContent) } }) t.Run("list forwards to upstream without merge", func(t *testing.T) { upstreamBody := `{"resource_providers":[{"uuid":"upstream-uuid","name":"upstream-rp","generation":1,"links":[]}]}` - s := newFlagOffShim(t, http.StatusOK, upstreamBody) + s := newPassthroughShim(t, http.StatusOK, upstreamBody) w := serveHandler(t, http.MethodGet, "/resource_providers", s.HandleListResourceProviders, "/resource_providers") if w.Code != http.StatusOK { @@ -974,7 +974,7 @@ func TestHandleResourceProviders_FeatureFlagOff(t *testing.T) { t.Errorf("expected upstream body passthrough, got %q", w.Body.String()) } if strings.Contains(w.Body.String(), validUUID) { - t.Errorf("should not contain k8s hypervisor UUID when flag is off, got %q", w.Body.String()) + t.Errorf("should not contain k8s hypervisor UUID in passthrough mode, got %q", w.Body.String()) } }) } diff --git a/internal/shim/placement/handle_root.go b/internal/shim/placement/handle_root.go index c0ac2b4fc..acad69dcb 100644 --- a/internal/shim/placement/handle_root.go +++ b/internal/shim/placement/handle_root.go @@ -4,7 +4,11 @@ package placement import ( + "encoding/json" + "io" "net/http" + "strconv" + "strings" logf "sigs.k8s.io/controller-runtime/pkg/log" ) @@ -36,22 +40,55 @@ type versionLink struct { // supported by the running service. Clients use this endpoint to discover API // capabilities and negotiate microversions before making further requests. // -// When features.enableRoot is true, the response is served from the static -// versioning config. When false, the request is forwarded to upstream placement. +// In passthrough mode, the request is forwarded to upstream placement. In +// hybrid mode, the shim returns the intersection (narrower range) of the +// upstream and local version configs. In crd mode, the response is served +// from the static versioning config alone. // // See: https://docs.openstack.org/api-ref/placement/#list-versions func (s *Shim) HandleGetRoot(w http.ResponseWriter, r *http.Request) { ctx := r.Context() log := logf.FromContext(ctx) - if !s.config.Features.EnableRoot { + switch s.config.Features.Root.orDefault() { + case FeatureModePassthrough: log.Info("forwarding GET / to upstream placement") s.forward(w, r) - return + + case FeatureModeHybrid: + log.Info("handling GET / in hybrid mode (version intersection)") + s.forwardWithHook(w, r, func(w http.ResponseWriter, resp *http.Response) { + if resp.StatusCode != http.StatusOK { + for k, vs := range resp.Header { + for _, v := range vs { + w.Header().Add(k, v) + } + } + w.WriteHeader(resp.StatusCode) + io.Copy(w, resp.Body) //nolint:errcheck + return + } + var upstream versionDocument + if err := json.NewDecoder(resp.Body).Decode(&upstream); err != nil { + log.Error(err, "failed to decode upstream version document") + s.writeJSON(w, http.StatusOK, s.staticVersionDocument()) + return + } + merged := s.intersectVersions(upstream) + s.writeJSON(w, http.StatusOK, merged) + }) + + case FeatureModeCRD: + log.Info("handling GET / with static version document") + s.writeJSON(w, http.StatusOK, s.staticVersionDocument()) + + default: + http.Error(w, "unknown feature mode", http.StatusInternalServerError) } +} - log.Info("handling GET / with static version document") - s.writeJSON(w, http.StatusOK, versionDocument{ +func (s *Shim) staticVersionDocument() versionDocument { + return versionDocument{ Versions: []versionEntry{{ ID: s.config.Versioning.ID, MaxVersion: s.config.Versioning.MaxVersion, @@ -59,5 +96,75 @@ func (s *Shim) HandleGetRoot(w http.ResponseWriter, r *http.Request) { Status: s.config.Versioning.Status, Links: []versionLink{{Rel: "self", Href: ""}}, }}, - }) + } +} + +// intersectVersions computes the intersection of the upstream and local +// version ranges. The result uses the higher min and lower max, yielding +// the narrowest compatible window. When no upstream version entries exist +// or the ranges don't overlap, the local config is returned as-is. +func (s *Shim) intersectVersions(upstream versionDocument) versionDocument { + if len(upstream.Versions) == 0 { + return s.staticVersionDocument() + } + uv := upstream.Versions[0] + localMin := s.config.Versioning.MinVersion + localMax := s.config.Versioning.MaxVersion + + mergedMin := maxVersion(localMin, uv.MinVersion) + mergedMax := minVersion(localMax, uv.MaxVersion) + + if compareVersions(mergedMin, mergedMax) > 0 { + return s.staticVersionDocument() + } + return versionDocument{ + Versions: []versionEntry{{ + ID: s.config.Versioning.ID, + MaxVersion: mergedMax, + MinVersion: mergedMin, + Status: s.config.Versioning.Status, + Links: []versionLink{{Rel: "self", Href: ""}}, + }}, + } +} + +// compareVersions compares two dot-separated version strings numerically. +// Returns -1, 0, or 1. +func compareVersions(a, b string) int { + aParts := strings.Split(a, ".") + bParts := strings.Split(b, ".") + for i := 0; i < len(aParts) || i < len(bParts); i++ { + var av, bv int + if i < len(aParts) { + if v, err := strconv.Atoi(aParts[i]); err == nil { + av = v + } + } + if i < len(bParts) { + if v, err := strconv.Atoi(bParts[i]); err == nil { + bv = v + } + } + if av < bv { + return -1 + } + if av > bv { + return 1 + } + } + return 0 +} + +func maxVersion(a, b string) string { + if compareVersions(a, b) >= 0 { + return a + } + return b +} + +func minVersion(a, b string) string { + if compareVersions(a, b) <= 0 { + return a + } + return b } diff --git a/internal/shim/placement/handle_root_test.go b/internal/shim/placement/handle_root_test.go index f1795fe2f..7216365ef 100644 --- a/internal/shim/placement/handle_root_test.go +++ b/internal/shim/placement/handle_root_test.go @@ -27,7 +27,7 @@ func TestHandleGetRootStatic(t *testing.T) { s := &Shim{ config: config{ PlacementURL: "http://should-not-be-called:1234", - Features: featuresConfig{EnableRoot: true}, + Features: featuresConfig{Root: FeatureModeCRD}, Versioning: &versioningConfig{ ID: "v1.0", MinVersion: "1.0", @@ -75,14 +75,14 @@ func TestHandleGetRootStatic(t *testing.T) { func TestHandleGetRootStaticDoesNotCallUpstream(t *testing.T) { upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - t.Fatal("upstream should not be called when enableRoot is true") + t.Fatal("upstream should not be called when root mode is crd") })) t.Cleanup(upstream.Close) down, up := newTestTimers() s := &Shim{ config: config{ PlacementURL: upstream.URL, - Features: featuresConfig{EnableRoot: true}, + Features: featuresConfig{Root: FeatureModeCRD}, Versioning: &versioningConfig{ ID: "v1.0", MinVersion: "1.0", @@ -101,3 +101,77 @@ func TestHandleGetRootStaticDoesNotCallUpstream(t *testing.T) { t.Fatalf("status = %d, want %d", w.Code, http.StatusOK) } } + +func TestHandleGetRootHybrid(t *testing.T) { + // Upstream returns a version document with maxVersion "1.35". + upstreamDoc := versionDocument{ + Versions: []versionEntry{{ + ID: "v1.0", + MaxVersion: "1.35", + MinVersion: "1.0", + Status: "CURRENT", + Links: []versionLink{{Rel: "self", Href: ""}}, + }}, + } + upstreamBody, err := json.Marshal(upstreamDoc) + if err != nil { + t.Fatalf("failed to marshal upstream doc: %v", err) + } + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusOK) + if _, err := w.Write(upstreamBody); err != nil { + t.Errorf("failed to write upstream body: %v", err) + } + })) + t.Cleanup(upstream.Close) + + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: upstream.URL, + Features: featuresConfig{Root: FeatureModeHybrid}, + Versioning: &versioningConfig{ + ID: "v1.0", + MinVersion: "1.0", + MaxVersion: "1.39", + Status: "CURRENT", + }, + }, + httpClient: upstream.Client(), + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + + w := serveHandler(t, "GET", "/{$}", s.HandleGetRoot, "/") + if w.Code != http.StatusOK { + t.Fatalf("status = %d, want %d", w.Code, http.StatusOK) + } + if ct := w.Header().Get("Content-Type"); ct != "application/json" { + t.Fatalf("Content-Type = %q, want %q", ct, "application/json") + } + + var doc versionDocument + if err := json.NewDecoder(w.Body).Decode(&doc); err != nil { + t.Fatalf("failed to decode response: %v", err) + } + if len(doc.Versions) != 1 { + t.Fatalf("versions count = %d, want 1", len(doc.Versions)) + } + v := doc.Versions[0] + // The intersection of upstream [1.0, 1.35] and local [1.0, 1.39] + // should yield min "1.0" and max "1.35" (the narrower range). + if v.MinVersion != "1.0" { + t.Errorf("min_version = %q, want %q", v.MinVersion, "1.0") + } + if v.MaxVersion != "1.35" { + t.Errorf("max_version = %q, want %q", v.MaxVersion, "1.35") + } + if v.ID != "v1.0" { + t.Errorf("id = %q, want %q", v.ID, "v1.0") + } + if v.Status != "CURRENT" { + t.Errorf("status = %q, want %q", v.Status, "CURRENT") + } +} diff --git a/internal/shim/placement/handle_traits.go b/internal/shim/placement/handle_traits.go index b35dd8b79..b509b2599 100644 --- a/internal/shim/placement/handle_traits.go +++ b/internal/shim/placement/handle_traits.go @@ -7,6 +7,7 @@ import ( "context" "encoding/json" "fmt" + "math/rand" "net/http" "net/url" "os" @@ -14,9 +15,12 @@ import ( "strings" "time" + "github.com/go-logr/logr" + "github.com/gophercloud/gophercloud/v2" corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" logf "sigs.k8s.io/controller-runtime/pkg/log" ) @@ -58,10 +62,17 @@ func (s *Shim) HandleListTraits(w http.ResponseWriter, r *http.Request) { ctx := r.Context() log := logf.FromContext(ctx) - if !s.config.Features.EnableTraits { + switch s.config.Features.Traits.orDefault() { + case FeatureModePassthrough, FeatureModeHybrid: s.forward(w, r) return + case FeatureModeCRD: + // Serve from local ConfigMaps. + default: + http.Error(w, "unknown feature mode", http.StatusInternalServerError) + return } + traitSet, err := s.getAllTraits(ctx) if err != nil { log.Error(err, "failed to list traits from configmaps") @@ -121,10 +132,17 @@ func (s *Shim) HandleShowTrait(w http.ResponseWriter, r *http.Request) { ctx := r.Context() log := logf.FromContext(ctx) - if !s.config.Features.EnableTraits { + switch s.config.Features.Traits.orDefault() { + case FeatureModePassthrough, FeatureModeHybrid: s.forward(w, r) return + case FeatureModeCRD: + // Serve from local ConfigMaps. + default: + http.Error(w, "unknown feature mode", http.StatusInternalServerError) + return } + name, ok := requiredPathParam(w, r, "name") if !ok { return @@ -156,10 +174,17 @@ func (s *Shim) HandleUpdateTrait(w http.ResponseWriter, r *http.Request) { ctx := r.Context() log := logf.FromContext(ctx) - if !s.config.Features.EnableTraits { + switch s.config.Features.Traits.orDefault() { + case FeatureModePassthrough, FeatureModeHybrid: s.forward(w, r) return + case FeatureModeCRD: + // Serve from local ConfigMaps. + default: + http.Error(w, "unknown feature mode", http.StatusInternalServerError) + return } + name, ok := requiredPathParam(w, r, "name") if !ok { return @@ -273,10 +298,17 @@ func (s *Shim) HandleDeleteTrait(w http.ResponseWriter, r *http.Request) { ctx := r.Context() log := logf.FromContext(ctx) - if !s.config.Features.EnableTraits { + switch s.config.Features.Traits.orDefault() { + case FeatureModePassthrough, FeatureModeHybrid: s.forward(w, r) return + case FeatureModeCRD: + // Serve from local ConfigMaps. + default: + http.Error(w, "unknown feature mode", http.StatusInternalServerError) + return } + name, ok := requiredPathParam(w, r, "name") if !ok { return @@ -462,3 +494,86 @@ func (s *Shim) syncTraitToUpstream(ctx context.Context, name string, incomingHea defer resp.Body.Close() log.Info("synced custom trait to upstream placement", "trait", name, "status", resp.StatusCode) } + +// startTraitSyncLoop runs a periodic goroutine that fetches traits from +// upstream placement and writes them into the static ConfigMap. Only active +// when features.traits is hybrid. The loop exits when ctx is cancelled. +func (s *Shim) startTraitSyncLoop(ctx context.Context) { + if s.config.Features.Traits.orDefault() != FeatureModeHybrid { + return + } + log := ctrl.Log.WithName("placement-shim").WithName("trait-sync") + jitter := time.Duration(rand.Int63n(int64(30 * time.Second))) //nolint:gosec + log.Info("starting trait sync loop", "jitter", jitter) + + select { + case <-ctx.Done(): + return + case <-time.After(jitter): + } + + s.syncTraitsFromUpstream(ctx, log) + + ticker := time.NewTicker(60 * time.Second) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + s.syncTraitsFromUpstream(ctx, log) + } + } +} + +// syncTraitsFromUpstream fetches GET /traits from upstream placement and +// writes the result into the static ConfigMap so that the shim's local +// view stays in sync with upstream. Uses the gophercloud ServiceClient +// for automatic token management (including reauth on 401). +func (s *Shim) syncTraitsFromUpstream(ctx context.Context, log logr.Logger) { + if s.placementServiceClient == nil { + log.V(1).Info("skipping upstream trait sync, no placement service client configured") + return + } + u, err := url.JoinPath(s.placementServiceClient.Endpoint, "/traits") + if err != nil { + log.Error(err, "failed to build upstream traits URL") + return + } + resp, err := s.placementServiceClient.Request(ctx, http.MethodGet, u, &gophercloud.RequestOpts{ + OkCodes: []int{http.StatusOK}, + MoreHeaders: map[string]string{ + "OpenStack-API-Version": "placement 1.6", + }, + KeepResponseBody: true, + }) + if err != nil { + log.Info("upstream trait sync failed", "error", err.Error()) + return + } + defer resp.Body.Close() + var body traitsListResponse + if err := json.NewDecoder(resp.Body).Decode(&body); err != nil { + log.Error(err, "failed to decode upstream trait list") + return + } + + cm := &corev1.ConfigMap{} + if err := s.Get(ctx, s.staticTraitsConfigMapKey(), cm); err != nil { + log.Error(err, "failed to get static traits configmap for sync") + return + } + traitSet := make(map[string]struct{}, len(body.Traits)) + for _, t := range body.Traits { + traitSet[t] = struct{}{} + } + if err := s.writeTraits(cm, traitSet); err != nil { + log.Error(err, "failed to serialize synced traits") + return + } + if err := s.Update(ctx, cm); err != nil { + log.Error(err, "failed to update static traits configmap with upstream data") + return + } + log.Info("synced traits from upstream placement", "count", len(body.Traits)) +} diff --git a/internal/shim/placement/handle_traits_e2e.go b/internal/shim/placement/handle_traits_e2e.go index 1a0ca5495..8a904b935 100644 --- a/internal/shim/placement/handle_traits_e2e.go +++ b/internal/shim/placement/handle_traits_e2e.go @@ -19,13 +19,13 @@ import ( // // Phase 1 — read-only (always runs): // -// 1. GET /traits — list all traits; when forwarding to upstream (enableTraits -// is false) verify at least one trait exists. +// 1. GET /traits — list all traits; when traits mode is passthrough +// (forwarding to upstream) verify at least one trait exists. // 2. GET /traits/{name} — show a known trait from the list and verify 200 // (skipped when the trait list is empty). // 3. GET /traits/{name} — show a nonexistent trait and verify 404. // -// Phase 2 — CRUD (only when enableTraits is true): +// Phase 2 — CRUD (only when traits mode is non-passthrough): // // 1. Pre-cleanup: DELETE any leftover test trait (ignore 404). // 2. PUT /traits/{name} — create a custom test trait → 201. @@ -80,10 +80,10 @@ func e2eTestTraits(ctx context.Context, _ client.Client) error { if err := json.NewDecoder(resp.Body).Decode(&listResp); err != nil { return fmt.Errorf("failed to decode GET /traits response: %w", err) } - // When traits are served locally (enableTraits=true) the static list may + // When traits are served locally (hybrid or crd mode) the static list may // be empty. Only require at least one trait when forwarding to upstream // placement, which always has standard traits. - if !config.Features.EnableTraits && len(listResp.Traits) == 0 { + if config.Features.Traits.orDefault() == FeatureModePassthrough && len(listResp.Traits) == 0 { return errors.New("GET /traits: expected at least one trait, got 0") } log.Info("Successfully retrieved traits", "count", len(listResp.Traits)) @@ -133,12 +133,12 @@ func e2eTestTraits(ctx context.Context, _ client.Client) error { // ==================== Phase 2: CRUD tests (feature-gated) ==================== - if !config.Features.EnableTraits { - log.Info("Skipping trait CRUD e2e tests because enableTraits is false") + if config.Features.Traits.orDefault() == FeatureModePassthrough { + log.Info("Skipping trait CRUD e2e tests because traits mode is passthrough") return nil } - log.Info("=== Phase 2: CRUD trait tests (enableTraits=true) ===") + log.Info("=== Phase 2: CRUD trait tests (traits mode non-passthrough) ===") const testTrait = "CUSTOM_CORTEX_E2E_TRAIT" diff --git a/internal/shim/placement/handle_traits_test.go b/internal/shim/placement/handle_traits_test.go index 3faeb6feb..bf692fd41 100644 --- a/internal/shim/placement/handle_traits_test.go +++ b/internal/shim/placement/handle_traits_test.go @@ -55,7 +55,7 @@ func newTraitShim(t *testing.T, staticTraits []string, customTraits ...string) * Client: cl, config: config{ PlacementURL: "http://should-not-be-called:1234", - Features: featuresConfig{EnableTraits: true}, + Features: featuresConfig{Traits: FeatureModeCRD}, Traits: &traitsConfig{ConfigMapName: "test-cm"}, }, maxBodyLogSize: 4096, @@ -65,7 +65,7 @@ func newTraitShim(t *testing.T, staticTraits []string, customTraits ...string) * } } -// --- Passthrough tests (enableTraits=false) --- +// --- Passthrough mode tests --- func TestHandleListTraitsPassthrough(t *testing.T) { var gotPath string @@ -107,7 +107,7 @@ func TestHandleDeleteTraitPassthrough(t *testing.T) { } } -// --- Handler tests (enableTraits=true) --- +// --- CRD mode handler tests --- func TestHandleListTraitsLocal(t *testing.T) { s := newTraitShim(t, []string{"CUSTOM_FOO", "HW_CPU_X86_AVX2", "STORAGE_DISK_SSD"}) diff --git a/internal/shim/placement/handle_usages.go b/internal/shim/placement/handle_usages.go index 7b99beaa2..2bbc5b472 100644 --- a/internal/shim/placement/handle_usages.go +++ b/internal/shim/placement/handle_usages.go @@ -20,5 +20,5 @@ import ( // microversion 1.38, an optional consumer_type query parameter allows // filtering the results. Available since microversion 1.9. func (s *Shim) HandleListUsages(w http.ResponseWriter, r *http.Request) { - s.forward(w, r) + s.dispatchPassthroughOnly(w, r, s.config.Features.Usages) } diff --git a/internal/shim/placement/handle_usages_test.go b/internal/shim/placement/handle_usages_test.go index 46d91681b..f106bcd70 100644 --- a/internal/shim/placement/handle_usages_test.go +++ b/internal/shim/placement/handle_usages_test.go @@ -19,3 +19,43 @@ func TestHandleListUsages(t *testing.T) { t.Fatalf("upstream path = %q, want /usages", gotPath) } } + +func TestHandleUsages_HybridMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{Usages: FeatureModeHybrid}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("GET returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/usages", + s.HandleListUsages, "/usages") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} + +func TestHandleUsages_CRDMode(t *testing.T) { + down, up := newTestTimers() + s := &Shim{ + config: config{ + PlacementURL: "http://should-not-be-called:1234", + Features: featuresConfig{Usages: FeatureModeCRD}, + }, + maxBodyLogSize: 4096, + downstreamRequestTimer: down, + upstreamRequestTimer: up, + } + t.Run("GET returns 501", func(t *testing.T) { + w := serveHandler(t, "GET", "/usages", + s.HandleListUsages, "/usages") + if w.Code != http.StatusNotImplemented { + t.Fatalf("status = %d, want %d", w.Code, http.StatusNotImplemented) + } + }) +} diff --git a/internal/shim/placement/shim.go b/internal/shim/placement/shim.go index 1ea8f19d5..273273983 100644 --- a/internal/shim/placement/shim.go +++ b/internal/shim/placement/shim.go @@ -20,6 +20,8 @@ import ( "github.com/cobaltcore-dev/cortex/pkg/resourcelock" "github.com/cobaltcore-dev/cortex/pkg/sso" hv1 "github.com/cobaltcore-dev/openstack-hypervisor-operator/api/v1" + "github.com/gophercloud/gophercloud/v2" + "github.com/gophercloud/gophercloud/v2/openstack" "github.com/prometheus/client_golang/prometheus" "k8s.io/apimachinery/pkg/api/resource" ctrl "sigs.k8s.io/controller-runtime" @@ -50,27 +52,72 @@ type requestIDContextKey struct{} // header value through the request lifecycle for tracing. var requestIDKey = requestIDContextKey{} -// featuresConfig holds feature flags that can enable or disable specific -// shim behaviors. All flags default to off (false). +// FeatureMode controls how an endpoint group interacts with upstream +// placement and the hypervisor CRD. +type FeatureMode string + +const ( + // FeatureModePassthrough forwards all requests to upstream placement + // without any shim logic. + FeatureModePassthrough FeatureMode = "passthrough" + // FeatureModeHybrid directs requests to both upstream placement and the + // hypervisor CRD. Upstream must respond; the shim keeps CRD state in + // sync to prepare for cutover. + FeatureModeHybrid FeatureMode = "hybrid" + // FeatureModeCRD serves requests exclusively from the hypervisor CRD. + // No upstream placement dependency is required. + FeatureModeCRD FeatureMode = "crd" +) + +// orDefault returns FeatureModePassthrough when m is the zero value. +func (m FeatureMode) orDefault() FeatureMode { + if m == "" { + return FeatureModePassthrough + } + return m +} + +// valid reports whether m is a recognized feature mode (including the +// zero value, which maps to passthrough). +func (m FeatureMode) valid() bool { + switch m { + case FeatureModePassthrough, FeatureModeHybrid, FeatureModeCRD, "": + return true + } + return false +} + +// dispatchPassthroughOnly forwards in passthrough mode, returns 501 for +// hybrid/crd, and 500 for unknown modes. +func (s *Shim) dispatchPassthroughOnly(w http.ResponseWriter, r *http.Request, mode FeatureMode) { + switch mode.orDefault() { + case FeatureModePassthrough: + s.forward(w, r) + case FeatureModeHybrid, FeatureModeCRD: + http.Error(w, fmt.Sprintf("%s mode is not yet implemented for this endpoint", mode), http.StatusNotImplemented) + default: + http.Error(w, "unknown feature mode", http.StatusInternalServerError) + } +} + +// featuresConfig controls the feature mode for each endpoint group. +// Every field defaults to passthrough (zero value) when omitted. type featuresConfig struct { - // EnableResourceProviders enables the KVM-specific resource provider - // logic (hypervisor lookups, merged listings, 409 conflicts). When - // false, all resource provider handlers forward to upstream placement - // as a pure passthrough. - EnableResourceProviders bool `json:"enableResourceProviders,omitempty"` - // EnableRoot makes the GET / handler return a static version discovery - // document from the Versioning config instead of forwarding to - // upstream placement. When false, GET / is forwarded as-is. - EnableRoot bool `json:"enableRoot,omitempty"` - // EnableTraits makes the trait handlers (GET /traits, GET /traits/{name}, - // PUT /traits/{name}, DELETE /traits/{name}) serve from a local ConfigMap - // instead of forwarding to upstream placement. When false, all trait - // requests are forwarded as-is. - EnableTraits bool `json:"enableTraits,omitempty"` + ResourceProviders FeatureMode `json:"resourceProviders,omitempty"` + Root FeatureMode `json:"root,omitempty"` + Traits FeatureMode `json:"traits,omitempty"` + ResourceProviderTraits FeatureMode `json:"resourceProviderTraits,omitempty"` + ResourceClasses FeatureMode `json:"resourceClasses,omitempty"` + Inventories FeatureMode `json:"inventories,omitempty"` + Aggregates FeatureMode `json:"aggregates,omitempty"` + Allocations FeatureMode `json:"allocations,omitempty"` + Usages FeatureMode `json:"usages,omitempty"` + AllocationCandidates FeatureMode `json:"allocationCandidates,omitempty"` + Reshaper FeatureMode `json:"reshaper,omitempty"` } // versioningConfig describes the Placement API version advertised by the -// static root endpoint when features.enableRoot is true. +// static root endpoint when features.root is hybrid or crd. type versioningConfig struct { ID string `json:"id"` MinVersion string `json:"minVersion"` @@ -79,7 +126,7 @@ type versioningConfig struct { } // traitsConfig configures the local trait store used when -// features.enableTraits is true. +// features.traits is hybrid or crd. type traitsConfig struct { // ConfigMapName is the name of the ConfigMap used to persist traits. // Must exist in the same namespace as the shim pod. @@ -123,14 +170,13 @@ type config struct { // Kubernetes resource.Quantity string (e.g. "4Ki"). Defaults to "4Ki" // when unset or empty. MaxBodyLogSize string `json:"maxBodyLogSize,omitempty"` - // Features holds feature flags for enabling or disabling specific - // shim behaviors. + // Features controls the feature mode for each endpoint group. Features featuresConfig `json:"features"` // Versioning configures the static version discovery document returned - // by GET / when features.enableRoot is true. + // by GET / when features.root is hybrid or crd. Versioning *versioningConfig `json:"versioning,omitempty"` // Traits configures the local trait store used when - // features.enableTraits is true. + // features.traits is hybrid or crd. Traits *traitsConfig `json:"traits,omitempty"` } @@ -140,23 +186,42 @@ func (c *config) validate() error { if c.PlacementURL == "" { return errors.New("placement URL is required") } - if c.Features.EnableRoot { + for name, mode := range map[string]FeatureMode{ + "resourceProviders": c.Features.ResourceProviders, + "root": c.Features.Root, + "traits": c.Features.Traits, + "resourceProviderTraits": c.Features.ResourceProviderTraits, + "resourceClasses": c.Features.ResourceClasses, + "inventories": c.Features.Inventories, + "aggregates": c.Features.Aggregates, + "allocations": c.Features.Allocations, + "usages": c.Features.Usages, + "allocationCandidates": c.Features.AllocationCandidates, + "reshaper": c.Features.Reshaper, + } { + if !mode.valid() { + return fmt.Errorf("features.%s has invalid mode %q (must be passthrough, hybrid, or crd)", name, mode) + } + } + rootMode := c.Features.Root.orDefault() + if rootMode == FeatureModeHybrid || rootMode == FeatureModeCRD { if c.Versioning == nil { - return errors.New("versioning config is required when features.enableRoot is true") + return fmt.Errorf("versioning config is required when features.root is %s", rootMode) } if c.Versioning.ID == "" || c.Versioning.MinVersion == "" || c.Versioning.MaxVersion == "" || c.Versioning.Status == "" { - return errors.New("versioning id, minVersion, maxVersion, and status are required when features.enableRoot is true") + return fmt.Errorf("versioning id, minVersion, maxVersion, and status are required when features.root is %s", rootMode) } } - if c.Features.EnableTraits { + traitsMode := c.Features.Traits.orDefault() + if traitsMode == FeatureModeHybrid || traitsMode == FeatureModeCRD { if c.Traits == nil { - return errors.New("traits config is required when features.enableTraits is true") + return fmt.Errorf("traits config is required when features.traits is %s", traitsMode) } if c.Traits.ConfigMapName == "" { - return errors.New("traits.configMapName is required when features.enableTraits is true") + return fmt.Errorf("traits.configMapName is required when features.traits is %s", traitsMode) } - if os.Getenv("POD_NAMESPACE") == "" { - return errors.New("pod namespace (POD_NAMESPACE) is required when features.enableTraits is true") + if traitsMode == FeatureModeCRD && os.Getenv("POD_NAMESPACE") == "" { + return errors.New("pod namespace (POD_NAMESPACE) is required when features.traits is crd") } } if c.Auth != nil && c.KeystoneURL == "" { @@ -207,6 +272,11 @@ type Shim struct { // resourceLocker serializes writes to the custom traits ConfigMap // across replicas using a Kubernetes Lease. resourceLocker *resourcelock.ResourceLocker + // placementServiceClient is an authenticated gophercloud service client + // used by background tasks (trait sync) to make requests to upstream + // placement with automatic token management (including reauth on 401). + // Nil when Keystone credentials are not configured. + placementServiceClient *gophercloud.ServiceClient } // Describe implements prometheus.Collector. @@ -272,13 +342,65 @@ func (s *Shim) initHTTPClient(ctx context.Context) error { return nil } +// initPlacementServiceClient creates an authenticated gophercloud +// ServiceClient that background tasks (e.g. the trait sync loop) use to +// make requests to upstream placement with automatic token management. +// After initial Keystone authentication the provider's HTTP transport is +// replaced with the shim's own transport (which carries SSO TLS certs) +// so that subsequent placement requests use the correct transport. +// Skipped when Keystone credentials are not configured. +func (s *Shim) initPlacementServiceClient(ctx context.Context) error { + if s.config.KeystoneURL == "" || s.config.OSUsername == "" || s.config.OSPassword == "" { + setupLog.Info("Keystone credentials not configured, background tasks will make unauthenticated upstream requests") + return nil + } + authOpts := gophercloud.AuthOptions{ + IdentityEndpoint: s.config.KeystoneURL, + Username: s.config.OSUsername, + DomainName: s.config.OSUserDomainName, + Password: s.config.OSPassword, + AllowReauth: true, + Scope: &gophercloud.AuthScope{ + ProjectName: s.config.OSProjectName, + DomainName: s.config.OSProjectDomainName, + }, + } + provider, err := openstack.NewClient(s.config.KeystoneURL) + if err != nil { + return fmt.Errorf("creating Keystone provider for upstream auth: %w", err) + } + provider.HTTPClient = http.Client{Timeout: 30 * time.Second} + if err := openstack.Authenticate(ctx, provider, authOpts); err != nil { + return fmt.Errorf("authenticating with Keystone for upstream auth: %w", err) + } + // After successful Keystone auth, switch the provider's HTTP transport + // to the shim's transport so placement requests use SSO TLS certs. + if s.httpClient != nil && s.httpClient.Transport != nil { + provider.HTTPClient.Transport = s.httpClient.Transport + } + s.placementServiceClient = &gophercloud.ServiceClient{ + ProviderClient: provider, + Endpoint: s.config.PlacementURL, + Type: "placement", + } + setupLog.Info("Placement service client initialized for background upstream requests") + return nil +} + // Start is called after the manager has started and the cache is running. func (s *Shim) Start(ctx context.Context) error { setupLog.Info("Starting placement shim") if err := s.initHTTPClient(ctx); err != nil { return err } - return s.initTokenIntrospector(ctx) + if err := s.initTokenIntrospector(ctx); err != nil { + return err + } + if err := s.initPlacementServiceClient(ctx); err != nil { + return err + } + go s.startTraitSyncLoop(ctx) + return nil } // Reconcile is not used by the shim, but must be implemented to satisfy the @@ -350,7 +472,8 @@ func (s *Shim) SetupWithManager(ctx context.Context, mgr ctrl.Manager) (err erro Buckets: prometheus.DefBuckets, }, []string{"method", "pattern", "responsecode"}) - if s.config.Features.EnableTraits { + traitsMode := s.config.Features.Traits.orDefault() + if traitsMode == FeatureModeHybrid || traitsMode == FeatureModeCRD { s.resourceLocker = resourcelock.NewResourceLocker( s.Client, os.Getenv("POD_NAMESPACE"), diff --git a/internal/shim/placement/shim_test.go b/internal/shim/placement/shim_test.go index 53a7137de..503b94c72 100644 --- a/internal/shim/placement/shim_test.go +++ b/internal/shim/placement/shim_test.go @@ -435,13 +435,13 @@ func TestConfigValidateAuthRequiresKeystoneURL(t *testing.T) { } } -func TestConfigValidateEnableRootRequiresVersioning(t *testing.T) { +func TestConfigValidateRootCRDRequiresVersioning(t *testing.T) { c := config{ PlacementURL: "http://placement:8778", - Features: featuresConfig{EnableRoot: true}, + Features: featuresConfig{Root: FeatureModeCRD}, } if err := c.validate(); err == nil { - t.Fatal("expected error when enableRoot is true without versioning config") + t.Fatal("expected error when root mode is crd without versioning config") } c.Versioning = &versioningConfig{ID: "v1.0"} if err := c.validate(); err == nil { @@ -458,15 +458,15 @@ func TestConfigValidateEnableRootRequiresVersioning(t *testing.T) { } } -func TestConfigValidateEnableTraitsRequiresConfig(t *testing.T) { +func TestConfigValidateTraitsCRDRequiresConfig(t *testing.T) { t.Setenv("POD_NAMESPACE", "") c := config{ PlacementURL: "http://placement:8778", - Features: featuresConfig{EnableTraits: true}, + Features: featuresConfig{Traits: FeatureModeCRD}, } if err := c.validate(); err == nil { - t.Fatal("expected error when enableTraits is true without traits config") + t.Fatal("expected error when traits mode is crd without traits config") } c.Traits = &traitsConfig{} if err := c.validate(); err == nil {