fix: pre-populate rule cache on first NodeReconciler pass to prevent bootstrap-only taint bypass

[Karman580]

Description

RuleReadinessController maintains an in-memory ruleCache populated lazily as RuleReconciler processes each NodeReadinessRule. NodeReconciler derives applicable rules exclusively from this cache. Both controllers start consuming their work queues concurrently after informer sync, creating a race where node events are processed against an empty or partially-warm cache.

For continuous enforcement this window is self-healing — the next condition, taint, or label change re-triggers evaluation and the controller converges. For bootstrap-only rules it is a permanent correctness gap: if a node is evaluated before its applicable rule reaches the cache, the taint is never applied, markBootstrapCompleted is never reached, the bootstrap completion annotation is never written, and the intended admission gate is silently bypassed with no error surfaced.

Root Cause

ensureCacheWarmed did not exist. The ruleCache was built entirely as a side effect of RuleReconciler.Reconcile(). Although the controller-runtime manager guarantees informers are synced before controllers start, there was no mechanism to seed ruleCache from the already-available informer data before NodeReconciler began processing events.

Fix

Add ensureCacheWarmed to RuleReadinessController. On the first call it lists all NodeReadinessRules from the already-synced informer cache (no live API round-trip) and adds any rule not yet present in ruleCache. Subsequent calls are no-ops guarded by an atomic.Bool. Existing cache entries — including rules already marked for deletion by RuleReconciler — are intentionally preserved so that in-flight state is not overwritten.

ensureCacheWarmed is called at the top of NodeReconciler.Reconcile before any rule evaluation takes place.

Related Issue

Fixes #253

Type of Change

/kind bug

Testing

• Added cold-start rule cache warm-up integration test context in node_controller_test.go with two cases:
  • Verifies that a bootstrap-only taint is applied even when ruleCache is empty at reconcile time (the rule is discovered via warm-up).
  • Verifies that the warm-up is not repeated on subsequent reconciles (cacheWarmed remains true).
• All 49 existing tests continue to pass.
• Coverage: internal/controller 73.5% → 73.9%.

Checklist

• make test passes
• make lint passes

Does this PR introduce a user-facing change?

Fix bootstrap-only taint enforcement correctness gap: NodeReconciler now pre-populates the rule cache from the informer cache on its first reconcile, preventing nodes from bypassing the admission gate during the controller startup race window.

[netlify]

✅ Deploy Preview for node-readiness-controller canceled.

Name	Link
🔨 Latest commit	5c83d8f
🔍 Latest deploy log	https://app.netlify.com/projects/node-readiness-controller/deploys/6a05d69674a8410008bd86c5

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Karman580
Once this PR has been reviewed and has the lgtm label, please assign ajaysundark for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @Karman580. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: Karman580 / name: KARMAN SINGH TALWAR (5c83d8f)

[Karman580]

recheck

[Priyankasaggu11929]

Thanks again for writing the gist, @Karman580. I tried to reproduce it locally.

You' are using Ready=True node status condition in your example NRR rule.
In a regular kind cluster setup, the node is (almost always) healthy (i.e, Ready=True is already satisfied) before the controller even starts, and so the taint is never applied in the first place and the cold-cache timing window is never hit.

what you are actually seeing with your repro steps is a different scenario (the default case (case 4) in the switch block) - node is already healthy on first evaluation

IMO, a better reproduction setup would be to use a custom node status condition that kind does not satisfy by default, so, we can actually observe the stated race condition.

I did try do this ^ locally.

While the PR change address the one scenario you described (ie. on a controller restart, (Node Controller) Reconciler will first seed the in-memory rule cache from the informer, before it proceeds with further evaluation on the node, but we still need to address the following gap (which remains regardless of the cache is warm or not)

bootstrap annotation is only added in case 1 of the switch block

  1. shouldRemoveTaint=true  + currentlyHasTaint=true  -> remove taint and add annotation
  2. shouldRemoveTaint=false + currentlyHasTaint=false -> add taint
  3. shouldRemoveTaint=false + currentlyHasTaint=true  -> adopt taint
  4. shouldRemoveTaint=true  + currentlyHasTaint=false -> "No taint action needed" # no annotation

In case of a node which is already healthy on first evaluation (the case that your gist is actually testing with Ready=True node condition), it will always go to the default case, so the annotation is never added.

And what happens then is if that node later has a transient condition failure (e.g. a kubelet restart or network interruption that flips the condition to unknown/False), the controller has no record of its bootstrap stage being completed and it will re-apply the taint and block any workload.

I think we should discuss it in the upcoming project call about whether we should design the default case (shouldRemoveTaint=true + currentlyHasTaint=false) to also write the annotation in bootstrap-only mode.

cc: @ajaysundark

[Karman580]

Thanks for the detailed review and for reproducing it locally.

You're right the current repro using Ready=True overlaps with the default healthy-node path rather than isolating the cache timing race itself. I also see the larger bootstrap annotation gap in the shouldRemoveTaint=true + currentlyHasTaint=false case that you pointed out.

I’ll revisit the reproduction using a custom condition to isolate the cache behavior more clearly, and I agree the bootstrap annotation semantics are probably better discussed separately before I iterate further on the PR.