OCPBUGS-77254: fix(globalps): watch Machine updates to fix node labeling race condition by judexzhu · Pull Request #7803 · openshift/hypershift

judexzhu · 2026-02-25T18:31:50Z

Summary

Fixes a ~15-minute delay in scheduling the global-pull-secret-syncer DaemonSet pod on newly created nodes
Adds a watch on capiv1.Machine updates on the management cluster, triggering reconciliation when Machine.Status.NodeRef transitions from nil to non-nil
Adds a unit test verifying that nodes with nil Machine.Status.NodeRef are correctly skipped during labeling

Problem

The globalps controller had a race condition: when a new Node was created on the hosted cluster, the controller received the Node CREATE event and ran labelNodesForGlobalPullSecret(). However, the CAPI Machine controller on the management cluster had not yet set Machine.Status.NodeRef, so the new node could not be mapped to its Machine and was skipped for labeling with hypershift.openshift.io/nodepool-globalps-enabled: "true".

With no Machine watch, no Node update handling, and no RequeueAfter, the node remained unlabeled until an unrelated event (e.g., a kube-system Secret change) triggered a reconcile -- often ~15 minutes later.

Solution

Watch capiv1.Machine on the CP cluster cache with a predicate that only fires when Status.NodeRef transitions from nil to non-nil. This ensures the controller reconciles at the exact moment CAPI links the Machine to the Node, so labelNodesForGlobalPullSecret can label the node and the DaemonSet pod is scheduled promptly.

Test plan

New unit test: "When Machine.Status.NodeRef is nil it should skip the node for labeling"
All existing globalps unit tests pass with race detection
make lint-fix passes with 0 issues

Fixes

OCPBUGS-77254

Made with Cursor

Summary by CodeRabbit

Tests
- Added unit tests covering node-labeling behavior and event predicates, including cases where machines lack node references and when node references appear.
Improvements
- Controller now watches machine state changes and reconciles when a machine becomes linked to a node, ensuring timely labeling and consistent cluster state.

openshift-ci-robot · 2026-02-25T18:31:53Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

openshift-ci-robot · 2026-02-25T18:31:58Z

@judexzhu: This pull request references Jira Issue OCPBUGS-77254, which is invalid:

expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Fixes a ~15-minute delay in scheduling the global-pull-secret-syncer DaemonSet pod on newly created nodes

Adds a watch on capiv1.Machine updates on the management cluster, triggering reconciliation when Machine.Status.NodeRef transitions from nil to non-nil

Adds a unit test verifying that nodes with nil Machine.Status.NodeRef are correctly skipped during labeling

Problem

The globalps controller had a race condition: when a new Node was created on the hosted cluster, the controller received the Node CREATE event and ran labelNodesForGlobalPullSecret(). However, the CAPI Machine controller on the management cluster had not yet set Machine.Status.NodeRef, so the new node could not be mapped to its Machine and was skipped for labeling with hypershift.openshift.io/nodepool-globalps-enabled: "true".

With no Machine watch, no Node update handling, and no RequeueAfter, the node remained unlabeled until an unrelated event (e.g., a kube-system Secret change) triggered a reconcile -- often ~15 minutes later.

Solution

Watch capiv1.Machine on the CP cluster cache with a predicate that only fires when Status.NodeRef transitions from nil to non-nil. This ensures the controller reconciles at the exact moment CAPI links the Machine to the Node, so labelNodesForGlobalPullSecret can label the node and the DaemonSet pod is scheduled promptly.

Test plan

New unit test: "When Machine.Status.NodeRef is nil it should skip the node for labeling"

All existing globalps unit tests pass with race detection

make lint-fix passes with 0 issues

Fixes

OCPBUGS-77254

Made with Cursor

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-02-25T18:34:17Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 13ac3de8-ae08-45c9-bd46-9c236da3e127

📥 Commits

Reviewing files that changed from the base of the PR and between 0c997f5 and b7bf8e6.

📒 Files selected for processing (2)

control-plane-operator/hostedclusterconfigoperator/controllers/globalps/globalps_test.go
control-plane-operator/hostedclusterconfigoperator/controllers/globalps/setup.go

🚧 Files skipped from review as they are similar to previous changes (1)

control-plane-operator/hostedclusterconfigoperator/controllers/globalps/globalps_test.go

Walkthrough

Adds a watch on Cluster API Machines that enqueues reconciliation when a Machine's Status.NodeRef transitions from nil to non-nil, and expands unit tests to cover Machine NodeRef update predicates and skipping machines with nil NodeRef.

Changes

Cohort / File(s)	Summary
Machine watch & predicate `control-plane-operator/hostedclusterconfigoperator/controllers/globalps/setup.go`	Adds a watch on CAPI `Machine` objects in the management cluster; enqueues reconciles for the Machine's NamespacedName when `Status.NodeRef` changes from nil to non-nil. Introduces `machineNodeRefPredicate` and imports `capiv1`.
Predicate & labeling tests `control-plane-operator/hostedclusterconfigoperator/controllers/globalps/globalps_test.go`	Adds `TestMachineNodeRefPredicate` with subtests for create/delete/generic/update scenarios and extends `TestLabelNodesForGlobalPullSecret` to verify machines with nil `NodeRef` are skipped during labeling. Imports controller-runtime `event` for predicate tests.

Sequence Diagram(s)

sequenceDiagram
    participant Machine as Machine (CAPI)
    participant APIServer as API Server
    participant Controller as globalps Controller
    participant Node as Node

    rect rgba(200,200,255,0.5)
    Machine->>APIServer: Update Status.NodeRef (nil -> non-nil)
    end

    rect rgba(200,255,200,0.5)
    APIServer->>Controller: Watch triggers (mapped enqueue)
    Controller->>APIServer: Request Machine by NamespacedName
    Controller->>Node: Label node referenced by Machine.Status.NodeRef
    Node-->>Controller: Confirm labels applied
    end

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Structure And Quality	⚠️ Warning	TestMachineNodeRefPredicate violates single responsibility, lacks assertion messages, and uses inconsistent testing style compared to existing Ginkgo tests.	Refactor to use Ginkgo patterns with Gomega assertions and meaningful failure messages; break into focused test blocks aligned with repository testing conventions.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: watching Machine updates to fix a node labeling race condition in the globalps controller.
Stable And Deterministic Test Names	✅ Passed	Test names are static and descriptive without dynamic information like timestamps, UUIDs, generated suffixes, or node/pod names that could change between runs.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Tip

Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs).
Share your feedback on Discord.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

enxebre · 2026-02-26T10:00:18Z

why are we labeling machines and let it propagate? why don't label nodes directly?
edit:
why do we need labelNodesForGlobalPullSecret at all? we can propagate this label down at machineDeployment reconciliation.

judexzhu · 2026-02-26T16:30:11Z

Hi @enxebre . I have no context why labelNodesForGlobalPullSecret there at the first place, I think this PR is a simple and safe bug fix for the currently architect design.

I think your idea is a better long-term design, (correct me if I understand wrong), when nodepool created, NodePool controller check if the upgrade Replace strategy, if true, directly add the label into the node template.

However this looks might need update changes the labeling mechanism cross multiple controller/operators and touches code paths used by every NodePool. I think if could, we could open another story for follow up architecture design improvement

csrwng · 2026-03-04T20:21:53Z

/approve

openshift-ci · 2026-03-04T20:22:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csrwng, judexzhu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [csrwng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

csrwng · 2026-03-04T20:22:47Z

/assign @jparrill

jparrill

Thanks for the PR!. I've Dropped some comments.

The globalps controller had a race condition where the Node CREATE event fired before CAPI set Machine.Status.NodeRef on the management cluster. This caused labelNodesForGlobalPullSecret to skip the new node because it could not map the Machine to the Node. With no retry mechanism, the node remained unlabeled until an unrelated event triggered a reconcile, resulting in a ~15-minute delay before the global-pull-secret-syncer DaemonSet pod was scheduled. Add a watch on capiv1.Machine via the CP cluster cache that triggers reconciliation when Machine.Status.NodeRef transitions from nil to non-nil, ensuring the node is labeled promptly. Fixes: https://issues.redhat.com/browse/OCPBUGS-77254 Signed-off-by: Jude Zhu <judzhu@redhat.com> Assisted-by: claude-4.6-opus (via Cursor) Co-authored-by: Cursor <cursoragent@cursor.com>

openshift-ci-robot · 2026-03-05T16:33:58Z

@judexzhu: This pull request references Jira Issue OCPBUGS-77254, which is invalid:

expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2026-03-05T16:44:00Z

@judexzhu: This pull request references Jira Issue OCPBUGS-77254, which is invalid:

expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

judexzhu · 2026-03-05T16:45:39Z

/jira refresh

openshift-ci-robot · 2026-03-05T16:45:47Z

@judexzhu: This pull request references Jira Issue OCPBUGS-77254, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.22.0) matches configured target version for branch (4.22.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

jparrill · 2026-03-05T17:06:11Z

/lgtm

openshift-ci-robot · 2026-03-05T17:06:26Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws

judexzhu · 2026-03-05T20:54:02Z

/retest-required

gaol · 2026-03-06T02:08:45Z

/test e2e-aks

gaol · 2026-03-06T14:56:47Z

/test e2e-aks

gaol · 2026-03-06T15:00:27Z

/verified by @gaol

openshift-ci-robot · 2026-03-06T15:00:41Z

@gaol: This PR has been marked as verified by @gaol.

Details

In response to this:

/verified by @gaol

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2026-03-06T17:57:23Z

@judexzhu: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot · 2026-03-06T18:02:21Z

@judexzhu: Jira Issue Verification Checks: Jira Issue OCPBUGS-77254
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-77254 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

Summary

Fixes a ~15-minute delay in scheduling the global-pull-secret-syncer DaemonSet pod on newly created nodes

Adds a watch on capiv1.Machine updates on the management cluster, triggering reconciliation when Machine.Status.NodeRef transitions from nil to non-nil

Adds a unit test verifying that nodes with nil Machine.Status.NodeRef are correctly skipped during labeling

Problem

The globalps controller had a race condition: when a new Node was created on the hosted cluster, the controller received the Node CREATE event and ran labelNodesForGlobalPullSecret(). However, the CAPI Machine controller on the management cluster had not yet set Machine.Status.NodeRef, so the new node could not be mapped to its Machine and was skipped for labeling with hypershift.openshift.io/nodepool-globalps-enabled: "true".

With no Machine watch, no Node update handling, and no RequeueAfter, the node remained unlabeled until an unrelated event (e.g., a kube-system Secret change) triggered a reconcile -- often ~15 minutes later.

Solution

Watch capiv1.Machine on the CP cluster cache with a predicate that only fires when Status.NodeRef transitions from nil to non-nil. This ensures the controller reconciles at the exact moment CAPI links the Machine to the Node, so labelNodesForGlobalPullSecret can label the node and the DaemonSet pod is scheduled promptly.

Test plan

New unit test: "When Machine.Status.NodeRef is nil it should skip the node for labeling"

All existing globalps unit tests pass with race detection

make lint-fix passes with 0 issues

Fixes

OCPBUGS-77254

Made with Cursor

Summary by CodeRabbit

Tests

Added unit tests covering node-labeling behavior and event predicates, including cases where machines lack node references and when node references appear.

Improvements

Controller now watches machine state changes and reconciles when a machine becomes linked to a node, ensuring timely labeling and consistent cluster state.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-merge-robot · 2026-03-07T16:12:01Z

Fix included in accepted release 4.22.0-0.nightly-2026-03-07-091925

gaol · 2026-03-09T05:42:50Z

/jira backport release-4.21, release-4.20

openshift-ci-robot · 2026-03-09T05:45:14Z

@gaol: The following backport issues have been created:

OCPBUGS-77966 for branch release-4.21
OCPBUGS-77967 for branch release-4.20

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.21
/cherrypick release-4.20

Details

In response to this:

/jira backport release-4.21, release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-cherrypick-robot · 2026-03-09T05:45:59Z

@openshift-ci-robot: new pull request created: #7882

Details

In response to this:

@gaol: The following backport issues have been created:

OCPBUGS-77966 for branch release-4.21

OCPBUGS-77967 for branch release-4.20

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.21
/cherrypick release-4.20

In response to this:

/jira backport release-4.21, release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-cherrypick-robot · 2026-03-09T05:46:43Z

@openshift-ci-robot: new pull request created: #7883

Details

In response to this:

@gaol: The following backport issues have been created:

OCPBUGS-77966 for branch release-4.21

OCPBUGS-77967 for branch release-4.20

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.21
/cherrypick release-4.20

In response to this:

/jira backport release-4.21, release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Feb 25, 2026

openshift-ci Bot added do-not-merge/needs-area area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release labels Feb 25, 2026

openshift-ci Bot requested review from csrwng and jparrill February 25, 2026 18:32

openshift-ci Bot removed the do-not-merge/needs-area label Feb 25, 2026

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 4, 2026

openshift-ci Bot assigned jparrill Mar 4, 2026

jparrill reviewed Mar 5, 2026

View reviewed changes

judexzhu force-pushed the OCPBUGS-77254 branch from 0c997f5 to b7bf8e6 Compare March 5, 2026 16:33

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Mar 5, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Mar 5, 2026

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Mar 6, 2026

openshift-merge-bot Bot merged commit 6c29416 into openshift:main Mar 6, 2026
23 checks passed

openshift-cherrypick-robot mentioned this pull request Mar 9, 2026

[release-4.20] OCPBUGS-77967: fix(globalps): watch Machine updates to fix node labeling race condition #7882

Closed

openshift-cherrypick-robot mentioned this pull request Mar 9, 2026

[release-4.21] OCPBUGS-77966: fix(globalps): watch Machine updates to fix node labeling race condition #7883

Closed

Conversation

judexzhu commented Feb 25, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Solution

Test plan

Fixes

Summary by CodeRabbit

Uh oh!

openshift-ci-robot commented Feb 25, 2026

Uh oh!

openshift-ci-robot commented Feb 25, 2026

Summary

Problem

Solution

Test plan

Fixes

Uh oh!

coderabbitai Bot commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

❌ Failed checks (2 warnings)

Uh oh!

enxebre commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

judexzhu commented Feb 26, 2026

Uh oh!

csrwng commented Mar 4, 2026

Uh oh!

openshift-ci Bot commented Mar 4, 2026

Uh oh!

csrwng commented Mar 4, 2026

Uh oh!

jparrill left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

openshift-ci-robot commented Mar 5, 2026

Uh oh!

openshift-ci-robot commented Mar 5, 2026

Uh oh!

judexzhu commented Mar 5, 2026

Uh oh!

openshift-ci-robot commented Mar 5, 2026

Uh oh!

jparrill commented Mar 5, 2026

Uh oh!

openshift-ci-robot commented Mar 5, 2026

Uh oh!

judexzhu commented Mar 5, 2026

Uh oh!

gaol commented Mar 6, 2026

Uh oh!

gaol commented Mar 6, 2026

Uh oh!

gaol commented Mar 6, 2026

Uh oh!

openshift-ci-robot commented Mar 6, 2026

Uh oh!

openshift-ci Bot commented Mar 6, 2026

Uh oh!

Uh oh!

openshift-ci-robot commented Mar 6, 2026

Summary

Problem

Solution

Test plan

Fixes

Summary by CodeRabbit

Uh oh!

openshift-merge-robot commented Mar 7, 2026

Uh oh!

judexzhu commented Feb 25, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Feb 25, 2026 •

edited

Loading

enxebre commented Feb 26, 2026 •

edited

Loading