ubuntu-noble warden stemcell: BOSH DNS runtime config needs recursor adjustment for BOSH Lite on GCP

[hamideh-amini-khezrabad]

When using the ubuntu-noble warden stemcell on BOSH Lite (GCP), DNS resolution breaks because the current BOSH DNS runtime config sets "disable_recursors: true" for ubuntu-noble.

A similar dedicated DNS runtime config adjustment needs to be added for the warden stemcell with:

disable_recursors: false
recursors:

• 169.254.169.254

169.254.169.254 is GCP's internal metadata server required for DNS resolution on BOSH Lite.

See also: 2b7f09f

[aramprice]

I asked Gemini to weigh in on this and got the following response (I have NOT validated this):

---

To enable access to the GCP metadata server (169.254.169.254) on Ubuntu systems, two primary systemd components work together:

1. systemd-networkd.service: This is the core unit that enables network routing to the metadata IP. On GCP, the route to the metadata server is provided by the DHCP server via DHCP Option 121 (classless static routes). systemd-networkd must be running and configured to accept DHCP routes (UseRoutes=true) to install the 169.254.169.254/32 route into the kernel routing table.
2. google-guest-agent.service (and historically google-network-daemon.service): Provided by the google-guest-agent package, this systemd unit actively manages the guest environment by polling 169.254.169.254 for metadata changes, setting up OS Login, and ensuring internal DNS routing (via gce-resolved.conf) is pointed at the metadata server.

If instances deployed from your stemcell are timing out when attempting to reach the metadata server, it is usually because systemd-networkd is misconfigured to ignore the DHCP host route, or the GCP guest agent services are masked/disabled.

How to update bosh-linux-stemcell-builder

To ensure these units are present, installed, and enabled on GCP stemcells, you must modify the build stages in the cloudfoundry/bosh-linux-stemcell-builder repository. The stemcell builder uses "stages" (shell scripts that run inside a chroot of the OS image) to configure the OS.

Here is how you would update the repository:

1. Update/Verify the GCP Package Stage

Navigate to the stage that installs the Google Guest environment (typically located in a path like stemcell_builder/stages/google_gce_packages/apply.sh or stemcell_builder/stages/google_guest_agent/apply.sh).

Ensure the google-guest-agent package is being pulled via apt-get and that the systemd unit is explicitly enabled inside the chroot:

#!/usr/bin/env bash
set -e

# Add GCP package repositories if not already present
# ... (standard apt repo setup) ...

# Install the guest agent packages
run_in_chroot $chroot "apt-get update"
run_in_chroot $chroot "apt-get install -y google-guest-agent"

# Explicitly enable the systemd unit to ensure it runs on boot
run_in_chroot $chroot "systemctl enable google-guest-agent.service"

2. Ensure DHCP Routes are Accepted in systemd-networkd

If your stemcell overrides the default systemd network configurations (usually managed in a networking stage like stemcell_builder/stages/system_network/apply.sh or stemcell_builder/stages/bosh_network/apply.sh), ensure that the .network file generated for the primary interface does not disable DHCP routes.

If generating an /etc/systemd/network/ file, ensure the [DHCP] section accepts routes:

[DHCPv4]
UseRoutes=true

(Note: UseRoutes=true is the default behavior in systemd, so you primarily need to ensure no stage is setting it to false.)

3. (Optional) Hardcode the Metadata Route as a Fallback

If BOSH networking completely replaces DHCP with static IP configurations via the BOSH agent, the DHCP Option 121 will not be processed. In this case, you must ensure the network builder scripts add a static route for 169.254.169.254.

You can modify the network setup scripts inside the stemcell builder to drop a static route config into /etc/systemd/network/:

# Inside the relevant apply.sh stage
cat <<EOF > $chroot/etc/systemd/network/10-gcp-metadata.network
[Match]
Name=e*

[Route]
Destination=169.254.169.254/32
Scope=link
EOF

4. Rebuild the Stemcell

After updating these stages, you can test the stemcell build locally by running the Rake tasks provided in the repository:

bundle exec rake stemcell:build_with_local_os_image[google,kvm,ubuntu,jammy,${PWD}/tmp/ubuntu_base_image.tgz]

---

[beyhan]

For context this is a finding made in the CF Deployment WG during the Noble validation. More details in this Slack message.

[aramprice]

Apparently there's a package for GCP VM's that we could be adding which would provide this internal DNS as a systemd unit for DNS. Some slop from Gemini:

---

Ensure the google-guest-agent package is being pulled via apt-get and that the systemd unit is explicitly enabled inside the chroot:

#!/usr/bin/env bash
set -e

# Add GCP package repositories if not already present
# ... (standard apt repo setup) ...

# Install the guest agent packages
run_in_chroot $chroot "apt-get update"
run_in_chroot $chroot "apt-get install -y google-guest-agent"

# Explicitly enable the systemd unit to ensure it runs on boot
run_in_chroot $chroot "systemctl enable google-guest-agent.service"

---

[hamideh-amini-khezrabad]

@aramprice is there any update on this issue?

[aramprice]

@hamideh-amini-khezrabad I haven't had time to look at what changes can / should be made in https://github.com/cloudfoundry/bosh-linux-stemcell-builder/ to address this.

It will probably be at least a week or two before I'm able to poke at it so if anyone else feels like taking a look that would be grand!

[beyhan]

@Ivaylogi98, @neddp could you help with this?