[gnoi-shutdown]: Use host GnoiClient for DPU shutdown

[dlanov]

Why I did it

Fixes sonic-net/sonic-buildimage#29188

gnoi-shutdown currently shells into the gnmi container to run gnoi_client for DPU shutdown. This couples the host shutdown daemon to gnmi.service and can leave the DPU graceful-shutdown path unavailable when gnmi is stopped or unavailable during service lifecycle operations such as config reload.

How I did it

Use the shared gNOI client and System stubs from sonic-py-common directly on the host:

• Import GnoiClient and system_pb2 from sonic_grpc.gnoi.
• Fetch the DPU's ephemeral TLS certificate with ssl.get_server_certificate() and pin it as root_certificates for the secure gRPC channel.
• Probe the configured gNMI port, 8080, and 50052 in order with System.Time, without probing duplicates.
• Bound certificate fetch plus System.Time to one 10-second budget per candidate port.
• Send one System.Reboot request with HALT on the selected port.
• Poll System.RebootStatus on the same port, bounded by the configured DPU halt timeout. Treat active=false with either no status field or STATUS_SUCCESS as complete.
• Remove gnmi.service from Wants= and After= in gnoi-shutdown.service.
• Keep Requires=database.service, pmon.service ordering, and WantedBy=multi-user.target; do not tie the daemon to sonic.target.

How to verify it

On a SmartSwitch NPU:

1. Confirm the daemon is active:

   systemctl is-active gnoi-shutdown

2. Run a configuration reload:

   sudo config reload -y

3. Confirm gnoi-shutdown remains available independently of gnmi.service.
4. Run a graceful DPU shutdown:

   sudo config chassis module shutdown DPU1

5. Confirm the host daemon probes the DPU directly, sends a single HALT request, and does not wait for the full halt-services timeout when RebootStatus reports completion.

Local verification

• git diff --check passes.
• python3 -m py_compile passes for the daemon and its test file.
• systemd-analyze verify passes against the modified unit with dependency stubs.
• Offline systemctl enable validation creates multi-user.target.wants/gnoi-shutdown.service.
• Integration-style smoke validation using the shared GnoiClient with a real TLS gRPC fake server covers TLS certificate pinning, port fallback, one Reboot(HALT), RebootStatus handling, and deadline-bounded polling.
• The full SmartSwitch hardware scenario requires a suitable SONiC SmartSwitch test environment.

Description for the changelog

Call DPU gNOI directly from the host shutdown daemon and remove its dependency on gnmi.service.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

Hi, there are workflow run(s) waiting for approval, you may be first-time contributor. I will notify maintainers to help approve once PR is approved. Thanks!

_{---Powered by SONiC BuildBot}

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[hdwhdw]

The direct host-client and systemd direction is correct. However, the current Transport Layer Security (TLS) implementation cannot connect to a data processing unit (DPU) that uses the default SONiC ephemeral certificate. Please resolve both blocking TLS issues below.

Can you run an end-to-end test between a SmartSwitch network processing unit (NPU) and a DPU using the default ephemeral TLS configuration?

• Run it immediately after DPU telemetry starts.
• Repeat it after telemetry has run for more than one hour.
• Verify System.Time, one Reboot(HALT), RebootStatus, and halt-flag clearing.
• Verify that the workflow succeeds while gnmi.service is stopped.

The checked-in tests mock TLS credential and channel creation. Passing continuous integration does not validate the production TLS handshake.

[dlanov]

Thanks, It's confirmed. We reproduced both TLS failures with real gNOI/TLS handshakes:

• The default SONiC ephemeral certificate uses SAN example.com, so the current client fails when connecting by DPU IP.
• ssl_target_name_override fixes the hostname mismatch, but not the second issue.
• The same certificate is valid for only one hour, so after that grpc-python rejects it as expired.
• gRPC Core supports encrypted TLS without server-certificate verification, matching the policy used by the Go DPU proxy.
• grpc-python does not currently expose that Core option through its Python API.

Before changing #427 again, would you prefer that we pursue:

• a shared gRPC credential path exposing the existing Core capability, or
• a shared sonic-py-common helper for this DPU TLS policy?

[hdwhdw]

These options address different layers.

Matching Go’s InsecureSkipVerify requires a Python binding for gRPC Core’s custom certificate verifier. This is a generic, security-sensitive grpcio capability. A sonic-py-common helper cannot provide it through the current public Python API.

sonic-py-common could centralize DPU endpoint configuration, leaf certificate retrieval, and credential creation. However, standard grpc-python verification still checks SANs and expiration. The sonic-gnmi --insecure server path calls testdata/tls.NewCert() in production. That certificate uses example.com as its SAN and expires after one hour. Wrapping ssl.get_server_certificate() does not address this server defect.

I recommend fixing the server contract rather than adding a generic verification-bypass API:

1. Add a DPU-specific ephemeral TLS mode to sonic-gnmi. Keep it separate from generic --insecure.
2. Generate a production certificate with SANs for the dynamically discovered listener and interface identities.
3. Issue the certificate for the supported uptime, or rotate it before expiration.
4. Update the sonic-buildimage DPU launcher to use this mode.
5. Optionally add a shared sonic_grpc helper that pins the ephemeral leaf certificate and creates a normally verified channel. PR [gnoi-shutdown]: Use host GnoiClient for DPU shutdown #427 can then use this helper.

This removes the production dependency on a test certificate. It preserves encrypted host-to-DPU gNOI without introducing a broad verification-bypass API.

I am happy to review the sonic-gnmi and sonic-buildimage follow-up PRs.

[dlanov]

Thanks, understood. This appears to require prerequisite changes in sonic-gnmi and sonic-buildimage. Are those changes already planned or owned by someone, and should #427 remain on hold until they are available?

[hdwhdw]

@dlanov, do you mind owning the prerequisite changes in sonic-gnmi and sonic-buildimage? This work has no current owner.

Scope:

• Add a DPU-specific ephemeral TLS mode in sonic-gnmi. Keep generic --insecure unchanged.
• Replace testdata/tls.NewCert() for this mode. Use dynamically discovered SANs and a supported certificate lifetime or rotation. Do not hard-code a midplane IP.
• Update the sonic-buildimage DPU launcher to select this mode.
• Update [gnoi-shutdown]: Use host GnoiClient for DPU shutdown #427 to use normal grpc-python verification after both prerequisite changes.

I can review both prerequisite PRs.

[hdwhdw]

By the way, I opened the two prerequisite PRs for this approach:

• telemetry: generate DPU ephemeral TLS certificates sonic-gnmi#772 adds the DPU-specific ephemeral TLS certificate mode.
• docker-sonic-gnmi: select DPU ephemeral TLS mode sonic-buildimage#29238 selects this mode in the DPU launcher while supporting older telemetry binaries.

All required checks pass. @dlanov, please take a look when you have time. If the direction looks right, I would appreciate your approval.

These changes let #427 use standard grpc-python verification without the generic skip-verify path.

[hdwhdw]

I verified the non-destructive path:

• sonic-net/sonic-gnmi#772 generated an ephemeral certificate with the DPU IP in its SAN.
• #427 fetched and pinned the certificate. System.Time succeeded directly from the NPU host to the DPU IP.
• System.Time also succeeded without gnmi.service running.

This test covered TLS setup and the direct System.Time RPC path. I did not send Reboot(HALT).

[dlanov]

Thanks for opening and validating these. I don’t have maintainer approval rights in these repositories, but I can review the changes from the #427 integration perspective and leave feedback. Formal approval will still need to come from a repository maintainer.