feat(controller): reconcile dependents when their cluster becomes ready

api/v1alpha1/conditions.go

@@ -84,4 +84,7 @@ const (
 	ReasonClusterNotFound = "ClusterNotFound"
 	// ReasonClusterNotReady indicates the referenced Temporal target is not ready.
 	ReasonClusterNotReady = "ClusterNotReady"
+	// ReasonFrontendUnavailable indicates the Temporal frontend is reachable
+	// but not yet accepting RPCs (transient startup window).
+	ReasonFrontendUnavailable = "FrontendUnavailable"
 )

docs/superpowers/specs/2026-07-24-cluster-readiness-watch-design.md

@@ -0,0 +1,150 @@
+# Reconcile dependent CRs on cluster readiness (watch + backoff fix)
+
+## Problem
+
+When a `TemporalCluster` and a dependent object (e.g. `TemporalNamespace`) are
+created at the same time, the dependent object registers much later than the
+cluster becomes ready. Two mechanics cause this:
+
+1. **No event-driven trigger.** The cluster-dependent controllers
+   (`TemporalNamespace`, `TemporalSchedule`, `TemporalSearchAttribute`,
+   `TemporalWorkflowRun`, `TemporalClusterConnection`) resolve their target
+   cluster via `resolveTarget`, and when the cluster is not `Ready` they simply
+   requeue on a timer. None of them **watch** the `TemporalCluster` /
+   `TemporalDevServer`, so nothing re-triggers a reconcile the instant the
+   cluster flips to `Ready` — the dependent waits for its next timed requeue.
+
+2. **Exponential backoff on transient not-ready.** Once the cluster reports
+   `Ready`, the controllers immediately call the frontend (e.g.
+   `ensureRegistered` → `Describe`/`Register`). If the frontend is up but not
+   yet serving RPCs, the call returns a raw error and the controllers do
+   `return ctrl.Result{}, err`. Controller-runtime then applies **exponential
+   backoff** (default max ~16.7 min), so the object is "caught in a backoff" and
+   registers long after the cluster is actually usable.
+
+This also slows the e2e (chainsaw) suites, which currently sequence
+cluster-first-then-dependent to sidestep the delay.
+
+## Goals
+
+- Dependent CRs reconcile promptly (event-driven) when their referenced cluster
+  becomes `Ready`.
+- Transient "frontend up but not serving yet" conditions never enter
+  exponential backoff.
+- Consistent behavior across all cluster-dependent controllers.
+- No API/CRD/Helm changes (no regen required).
+
+## Non-goals
+
+- No changes to how the `TemporalCluster` computes its own `Ready` condition.
+- No changes to deletion/finalizer semantics.
+- No broad refactor of the controllers beyond what serves this fix.
+
+## Approach
+
+Two shared mechanisms, applied to the five cluster-dependent controllers.
+
+### 1. Event-driven readiness (primary fix)
+
+Each dependent controller's `SetupWithManager` adds:
+
+```go
+.Watches(&temporalv1alpha1.TemporalCluster{},
+    handler.EnqueueRequestsFromMapFunc(mapClusterToX(r.Client)),
+    builder.WithPredicates(clusterReadinessChanged))
+.Watches(&temporalv1alpha1.TemporalDevServer{},
+    handler.EnqueueRequestsFromMapFunc(mapDevServerToX(r.Client)),
+    builder.WithPredicates(clusterReadinessChanged))
+```
+
+- **Map function:** given a changed cluster/devserver, list the dependent CRs in
+  the **same namespace** whose `ClusterRef` points at it, and enqueue reconcile
+  requests for them. For `TemporalClusterConnection`, match when *any* peer's
+  `ClusterRef` targets the changed cluster. An empty `ClusterRef.Kind` defaults
+  to `TemporalCluster` and must be honored by the matcher.
+- **Predicate (`clusterReadinessChanged`):** only enqueue on meaningful changes
+  — a generation change OR a transition of the `Ready` condition. `CreateFunc`
+  returns `true` so an already-`Ready` cluster still triggers dependents created
+  afterward. This avoids fan-out churn on every routine cluster status write.
+
+Result: the moment a cluster flips to `Ready`, its dependents reconcile without
+waiting on a timer.
+
+### 2. No exponential backoff on transient not-ready (Approach A)
+
+A shared helper classifies transient connectivity errors:
+
+```go
+// isTransientClusterErr reports whether err is a transient frontend-not-serving
+// condition (cluster reachable but not yet accepting RPCs) that should be
+// retried on a short fixed interval rather than triggering exponential backoff.
+func isTransientClusterErr(err error) bool {
+    switch status.Code(err) {
+    case codes.Unavailable, codes.DeadlineExceeded, codes.Canceled:
+        return true
+    default:
+        return false
+    }
+}
+```
+
+The four RPC-registering controllers (`TemporalNamespace`, `TemporalSchedule`,
+`TemporalSearchAttribute`, `TemporalWorkflowRun`) wrap their frontend-call error
+sites: if `isTransientClusterErr`, set a `Ready=False` /
+`FrontendUnavailable` condition and return
+`ctrl.Result{RequeueAfter: clusterUnavailableRequeue}` (short, ~5s) with a nil
+error, instead of `return ctrl.Result{}, err`. Terminal errors (invalid TLS
+material, invalid params, permission denied, etc.) still return as real errors
+so they remain visible and retried appropriately.
+
+`TemporalClusterConnection` already swallows transient per-peer failures and
+never returns them as errors, so it receives only the watch change (mechanism 1).
+
+The existing `ClusterNotReady` 15s requeue path is retained as a safety net but
+becomes largely moot once the watch fires immediately.
+
+## Components
+
+New shared file `internal/controller/cluster_watch.go`:
+
+- `clusterReadinessChanged predicate.Predicate` — generation-change or
+  `Ready`-condition-transition filter (compares old/new conditions in
+  `UpdateFunc`; `CreateFunc` → true; `DeleteFunc` → false).
+- Per-controller map-func builders (or a small generic helper) that list the CR
+  type and filter by `ClusterRef` name/kind within the cluster's namespace.
+- `isTransientClusterErr(err) bool` — wraps `google.golang.org/grpc/status` +
+  `codes`.
+- `clusterUnavailableRequeue = 5 * time.Second`.
+
+Per-controller edits:
+
+- `SetupWithManager`: add the two `.Watches(...)` calls with map func +
+  predicate.
+- namespace/schedule/searchattribute/workflowrun: at frontend-call error sites,
+  branch on `isTransientClusterErr` to requeue-short instead of erroring.
+
+## Testing
+
+- **Unit:** `clusterReadinessChanged` (transition vs no-op vs generation bump);
+  map functions (matches only referencing CRs, same-namespace only, honors empty
+  `Kind`); `isTransientClusterErr` classification (Unavailable → true, InvalidArgument → false).
+- **Controller (envtest):** a dependent CR created before its cluster is `Ready`
+  reconciles promptly once the cluster's `Ready` condition flips (watch fires).
+  A transient RPC error yields a short requeue rather than a returned error.
+- **e2e (chainsaw):** update `test/e2e/namespace` to apply the cluster and
+  namespace together in a single step (instead of cluster-first sequencing) and
+  assert the namespace registers, proving the fix and trimming wall-clock time.
+  Keep sibling-suite changes minimal.
+
+## Edge cases
+
+- Both `TemporalCluster` and `TemporalDevServer` target kinds are watched.
+- Cross-namespace safety: the map func only enqueues CRs in the cluster's own
+  namespace, matching `resolveTarget` semantics.
+- `ClusterRef.Kind == ""` defaults to `TemporalCluster`.
+- Deletion/finalizer paths are unchanged.
+
+## Rollout / verification
+
+- No API type changes → no CRD/Helm regeneration expected.
+- Run `make lint` and targeted `make test` before opening the PR.

internal/controller/cluster_watch.go

@@ -0,0 +1,105 @@
+/*
+Copyright 2026 Brian Morton.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package controller
+
+import (
+	"time"
+
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+	"k8s.io/apimachinery/pkg/api/meta"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/event"
+	"sigs.k8s.io/controller-runtime/pkg/predicate"
+
+	temporalv1alpha1 "github.com/bmorton/temporal-operator/api/v1alpha1"
+)
+
+// clusterUnavailableRequeue is the fixed, short delay used when a Temporal
+// frontend is reachable but not yet accepting RPCs. Requeuing on a fixed
+// interval (rather than returning an error) keeps the dependent object off
+// controller-runtime's exponential backoff queue, so it registers promptly once
+// the frontend finishes starting.
+const clusterUnavailableRequeue = 5 * time.Second
+
+// isTransientClusterErr reports whether err is a transient connectivity error
+// from a Temporal frontend that is up but not yet serving RPCs. Such errors
+// should be retried on a short fixed interval rather than triggering
+// exponential backoff. Terminal errors (invalid arguments, permission denied,
+// bad TLS material, etc.) return false so they surface as real reconcile
+// errors. status.Code unwraps fmt-wrapped errors via errors.As.
+func isTransientClusterErr(err error) bool {
+	if err == nil {
+		return false
+	}
+	switch status.Code(err) {
+	case codes.Unavailable, codes.DeadlineExceeded, codes.Canceled:
+		return true
+	default:
+		return false
+	}
+}
+
+// refTargets reports whether ref points at the target named name of the given
+// kind. An empty ref.Kind defaults to TemporalCluster (matching resolveTarget).
+func refTargets(ref temporalv1alpha1.ClusterReference, kind, name string) bool {
+	refKind := ref.Kind
+	if refKind == "" {
+		refKind = temporalv1alpha1.ClusterKindTemporalCluster
+	}
+	return refKind == kind && ref.Name == name
+}
+
+// targetReadyStatus returns the Ready condition status of a watched Temporal
+// target (TemporalCluster or TemporalDevServer), or ConditionUnknown if the
+// object is neither type or has no Ready condition.
+func targetReadyStatus(obj client.Object) metav1.ConditionStatus {
+	var conds []metav1.Condition
+	switch o := obj.(type) {
+	case *temporalv1alpha1.TemporalCluster:
+		conds = o.Status.Conditions
+	case *temporalv1alpha1.TemporalDevServer:
+		conds = o.Status.Conditions
+	default:
+		return metav1.ConditionUnknown
+	}
+	if c := meta.FindStatusCondition(conds, temporalv1alpha1.ConditionReady); c != nil {
+		return c.Status
+	}
+	return metav1.ConditionUnknown
+}
+
+// clusterReadinessChanged limits watch-driven enqueues of dependent CRs to
+// meaningful target changes: a create (so an already-Ready target still triggers
+// dependents created afterward), a generation change, or a transition of the
+// Ready condition. Routine status writes that do not move Ready are ignored to
+// avoid re-reconciling every dependent on each cluster status update.
+var clusterReadinessChanged predicate.Predicate = predicate.Funcs{
+	CreateFunc: func(event.CreateEvent) bool { return true },
+	DeleteFunc: func(event.DeleteEvent) bool { return false },
+	UpdateFunc: func(e event.UpdateEvent) bool {
+		if e.ObjectOld == nil || e.ObjectNew == nil {
+			return false
+		}
+		if e.ObjectOld.GetGeneration() != e.ObjectNew.GetGeneration() {
+			return true
+		}
+		return targetReadyStatus(e.ObjectOld) != targetReadyStatus(e.ObjectNew)
+	},
+	GenericFunc: func(event.GenericEvent) bool { return false },
+}