Resolved
This incident has been resolved.
Investigating
Impact: modules-dashboard.prd.gtpstratus.com intermittently returning 503 ("Micro-Frontend Unavailable" in the Stratus host) since ~18:35 UTC. Currently sustained, not self-resolving after 15+ min of monitoring.
Root cause chain (working theory, not yet fully confirmed):
A burst of unrelated Stratus core job workload (stratusjobs/stratus/stratusapi namespaces — importbundle, sendcutlist, runpdfreport, etc.) triggered the cluster autoscaler to rapidly add nodes to the shared AKS node pools (aks-jobsazlnx03, aks-primarylnx03) that Modules also runs on.
modules-dashboard and one Valkey pod (valkey-0) got rescheduled onto newly-joined nodes during the churn.
Istio/CNI routing convergence for pods on brand-new nodes appears to be taking longer than expected, causing "no route to host" both on modules-dashboard's ingress path and on modules-backend's connections to Valkey (port 6379, PassthroughCluster — 15-20 failed connection attempts per 25s window, sustained).
Side effect: metrics-server was also disrupted mid-restart during the churn, causing unrelated HPA "FailedGetResourceMetric" noise across the cluster (istiod, istio-ingressgateway, App Config) — cosmetic, not part of the outage.
Ruled out: this does NOT look like the Valkey Sentinel stale-ghost-entry bug fixed yesterday (PLAT-1879/PR #548) — that was permanently-dead entries never pruned; this is fresh/transient unreachability tied to active pod rescheduling.
Separate, likely unrelated minor issue: modules-backend logged real 400s on assembly-tracking-status and package-tracking-status around 18:35, reported by a user updating a package's tracking status in the Viewer. Worth investigating on its own but doesn't explain the dashboard/Valkey connectivity failures.
Status as of declare: still degraded — dashboard failing ~2/3 checks, Valkey still throwing ~20 connection errors per 25s window. Not converging on its own.
