ADR: EKS Provisioned Control Plane and 40x HPA Concurrency

The Kubernetes Horizontal Pod Autoscaler operates on a control loop. Every cycle — by default every 15 seconds — the controller manager evaluates all HPA objects in the cluster, queries the metrics server or an external metrics adapter, calculates the desired replica count, and issues an update on the Deployment or StatefulSet object. What most teams miss is that this loop is serially bounded by the --horizontal-pod-autoscaler-sync-period parameter combined with evaluation concurrency. In standard upstream Kubernetes, that concurrency is intentionally low — the project assumes a shared control plane where concurrent workloads could saturate the API server.

In the financial environments I operate, it is common to have between 200 and 800 active HPA objects in a single production cluster: quote services, risk engines, onboarding APIs, market event processing workers. Each of those objects needs to be evaluated within the sync window. When load arrives — a volume spike at market open, a news event triggering simultaneous requests — the control plane must process all those objects before any scaling happens. With low concurrency and 400 HPA objects, you may be waiting through multiple 15-second cycles before the critical workload is evaluated. This translates to scaling latency of 45 to 90 seconds in realistic scenarios — enough time for an availability SLO to be violated.

The architectural force here is clear: autoscaler reaction speed is a component of the availability SLO, not just an infrastructure metric. Teams that define SLOs of p99 < 500ms for trading APIs need to include scaling latency in their error budget — and most do not.

The upstream kube-controller-manager exposes the flag --concurrent-horizontal-pod-autoscaler-syncs, which controls how many parallel goroutines process HPA objects simultaneously. The historical default value is 5. This means that in a cluster with 500 HPA objects, the controller needs at least 100 serial goroutine iterations to complete a full evaluation cycle. If each evaluation takes ~100ms (including metrics server call, calculation, and write to the API server), a full cycle takes ~10 seconds — already consuming 67% of the default 15-second sync window.

EKS Provisioned Control Plane, per the July 28, 2026 announcement, increases this concurrency to up to 40x the default value. Mathematically, this means the same 500-HPA-object cluster can be evaluated in approximately 2.5 goroutine iterations instead of 100 — reducing cycle time from ~10 seconds to sub-second under normal conditions. The practical effect is that the time between load detection and the start of scaling collapses from multiple cycles to a fraction of the first cycle.

It is important to understand that Provisioned Control Plane is AWS's premium offering where you get a dedicated, non-shared control plane. This is what makes it viable to increase concurrency without impacting other tenants — the underlying API server has reserved capacity to absorb the additional write and read throughput that concurrent evaluation generates. On a standard shared control plane, aggressively increasing this concurrency could generate API server throttling and worsen the situation. Here, the AWS architectural decision is coherent: the benefit is only safe because resource isolation already exists.

The decision to adopt EKS Provisioned Control Plane is not motivated solely by the HPA concurrency improvement announced in July 2026. It is motivated by an architectural posture shift: the Kubernetes control plane is critical data infrastructure, not a convenience service. In financial environments, the control plane determines the system's reaction time to market events. Treating it as a shared commodity resource is the same as sharing the data bus of a trading system — it works until it does not.

Provisioned Control Plane offers dedicated API server capacity, which means the additional throughput generated by 40x more HPA evaluation goroutines does not compete with other tenants. This is fundamental: increasing concurrency without resource isolation is a recipe for cascading degradation. The Kubernetes API server has configurable rate limiting limits (--max-requests-inflight, --max-mutating-requests-inflight), and on a shared control plane, parallel evaluation of 200 HPA objects can easily saturate these limits, generating 429 Too Many Requests and delaying not just autoscaling, but also deployment operations, health checks, and reconciliation of other controllers.

The decision is also consistent with the AWS platform trajectory. EKS Auto Mode — which includes Karpenter with 43% faster node scale-out and 69% faster consolidation — runs on Provisioned Control Plane. AWS is clearly building its highest-value capabilities on this layer. Teams that have not migrated to it are, effectively, accumulating platform technical debt.

For clusters with fewer than 50 HPA objects and workloads without defined scaling latency SLOs, the standard tier may be sufficient. But that is an exception, not the rule in financial production environments.

The first operational consequence is positive and immediate: existing clusters on Provisioned Control Plane receive the benefit without any action. This is rare in infrastructure improvements and deserves recognition — AWS absorbed the tuning complexity into the managed service.

The second consequence is more subtle and potentially problematic: workloads that were sized based on slow HPA behavior may now scale more aggressively than expected. Imagine a service that had scaleUp.stabilizationWindowSeconds configured at 120 seconds precisely because the team knew the HPA was slow to react and wanted to avoid flapping. With the new concurrency, the HPA reacts in seconds — and the stabilization window may no longer be sufficient to absorb transient metric spikes. The result could be unnecessary scale-up followed by scale-down, generating additional cost and potential instability.

The third consequence affects observability. If you monitor kube_horizontalpodautoscaler_status_current_replicas versus kube_horizontalpodautoscaler_spec_max_replicas as a proxy for scaling pressure, the time series behavior will change. What was previously a gradual upward curve (because the HPA took time to react) may now be a near-instantaneous step function. Capacity planning dashboards that depend on the slope of that curve to predict node needs must be recalibrated.

Finally, there is the interaction with Karpenter/EKS Auto Mode. The faster HPA will generate Pending pods more quickly, which triggers Karpenter to provision nodes. With Karpenter 43% faster (June 2026 announcement), the complete pod-scale → node-provision chain now operates within a much smaller time window. This is excellent for SLOs — but it means the Karpenter NodePool needs well-defined maximum capacity limits to avoid excessive provisioning in runaway scaling scenarios.

Migration to Provisioned Control Plane with 40x HPA concurrency requires a review of the observability model. The set of metrics you need to monitor changes in character, not just in values.

Control plane metrics that become more important: apiserver_request_duration_seconds for patch and update verbs in the autoscaling/v2 group now reflects actual HPA throughput. On a dedicated control plane, you should see latencies consistently below 50ms for these requests. If you observe spikes above 200ms, this is a signal that the metrics server is the bottleneck, not the HPA controller. Instrument metrics_server_api_metric_freshness_seconds to ensure metrics feeding the HPA are fresh enough — with faster HPA, metrics with 30-60 second staleness become a real limiter.

Scaling latency SLO: Explicitly define an SLO for scaling time. A useful composite metric is: time_from_metric_threshold_breach_to_first_new_pod_ready. This requires correlation between HPA events (kube_horizontalpodautoscaler_status_current_replicas increasing), pod creation (kube_pod_created), and readiness (kube_pod_status_ready). With OpenTelemetry, you can create a synthetic span covering this complete pipeline — from breach to pod ready — and expose it as a measurable SLI.

Runaway scaling alerts: Configure an alert in CloudWatch or Datadog for kube_horizontalpodautoscaler_status_current_replicas / kube_horizontalpodautoscaler_spec_max_replicas > 0.85 with a 2-minute window. This detects when a workload is rapidly approaching the replica ceiling — a situation that, with slower HPA, took enough time for manual intervention, but can now happen before any traditional latency alert fires.

Cost observability: Enable AWS Cost Anomaly Detection with a monitor specific to the cluster tag. Faster scale-up means EC2/Fargate cost spikes also happen faster. The cost error budget needs to be recalibrated alongside the availability error budget.