# Training and inference on one GPU fleet: four ways to share 10,000 accelerators

China Merchants Bank won the CNCF End User Case Study Contest by putting training, fine-tuning and inference on a single Kubernetes control plane over nearly 10,000 heterogeneous accelerators: average utilization from 35% to over 60% and cost per million tokens cut by more than 60%. I break down what makes that number happen — quota-aware queueing, metric-driven autoscaling, GPU virtualization and data caching — and compare four ways to reproduce it on AWS, with the decision matrix I would use at a bank.

- URL: https://fernando.moretes.com/blog/treino-e-inferencia-na-mesma-gpu-quatro-formas-de-partilhar-10-mil-ace

- Markdown: https://fernando.moretes.com/blog/treino-e-inferencia-na-mesma-gpu-quatro-formas-de-partilhar-10-mil-ace/article.md?lang=en

- Published: 2026-09-15T10:15:23.382Z

- Category: AI & Agents

- Tags: kubernetes, eks, gpu, kueue, keda, finops, sagemaker-hyperpod, ai-platform

- Reading time: 7 min

- Source: [China Merchants Bank Wins CNCF End User Case Study Contest for Unifying AI Training and Inference on Kubernetes](https://www.cncf.io/announcements/2026/09/07/china-merchants-bank-wins-cncf-end-user-case-study-contest-for-unifying-ai-training-and-inference-on-kubernetes/)

---

After 16 years operating financial platforms on AWS, the question I have been asked most about GPUs in the last two years was not "which instance?" — it was "why does the bill grow while `nvidia-smi` shows 30%?". China Merchants Bank's case, winner of the CNCF End User Case Study Contest announced on September 8, 2026, answers with numbers: 99% of nearly 10,000 heterogeneous accelerators under a single Kubernetes control plane, average utilization from 35% to over 60%, cost per million tokens cut by more than 60%. None of those gains came from a new chip. They came from queues, quotas, preemption and caching — architecture decisions any team running EKS can make, and that four different designs make in very different ways.

## The real choice: who decides where the next GPU goes

The common mistake is framing the problem as "training versus inference". It is not a fight between workloads — it is a fight between two demand regimes that never line up in time. Training is batch: it arrives in bursts, wants 64 cards for days, tolerates waiting in a queue and tolerates being interrupted if there is a checkpoint. Inference is online: it arrives as a traffic curve, wants a predictable p99, tolerates neither waiting nor preemption. When each regime gets its own pool, each pool is sized for its own peak, and the sum of both peaks is the bill. CMB's 35% before the change is exactly that sum: half the fleet idle, waiting for the neighbor's peak.

The architecture decision, then, is **who arbitrates the boundary between the two regimes** — and at what granularity. There are four honest answers:

1. **Nobody arbitrates:** separate pools, boundary fixed at provisioning time.
2. **An open scheduler arbitrates inside the cluster:** Kueue for quota-aware admission, KEDA to scale inference on metrics, HAMi to slice the card, Fluid so the data lands before the GPU goes idle. This is CMB's design.
3. **AWS arbitrates with the same scheduler, packaged:** SageMaker HyperPod with EKS orchestration and task governance, which installs Kueue and exposes quota, fair-share and borrowing as console policy.
4. **A managed service arbitrates outside the cluster:** inference on Bedrock or SageMaker endpoints, training as SageMaker jobs — no fleet of your own to share.

The rest of this article compares the four. The answer is not the same for a bank with 10,000 cards and a fintech with 40.

## What CMB assembled, piece by piece

The announcement names five projects and gives each a role. It is worth translating each role into the parameter it controls, because that is where utilization is won or lost.

**Kueue — quota-aware admission:** each team submits to a `LocalQueue` in its own namespace; the `LocalQueue` points to a `ClusterQueue` with a `nominalQuota` per `ResourceFlavor` (one flavor per card type — that is how 10,000 heterogeneous accelerators become one queue). `ClusterQueues` in the same `cohort` lend idle quota to each other up to `borrowingLimit`, and `spec.preemption.reclaimWithinCohort: Any` takes the loan back when the owner needs it. The main gain is what the announcement calls "not reserving capacity prematurely": the job only creates pods once its whole quota is guaranteed, so 64 cards do not sit half-allocated waiting for the other 32.

**KEDA — inference scales on a demand signal:** a `ScaledObject` with the `prometheus` scaler (`serverAddress`, `query`, `threshold`, `activationThreshold`) scales model-server replicas by queue depth or tokens per second, not CPU. When traffic drops, replicas disappear and quota flows back to the cohort for training.

**HAMi — the card becomes a fractional resource:** a CNCF Incubating project that exposes `nvidia.com/gpumem` (in MB) and `nvidia.com/gpucores` alongside `nvidia.com/gpu`, supporting NVIDIA, Ascend, Cambricon and others. It is what lets five LoRA tenants share one base-model instance — Twinkle, the bank's in-house framework, does this by default and reports 80% fewer accelerators for the same work.

**Fluid — data before GPU:** a CNCF Incubating project with `Dataset` + cache runtime (Alluxio, JuiceFS) and warm-up. Checkpoint and model are already on the node when the pod starts. A GPU waiting on object storage is zero utilization that shows up as 100% "allocated".

**Prometheus** closes the loop: it is the source of truth both for KEDA and for the dashboard that proved the 60%.

## Unified control plane: who decides where the next GPU goes

Three regimes enter through different queues but compete for the same quota; metrics close the loop and the cache takes data wait off the critical path.

### 🎛 Plano de controle — admissão e escala

- Kueue ClusterQueue + cohort + preempção (compute)
- KEDA ScaledObject (prometheus) (compute)
- HAMi gpumem / gpucores (ai)
- Prometheus utilização, fila, tokens/s (data)

### 🟧 AWS — frota heterogênea (EKS)

- p5 / p6 ResourceFlavor: treino (compute)
- g6e / g7e ResourceFlavor: inferência, MIG (compute)

### 📦 Dados — cache antes da GPU

- Fluid Dataset + warm-up (storage)
- S3 modelos, checkpoints (storage)

### Flows

- train -> kueue: LocalQueue
- lora -> kueue: LocalQueue
- infer -> keda: replicas
- kueue -> hami: admits when quota is whole
- keda -> hami: scales on metric
- hami -> p5: whole card
- hami -> g6e: slice / MIG
- prom -> keda: query
- p5 -> prom: DCGM
- g6e -> prom: DCGM
- s3 -> fluid: prefetch
- fluid -> p5: local checkpoint
- fluid -> g6e: local weights

## Four ways to share the fleet
| Criterion | Who arbitrates | Granularity | Isolation | Cost to maintain | Typical utilization |
| --- | --- | --- | --- | --- | --- |
| A. Separate pools | Nobody — boundary fixed at provisioning | Whole node group | Total, by construction | Low in software, high in idle capacity | 30-40% (sum of two peaks) |
| B. EKS + Kueue/KEDA/HAMi/Fluid | Your platform team, with an open scheduler | Card fraction (gpumem/gpucores) or MIG | Software (HAMi) or hardware (MIG) — you choose per flavor | High: 5 projects, versions, CRDs, driver upgrades | 60%+ (CMB's number) |
| C. SageMaker HyperPod + task governance | AWS, with Kueue packaged as policy | Instance, accelerator or MIG partition per team | Namespace per team + MIG; no software slicing | Medium: managed add-on, policy via console/CLI | 50-60% if fair-share is well calibrated |
| D. Bedrock + SageMaker jobs | AWS, outside your cluster | Token / job-hour | Total, per service | Minimal in software; price already embeds someone else's idle | Not your metric — you pay per use |

## Isolation: the hidden cost of utilization

Every percentage point of utilization above 50% is bought with less isolation. At a bank that is not a detail — it is the first thing the auditor asks.

There are three mechanisms to put two pods on the same card, and they are not equivalent:

**Time-slicing:** the device plugin or DRA driver advertises N slots per GPU and the CUDA scheduler takes turns. It works on every NVIDIA instance on AWS, needs no special hardware — and the EKS documentation is explicit: it **offers no memory or compute isolation** between pods. A tenant with a memory leak takes the neighbor down with `CUDA out of memory`. Fine for dev and sparse inference; not fine for two distinct business tenants.

**MIG:** hardware partitioning, up to 7 instances per A100/H100/H200/B200 (fewer on Blackwell `g7`/`g7e`), each with dedicated memory, compute and bandwidth. It is the only one of the three with hardware fault isolation — the kind PCI-DSS or a BACEN 4.893 audit trail accepts without debate. The price: the partition is static per node (`nvidia.com/mig.config.state` must be `success`) and only exists on the P family and Blackwell G instances.

**HAMi:** software virtualization with per-pod memory and core limits, enforced where the device backend allows. Finer than MIG (any fraction, not just 7 profiles), safer than time-slicing (the limit exists), and the only one of the three that speaks to non-NVIDIA accelerators — relevant for CMB with a heterogeneous fleet, less so if you are 100% on EC2. Fault isolation remains software-level.

The rule I use: **MIG for multi-tenant inference with an SLO**, HAMi or time-slicing for LoRA fine-tuning and experimentation, whole card for pre-training. Option B lets you choose per `ResourceFlavor`; option C only offers MIG or whole card. If your case is "five LoRA tenants on one base", that decides it.

## Decision matrix

### A. Separate pools

**Pros**
- Total isolation with zero configuration
- Driver-upgrade blast radius stays in one pool
- Trivial chargeback: the node group is the cost center

**Cons**
- Pays for two peaks; 35% is the expected result, not an accident
- Training waits in queue while inference GPUs idle overnight

**Verdict:** Only up to ~40 cards or until there is a platform team

### B. EKS + Kueue/KEDA/HAMi/Fluid

**Pros**
- Maximum granularity: fraction, MIG or whole card per flavor
- Works with heterogeneous accelerators, including outside AWS
- 60%+ utilization proven on 10,000 cards

**Cons**
- Five projects to version; DRA only recommended from K8s 1.34 and does not run on EKS Auto Mode
- Miscalibrated preemption kills training without checkpoints
- Software isolation needs justification in an audit

**Verdict:** Above ~500 cards with a dedicated platform team of 4+ people

### C. HyperPod + task governance

**Pros**
- Kueue, quota, fair-share (weight 0-100) and borrowing (up to 10,000% of quota) as policy, not YAML
- `IdleResourceSharing` reconciles unallocated capacity on its own
- Per-team queue and wait observability comes in the dashboard

**Cons**
- Only HyperPod-supported instance types; no mixed fleet outside AWS
- GPU fractions only via MIG — no HAMi equivalent
- Metric-driven inference elasticity is still your problem (KEDA still needed)

**Verdict:** From 50 to 2,000 cards, 100% AWS, small platform team

### D. Bedrock + SageMaker jobs

**Pros**
- Zero idle GPU in your account
- Guardrails, audit trail and isolation come from the service

**Cons**
- No control over placement, cache or slicing — the per-token price already embeds someone else's idle
- In-house model with data-residency rules may not fit

**Verdict:** Default for anyone without an in-house model or fleet

## Preemption and freshness: the two failure modes nobody draws

Unified-platform diagrams show the happy path. The two incidents I have watched happen sit outside the drawing.

**Preemption without checkpoints.** When inference scales up at 9 AM and the cohort reclaims borrowed quota, Kueue evicts the training job that was using someone else's GPUs. If the job does not checkpoint every N steps, it loses everything since the start — and the next day someone will ask to "turn off that preemption", which sends the platform back to 35%. The configuration that prevents this is not in Kueue; it is in the job: checkpoint at an interval short enough that average rework stays under 15 minutes, written to the Fluid cache with write-back to S3. In Kueue, `withinClusterQueue: LowerPriority` protects training from being interrupted by another training job from the same team; `reclaimWithinCohort: Any` says borrowed quota can be taken back at any time. Those two lines together are the contract.

**Oscillating scale.** KEDA with a tight `threshold` and a short `cooldownPeriod` produces flapping: replicas go up, training gets preempted, traffic drops, replicas go down, training restarts from checkpoint, traffic goes up. Each cycle costs a model cold start (tens of seconds for 30 GB of weights without cache) and training rework. Use an `activationThreshold` above zero so noise does not wake a replica, and a `cooldownPeriod` measured in minutes, not seconds.

**Cold data.** Without Fluid or an equivalent, the inference pod starts, asks S3 for 30 GB of weights and sits allocated for two minutes at 0% use. On the dashboard that counts as "GPU in use". That is the difference between allocated and real utilization — measure `DCGM_FI_DEV_GPU_UTIL`, not the pod's `requests`.

> **The arithmetic that justifies the project:** From 35% to 60% on the same fleet is 1.7x the work for the same fixed cost — or, on 10,000 cards, the equivalent of 2,500 accelerators you do not need to buy. On a fleet of 200 p5 cards the same jump is worth 50 cards. Before choosing between B and C, compute that number at your scale: if it is smaller than the annual cost of one platform engineer, the answer is C or D, not B.

## Anti-patterns that send the fleet back to 35%

- **Time-slicing for distinct business tenants:** with no memory isolation, a leak in one tenant becomes `CUDA out of memory` in the other — and the audit trail cannot explain it.
- **Quota equal to fleet size:** if the sum of `nominalQuota` hits the total, nobody has anything to lend and the cohort becomes decoration.
- **Preemption on, checkpoints off:** the first lost training run triggers the request to turn preemption off, and the platform is back where it started.
- **Measuring utilization by `requests`:** an allocated GPU waiting for weights from S3 shows as 100% and hides exactly the problem Fluid solves.

> **Curator's note:** If I had to build this tomorrow at a 100%-AWS bank, I would start with option C: HyperPod with task governance gives me Kueue, fair-share and borrowing without maintaining the CRDs, and the per-team wait dashboard is the argument that convinces the CFO the queue is real. I would only move to option B the day I needed software-level fractions for multi-tenant LoRA or accelerators outside the P/G families — and on that day I would hire before I installed. The hard-won lesson behind this: the cost of a unified control plane is not standing up five Helm charts — it is maintaining five release cycles, an NVIDIA driver that changes with every AMI and a preemption policy that has to be re-explained to every new team, for years.

## References

- [CNCF — China Merchants Bank wins End User Case Study Contest (Sep 8, 2026)](https://www.cncf.io/announcements/2026/09/07/china-merchants-bank-wins-cncf-end-user-case-study-contest-for-unifying-ai-training-and-inference-on-kubernetes/)
- [Kueue — Concepts: ClusterQueue, LocalQueue, ResourceFlavor, cohort, preemption](https://kueue.sigs.k8s.io/docs/concepts/)
- [HAMi — Heterogeneous AI Computing Virtualization Middleware (CNCF Incubating)](https://github.com/Project-HAMi/HAMi)
- [Fluid — Cloud-native data access platform (CNCF Incubating)](https://fluid-cloudnative.github.io/)
- [KEDA 2.18 — Prometheus scaler](https://keda.sh/docs/2.18/scalers/prometheus/)
- [Amazon EKS — Manage NVIDIA GPUs: DRA, device plugin, MIG and time-slicing](https://docs.aws.amazon.com/eks/latest/userguide/device-management-nvidia.html)
- [SageMaker HyperPod task governance — Policies (fair-share, borrowing, preemption)](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-console-ui-governance-policies.html)
- [CNCF blog — Building a reliable cloud native foundation for distributed AI training (Atlassian, Sep 11, 2026)](https://www.cncf.io/blog/2026/09/11/building-a-reliable-cloud-native-foundation-for-distributed-ai-training/)

## Verdict

Use **separate pools (A)** while you have fewer than ~40 cards or no platform team — the waste costs less than the operation. Use **HyperPod with task governance (C)** from 50 to ~2,000 cards, 100% AWS, when fair-share, borrowing and MIG solve your multi-tenancy: it is the same Kueue CMB runs, with the maintenance cost shifted to AWS. Use **EKS with Kueue, KEDA, HAMi and Fluid (B)** above that, or when you need software-level fractions and a heterogeneous fleet — and treat the five projects as an internal product, with an owner, a release cadence and a runbook. Use **Bedrock and SageMaker jobs (D)** whenever you have no model of your own; a shared fleet only makes sense when there is a fleet. In every case, measure `DCGM_FI_DEV_GPU_UTIL`, not `requests` — the utilization CMB reported is the real one, and it is the only one that turns into a smaller bill.

**Rating:** C for most banks on AWS; B above ~2,000