Fractional GPU on ECS: Architecture Decision Record for AI Inference

Before this feature, ECS treated GPU as a binary resource: either you allocated an entire GPU to a container, or you allocated none. In practice, this meant a credit risk classification model with 1.2 billion parameters — which fits comfortably in 4 GB of VRAM — consumed an entire 24 GB L4 GPU, wasting 83% of accelerator memory capacity. For inference platforms serving dozens of different models with asymmetric traffic patterns, that waste translated directly into unnecessary instance cost and cold-start latency because auto-scaling had to provision full instances for traffic spikes on small models.

G6f instances were announced in GA in July 2025 with native GPU partitioning support via NVIDIA MIG-like partitioning on L4 Tensor Core GPUs. The integration with the ECS scheduling plane, however, only arrived in August 2026 — a 13-month gap that forced teams to use workarounds like custom device plugins on EKS or manual GPU allocation via environment variables. Now, with GPU=0.125, GPU=0.25, and GPU=0.5 as valid values in ECS task definitions, the scheduler achieves real bin-packing: up to 8 tasks on a single G6f instance, each with 3 GB of isolated VRAM.

The financial context adds layers of complexity. Regulators like the Brazilian Central Bank and the SEC require traceability of ML model decisions used in credit, fraud detection, and derivatives pricing. This means multiple models must coexist with auditable memory isolation — exactly what GPU partitioning offers, provided the configuration is done correctly and monitored continuously.

The decision to adopt fractional GPU on ECS is not simply 'enable and save'. There are genuinely competing architectural forces that must be resolved before any production change.

Density vs. Blast Radius Isolation: When you place 8 tasks on a single G6f instance, a GPU driver failure or a hung CUDA kernel can affect all 8 partitions simultaneously. ECS Managed Instances includes automatic health monitoring that detects GPU hardware failures and replaces unhealthy instances, but the detection and replacement time — typically 2-5 minutes — is unacceptable for real-time fraud detection workloads with P99 latency SLOs below 200ms.

Cost per Inference vs. Reserved Capacity Cost: A G6f instance with 8 one-eighth GPU partitions can serve 8 small models simultaneously, but if traffic is seasonal and only 2-3 partitions are active most of the time, the idle instance cost outweighs the bin-packing benefit. The correct calculation requires real GPU utilization data per model, not estimates.

VRAM Memory Management vs. Runtime Overhead: With 3 GB of VRAM per 1/8 partition, models using aggressive INT8 or FP16 quantization fit easily. But frameworks like TensorRT and PyTorch Serve have runtime overheads that consume 200-400 MB of VRAM before even loading model weights. On 3 GB partitions, that overhead represents 7-13% of total capacity — a factor that must enter the sizing calculation.

ECS Managed Instances vs. ECS on EC2 Self-Managed: The choice between the two operating modes has direct implications for operational control, AMI customization, and management cost. For financial environments with OS hardening requirements and CIS Benchmark compliance, the ECS Managed Instances managed AMI may not satisfy requirements without additional configuration.

For an AI inference platform in a financial environment serving credit scoring, document classification, and anomaly detection models — all with maximum VRAM of 8 GB and throughput of 50-500 req/s per model — the decision is to adopt ECS with fractional GPU on G6f instances using ECS Managed Instances, with the following specific configurations.

Each inference task definition declares GPU=0.25 as default (6 GB of VRAM, 4 tasks per instance), reserving headroom for TensorRT runtime overhead (≈350 MB) and batch size spikes. Models with less than 2 GB of INT8-quantized weights can use GPU=0.125, but only on services with minimumHealthyPercent=100 and maximumPercent=200 to ensure that an instance replacement does not reduce available capacity below the minimum SLO.

The capacity provider is configured with a dedicated Auto Scaling Group (ASG) for G6f instances, with targetCapacityPercent=70 — maintaining 30% headroom to absorb spikes without provisioning latency. managedTerminationProtection=ENABLED on the capacity provider ensures that instances with active tasks are not terminated during scale-in, avoiding interruptions to in-flight inferences.

For IAM, each task role has minimum permissions via condition keys: aws:RequestedRegion fixed to the operating region, aws:SourceVpc restricted to the inference VPC, and S3 access for model artifact reads limited by prefix via s3:prefix condition. Dedicated KMS CMK per environment (dev/staging/prod) for model artifact encryption at rest, with key policy allowing only the ECS task role and the model deployment CI/CD pipeline.

The decision to use ECS Managed Instances over self-managed ECS on EC2 is motivated by automatic GPU health monitoring — specifically hardware failure detection that replaces instances without manual intervention. In a financial environment with 99.9% SLA for inference, that automation is more valuable than the additional AMI control that self-managed mode offers.

Correct configuration of a task definition for fractional GPU requires attention to details not obvious in the launch documentation. The resourceRequirements field in the container definition accepts type: GPU with value as a string representing the fraction: "0.125", "0.25", or "0.5". Important: the value is a string, not a float — a common mistake that results in silent validation failure and task placement on instances without GPU.

For the capacity provider with G6f, the ASG configuration must include instanceWarmupPeriod of at least 180 seconds to accommodate NVIDIA driver and container runtime initialization time. With managedScaling enabled, targetCapacityPercent=70 on the capacity provider ensures ECS maintains enough instances to absorb a 43% spike above current capacity without provisioning latency. For fraud detection workloads with P99 SLO < 200ms, any instance cold-start latency is unacceptable — the 30% headroom is the insurance cost for that SLO.

GPU observability via CloudWatch Container Insights with ECS Managed Instances exposes metrics like GPUUtilization, GPUMemoryUsed, and GPUMemoryTotal per task and per instance. Alarm configuration must include: (1) GPUMemoryUsed / GPUMemoryTotal > 90% for 5 minutes as a memory pressure signal that precedes OOM kills; (2) GPUUtilization < 15% for 15 minutes as a safe scale-in signal; (3) TaskCount < desiredCount for 2 minutes as a service degradation alarm that triggers PagerDuty.

For regulatory traceability, each inference request must include in the response header the X-Model-Version and X-Inference-TaskArn — the ECS Task ARN allows correlating the model decision with the exact instance and GPU partition that executed it, which is required for credit model audits under Central Bank guidelines.

The cost reduction promise of fractional GPU is real, but conditional. The calculation must be done with real utilization data, not the optimistic premise that all partitions will be occupied all the time.

Consider a concrete scenario: a financial platform with 12 small inference models, each with a peak of 100 req/s and 4 GB VRAM. Under the previous model (full GPU per task), you would need 12 G6 instances (one GPU per model), at approximately $X per hour. With fractional GPU using GPU=0.25 (4 tasks per G6f instance), you need a minimum of 3 G6f instances for the 12 models — a theoretical 75% reduction in instance count.

But the real calculation includes: (1) 30% headroom to absorb spikes without cold-start → 3 instances become 4; (2) minimum redundancy of 3 instances per critical model for blast radius → critical models need dedicated instances; (3) instanceWarmupPeriod of 180s means the ASG cannot scale fast enough for abrupt spikes without pre-warming.

In practice, for 12 models with mixed distribution (4 critical + 8 experimental), the optimal configuration is: 4 dedicated G6f instances for critical models (one model per instance, using GPU=0.25 to leave headroom for simultaneous A/B versions) + 2 shared G6f instances for the 8 experimental models. This results in 6 G6f instances versus 12 previous G6 instances — a 50% reduction, not 75%, but still significant. The real benefit is that experimental models move from dedicated to shared instances, freeing capacity for critical ones without cost increase.

To calculate break-even, the correct metric is cost per 1000 inferences per model, not instance cost. With bin-packing, the denominator increases proportionally to the number of tasks per instance, but the numerator (instance cost) remains fixed — which improves cost per inference only when aggregate partition utilization is high enough to justify the shared instance cost.