Critical or non-critical: the failure contract of ECS daemons

The sidecar-per-task model solves distribution and fails at economics and governance. An instance running 40 tasks runs 40 copies of the log shipper; at 128 MiB each, that is 5 GiB of RAM spent duplicating one function. Worse than the waste is the drift: the agent version becomes the property of each application team, and the security team finds out during an audit that three services still run a seven-month-old image.

ECS already had the DAEMON scheduling strategy for the EC2 launch type, but that is placement only — one daemon per instance, with no guarantee that it is up before the first application task serves traffic. For telemetry that is a hole in the graph. For a runtime security agent, it is a window in which the instance processes transactions without coverage.

Managed Daemons, GA on April 1, 2026 for ECS Managed Instances capacity providers, closed that hole by inverting the relationship: the instance only transitions the application task to RUNNING after the daemon is running. The price of that guarantee was rigid coupling — any daemon failure drained and replaced the instance. Correct for a compliance agent. Absurd for a metrics exporter. The September launch does not add a new capability; it undoes a coupling that should never have been the only option.

The pattern has three distinct resources. First, a daemon task definition — its own resource type with its own ARN (daemon-task-definition/...), separate from your application task definition. Second, a daemon, created with CreateDaemon, binding that definition to a cluster and to one or more ECS Managed Instances capacity providers. Third, the EC2 instances provisioned by those capacity providers, on each of which ECS keeps exactly one daemon task.

The first guarantee is ordering: ECS provisions the instance, starts the daemon task, and only then promotes the application task to RUNNING. Independent measurements published in April 2026 showed the daemon reaching RUNNING 18 to 21 seconds ahead of the application, with the instance held in REGISTERING during that window. That ordering applies to every daemon, critical or not. This is the point I most often see misread: marking a daemon non-critical does not cost you the guarantee that the agent comes up before traffic; it only costs you the guarantee that it stays up.

The second guarantee is remediation, and that is what the critical switch governs. Note how ECS remediates: it does not restart the container, it drains and replaces the whole instance. That is deliberately blunt — an agent that died may have left eBPF probes detached, network rules half-applied, buffers lost. Replacing the instance is the only cheap way back to a known state.

The rule fits in one line: mark as critical the agent whose absence changes what the system is allowed to do; mark as non-critical the agent whose absence only changes what you can see. A runtime EDR, a network policy enforcer, an mTLS sidecar, or a file-integrity agent required by PCI DSS fall in the first group — without them the instance should not be eligible to process a transaction. An OTel metrics collector, a node exporter, a continuous profiler fall in the second.

The nuance lives in the log router. In financial environments the log often is the audit trail: "what you can see" becomes "what you can prove", and Fluent Bit stops being auxiliary. Do not classify by tool category; classify by the regulatory obligation that stream underwrites. If the agent's local buffer survives the outage window without dropping events, non-critical is defensible; if it drops them, it is critical.

The design corollary is what pays off most in practice: because criticality is per daemon, and a capacity provider accepts multiple daemons, do not package agents with different contracts into the same task definition. A task definition bundling the EDR and the metrics collector forces the stricter contract on both — and turns a collector memory leak into fleet replacement. Two daemons, two release cycles, two owners, two criticalities.

Too critical. A bad agent revision makes the task flap, and every flap becomes an instance replacement. On a 60-instance fleet with the drain percentage at its default of 25%, that is 15 simultaneous replacements per wave; at the measured ~4m50s per wave, a one-line agent change costs roughly 20 minutes of fleet rotation. The direct bill impact is modest and computable: the Managed Instances management fee for a c6a.2xlarge is US$0.037/hour per instance (Oregon), so 60 instances add up to ~US$1,620/month in fee alone, plus EC2 and the overlap during draining. The expensive bill is elsewhere: application task rescheduling, drained connections, cold caches, cold JIT.

Two defaults conspire here. Bake time defaults to 0 minutes — meaning that once the last wave finishes, the window in which the circuit breaker watches alarms to roll back is nil. A regression that only shows up under load sails right through.

Too non-critical. Fail-open emits no pain: the instance stays ACTIVE, the application SLO does not move, and agent coverage decays in silence. Add the documented operational trap: daemon configuration settings are not persisted between updates, and critical defaults to true. An UpdateDaemon fired by a pipeline without an explicit --no-critical re-arms fleet-wide churn that nobody asked for. It is exactly the kind of change that only surfaces at the next incident.