Controller lifecycle

The controller drives every Agent from creation to a terminal phase by reconciling its status against the world. Reconciliation is a pure function with all I/O injected, which keeps it unit-testable.

State machine

stateDiagram-v2
    [*] --> Pending : Agent created
    Pending --> Running : Job created, status patched
    Pending --> Failed : Station or AgentDefinition missing
    Running --> Running : Job still running
    Running --> Succeeded : Job exit 0
    Running --> Failed : Job failed
    Succeeded --> [*]
    Failed --> [*]
    note right of Running : on a terminal transition, prune old runs per history limits

Transitions

Pending → Running

1. Resolve Station by spec.stationRef; if missing, set phase = Failed with a reason and stop.
2. Resolve AgentDefinition by station.spec.agentDefRef; if missing, fail the same way.
3. Build the Job (see Agent runtime) and create it. Creation is idempotent. An already-existing Job is fine.
4. Patch status: phase = Running, jobName, startedAt.

Running → Succeeded | Failed

1. Read the Job outcome. While it is still running, do nothing.
2. On success, patch phase = Succeeded with exitCode and output (the truncated tail of the pod log).
3. On failure, patch phase = Failed with exitCode, failureReason, and output (the truncated tail).
4. After any terminal transition, prune history.

Every status patch carries the Agent’s resourceVersion as a precondition, so a write computed from a stale read is rejected (409 Conflict) instead of clobbering a newer update. The watch and the poll reconcile from independent snapshots, so this optimistic-concurrency guard is what keeps them from lost-updating each other: a conflicted write is dropped and the next reconcile recomputes from fresh state.

If the run pod was already garbage-collected when the controller reads back (so its captured stdout is gone), reading it back is best-effort: the Agent still reaches its terminal phase — keeping any Job-level failure reason, or recording run output unavailable: pod garbage-collected when there is none — instead of leaving output silently empty or letting the read error leave the Agent stuck in Running. If the Job itself was garbage-collected first (the controller was down or backlogged past ttlSecondsAfterFinished), the outcome is unrecoverable, so the Agent is reported Failed with run record unavailable: Job garbage-collected before its result was observed rather than being left stuck in Running.

exitCode always agrees with the phase: a Failed run reports a non-zero code even when the pod’s container status reads back as 0 (a GC race can leave it unpopulated), and a Succeeded run reports 0.

Terminal → terminal is a no-op; reconciling a finished Agent does nothing.

Batch runs

An Agent whose AgentDefinition sets execution: batch goes through the same phases, but the controller does the work itself instead of creating a Job. See batch runs.

Pending → Running (submit). After resolving the refs, the controller checks that the definition can run as a batch (see batch validation) and that the controller runs with BATCH_ENABLED=true (the components/batch kustomize component), and fails the Agent with the reason if not. Station concurrency applies as for Jobs; under Replace a full Station’s oldest Running run is deleted and the new Agent submits its batch. It then reads the API key from agent-secrets, submits the rendered prompt with the run’s model (the Agent’s spec.model when set, else the definition’s, else claude-opus-5-5), batch.maxTokens and any output.schema to the batch API of the model’s vendor (Anthropic, Gemini or OpenAI), patches phase = Running, batchId, batchProvider and startedAt, and adds the agents.re-cinq.com/batch-cleanup finalizer.

Running (poll). While the batch is in flight, the controller polls it at most once every BATCH_POLL_SECONDS (default 60) rather than on every watch event. A pending batch leaves the Agent Running. A finished batch moves it to Succeeded or Failed with output set to the vendor CLI’s event lines built from the response; a failed request’s message becomes failureReason.

Timeout. Once batch.timeoutMinutes have passed since startedAt, the controller cancels the batch. With onTimeout: fail the Agent becomes Failed with batch timed out. With onTimeout: fallback it patches batchFallback = true (keeping batchId) and creates the ordinary Job for the same Agent; from then on the Job transitions above apply. If the Station is full, the Agent waits in Running until a slot frees. A poll that reports the result in the same pass wins: the Agent completes and no fallback starts.

Sink events. The controller sends the definition’s output.sinks the events a pod’s supervisor would: agent/started after the submit patch, the CLI lines (through output.select) and agent/succeeded or agent/failed after the completing patch, and a batch_timeout lifecycle event after a timeout patch (failed for onTimeout: fail, running for fallback, where the Job’s supervisor reports the end). Each is sent after its patch succeeds, at most once. See batch runs.

Cleanup and the finalizer. After the result is recorded, and after a timeout, the controller deletes the batch at the provider and removes the finalizer. When an Agent with the finalizer is deleted, the controller cancels its batch if it is still in flight, cleans it up, then removes the finalizer so the delete goes through. Failed cleanups are retried and counted in status.batchCleanupAttempts; after 5 the finalizer is removed anyway and the provider object is left behind; each failed attempt also increments controller_batch_cleanup_failures_total. Finalizer changes are JSON patches that touch only this one entry of metadata.finalizers.

Realtime runs

An Agent whose AgentDefinition sets execution: realtime goes through the same phases and gets a Job like an interactive run. The controller fails it at Pending with realtime execution is disabled on this controller unless it runs with the realtime component, and with the matching validation reason when the definition sets resources.conversation.id or pin, or when a resumeSession cannot be honoured (resumeSession requires realtime execution, resumeSession needs resources.conversation.source, <vendor> cannot resume a session). The Job’s supervisor serves prompts on port 8090 and accepts only the run’s own token, which the controller derives from the Agent’s uid. The pod carries the agents.re-cinq.com/execution: realtime label. With spec.resumeSession set, the Job restores that conversation and sends no initial prompt.

While the Agent is Running, the controller’s prompt API forwards prompts and closes to the pod and patches status.sessionId when a prompt reveals a new one. The Job completes when the session ends, and the Agent goes terminal as any run does, with status.sessionId read from the pod log. See realtime sessions.

Launch sequence

sequenceDiagram
    autonumber
    participant Caller
    participant API as Kubernetes API
    participant Ctrl as Controller
    participant Job
    participant Pod
    participant Sink as Output sink

    Caller->>API: create Agent (phase Pending)
    API-->>Ctrl: watch event (ADDED)
    Ctrl->>API: get Station + AgentDefinition
    Ctrl->>API: create Job (owner ref to Agent)
    Ctrl->>API: patch status = Running
    Job->>Pod: schedule pod
    Pod->>Pod: initContainer injects bundle
    Pod->>Pod: supervisor runs agent process
    Pod->>Sink: stream-json events
    Ctrl->>API: read Job outcome
    Ctrl->>API: patch status = Succeeded / Failed
    Caller->>API: watchAgent polls status until terminal

Job ownership

Each Job is created with an owner reference back to its Agent (controller: true, blockOwnerDeletion: true). Deleting the Agent garbage-collects the Job. Jobs also set ttlSecondsAfterFinished: 3600, so finished Jobs (and their pods) self-delete one hour after completion while the Agent’s status retains the result. This is the window the controller has to read the pod’s exit code and captured stdout back into status; one hour comfortably survives a controller restart or backlog. The trade-off is that finished pods linger that long, but they are terminal (only an etcd object plus node log disk, no CPU/memory), and history pruning already cascade-deletes most of them sooner. Past the window the run is reported terminal with a clear failureReason rather than silently losing its output.

History pruning

On every terminal transition the controller groups its cached Agents for the Station by phase, sorts by completedAt (newest first), and deletes any beyond the Station’s successfulRunsHistoryLimit / failedRunsHistoryLimit. This bounds how many finished Agents accumulate without losing the most recent ones. Batch Agents are pruned the same way.

Watch + poll + cache

The controller combines a low-latency watch with a ~15s poll, sharing one in-memory cache of Agents. The watch seeds the cache and its starting resourceVersion with a full, paginated LIST (?limit=&continue=), then resumes from that resourceVersion, applying each event to the cache (ADDED/MODIFIED upsert, DELETED evict) and reconciling the changed Agent. When the watch closes it resumes from the last resourceVersion seen — it does not replay the whole collection. A 410 Gone (the change history was compacted past our cursor) triggers a fresh paginated re-list. The watch carries a server-side timeoutSeconds, so the API server periodically ends the long poll and the loop reconnects — a silently dropped (half-open) connection can’t leave it blocked on a dead socket. On a real error (API blip, TLS, 5xx) the loop backs off exponentially with jitter rather than retrying every couple of seconds, so a degraded API server isn’t hammered in lockstep.

The poll runs independently every ~15s: a full paginated LIST that refreshes the cache and reconciles every Agent. This is the safety net — it guarantees an Agent whose watch event was missed or dropped is still reconciled, since the watch is the fast path but not a guarantee. Because concurrency counts and history pruning read the cache rather than doing their own LIST per reconcile, reconcile work is O(changed) even though the poll itself lists. The /metrics endpoint exposes controller_resyncs_total (full LISTs) and controller_watch_reconnects_total.

The same endpoint tells how runs go. controller_agents{phase} is the number of Agents in each phase at the last poll; controller_runs_completed_total{phase,exit_code} counts every terminal transition; controller_run_duration_seconds{phase} is a histogram from status.startedAt to the terminal patch; controller_run_queue_seconds is a histogram of the wait from an Agent’s creationTimestamp to its Job, which is how long it sat behind a Station’s concurrency limit; and controller_run_failures_total{kind} counts the runs that ended with a reason, by kind (missing_ref, job_failed, output_unavailable, preempted) rather than by message, so the series stay few. A run failed for a missing reference is counted and not timed, since it never ran; a run preempted under the Replace policy has no terminal status and appears only under preempted. controller_batch_cleanup_failures_total counts failed attempts to delete a batch run’s provider-side objects (see batch runs); a steady rise means batches are being left at the provider. Every number is the leader’s own process state: a restart starts them from zero, which Prometheus reads as a counter reset.

Leader election

The controller runs two replicas for availability, but only one reconciles at a time. The replicas contend for a single coordination.k8s.io/v1 Lease named agent-controller; the holder is the leader and runs the watch + poll loop, while standbys stay idle.

stateDiagram-v2
    [*] --> Standby
    Standby --> Leader : Lease absent, or held by us, or expired and we take it over
    Leader --> Leader : renew the Lease every ~5s
    Leader --> Standby : renewal lost (another replica wrote, or the API is unreachable)
    note right of Leader : only the leader reconciles

Each tick (~5s) a replica reads the Lease, folds it into a running observation, and decides:

• no Lease → create one we hold;
• we hold it → renew its renewTime;
• someone else holds it, unrenewed past the 15s lease duration → take it over;
• someone else holds a still-valid Lease → stand by.

Expiry is judged against the replica’s own clock — the time since it first saw the current renewTime — not the remote timestamp, so clock skew between replicas can’t trigger a premature takeover. Writes carry the Lease’s resourceVersion as an optimistic-concurrency precondition, so two standbys can’t both win a takeover and a leader that fails to renew (lost the race, or lost the API server) immediately steps down and stops reconciling.

When the leader’s pod is deleted it stops renewing; a standby sees the renewTime stop changing and takes over within the lease duration. The brief overlap a partition could cause is harmless anyway: Job creation is keyed on the Agent name and idempotent (an existing Job is a 409, not a duplicate), so two reconcilers never produce two Jobs for one Agent.