The broker can reclaim VRAM it did not allocate, but nothing stops a direct client taking it — a bare ollama run still OOM-kills whisper #13

New Issue

mAi · 2026-07-29T10:34:34Z

mAi commented

2026-07-29 10:34:34 +00:00

Split out of #6. #6 asked for the broker to be the sole VRAM authority. What landed on 2026-07-29 (#4, #6, #8, main 255828d) is reclaim under pressure: when a consumer asks the broker for VRAM, the broker now measures real per-PID residency and can evict holds it never allocated. That closes the reported incident, where an idle 24-hour ollama keep-alive starved ImaGen.

This issue is the half that is still open: prevention.

Reproduced, not theorised

During #8 verification, a plain model load straight at ollama's own port:

POST localhost:11434/api/generate  {"model":"gemma3:12b"}   # ~11.3 GB

whisper-server went into a crash loop for want of VRAM. STT was down until the model was unloaded by hand. The broker was active throughout and did nothing, correctly — nothing in that path touches it.

Why reclaim cannot cover this

Eviction is demand-driven. It runs when something asks the broker for VRAM. A direct client never asks, so there is no decision point at which the broker could intervene. By the time the allocation exists, the damage is done: the victim is not evicted, it is OOM-killed.

The asymmetry matters. A broker-mediated consumer gets a scheduled, ordered eviction and comes back on the next request. A bystander to a direct load gets killed mid-request with no restart path.

The two shapes a fix could take

These are genuinely different, and the choice is a design call, not an implementation detail:

Admission control — put ollama behind the broker for real, so no client can load a model without passing the scheduler. Strongest guarantee. Costs: every direct consumer of :11434 must be redirected (youpc batch embeddings, the agentic-swap model hub, anything a human types), and a direct call must then either fail or be proxied. This is the "sole VRAM authority" #6 literally asked for.
Survivability — accept that VRAM can vanish underneath a service and make services recover rather than crash-loop. Weaker guarantee, much smaller blast radius, and it also covers the case no admission control can: a game, a browser, or an RDP session taking memory outside any of this.

They are not exclusive. (2) is the safety net that makes (1)'s remaining gaps survivable.

Not urgent, and worth saying why

The incident that opened #6 is fixed. This is the residual: it needs a direct load large enough to starve a resident service, at a moment when one is resident. It has happened exactly once, under a deliberate test. Filing it so #6 can close honestly rather than closing with an unstated limit.

What #6 now guarantees, precisely

Per-consumer residency is measured from the GPU, not from broker bookkeeping (#4). All four consumers report real numbers.
A hold the broker did not allocate is a legitimate eviction victim and gets reclaimed under pressure (#6).
Eviction of whisper-server really stops the unit and the scheduler restarts it on the next request (#8).
Not guaranteed: that VRAM stays available to a running service. A direct client, or any non-broker GPU user, can still take it.

Provenance

Found by tesla (mgpumanager/gitster) while verifying #8 on 2026-07-29, when its own pressure test crash-looped whisper. Full statement of the limit, with the fixture: #6 comment 17196. Left unassigned — this is a scope decision for m before it is work for anyone.

Related: #6 (reclaim, shipped), #4 (residency measurement, shipped), #8 (whisper stop lever, shipped), #12 (what a VRAM budget can express).

Split out of #6. #6 asked for the broker to be the sole VRAM authority. What landed on 2026-07-29 (#4, #6, #8, main `255828d`) is **reclaim under pressure**: when a consumer asks the broker for VRAM, the broker now measures real per-PID residency and can evict holds it never allocated. That closes the reported incident, where an idle 24-hour ollama keep-alive starved ImaGen. This issue is the half that is still open: **prevention**. ## Reproduced, not theorised During #8 verification, a plain model load straight at ollama's own port: ``` POST localhost:11434/api/generate {"model":"gemma3:12b"} # ~11.3 GB ``` whisper-server went into a crash loop for want of VRAM. STT was down until the model was unloaded by hand. The broker was active throughout and did nothing, correctly — nothing in that path touches it. ## Why reclaim cannot cover this Eviction is demand-driven. It runs when *something asks the broker* for VRAM. A direct client never asks, so there is no decision point at which the broker could intervene. By the time the allocation exists, the damage is done: the victim is not evicted, it is OOM-killed. The asymmetry matters. A broker-mediated consumer gets a scheduled, ordered eviction and comes back on the next request. A bystander to a direct load gets killed mid-request with no restart path. ## The two shapes a fix could take These are genuinely different, and the choice is a design call, not an implementation detail: 1. **Admission control** — put ollama behind the broker for real, so no client can load a model without passing the scheduler. Strongest guarantee. Costs: every direct consumer of `:11434` must be redirected (youpc batch embeddings, the agentic-swap model hub, anything a human types), and a direct call must then either fail or be proxied. This is the "sole VRAM authority" #6 literally asked for. 2. **Survivability** — accept that VRAM can vanish underneath a service and make services recover rather than crash-loop. Weaker guarantee, much smaller blast radius, and it also covers the case no admission control can: a game, a browser, or an RDP session taking memory outside any of this. They are not exclusive. (2) is the safety net that makes (1)'s remaining gaps survivable. ## Not urgent, and worth saying why The incident that opened #6 is fixed. This is the residual: it needs a direct load large enough to starve a resident service, at a moment when one is resident. It has happened exactly once, under a deliberate test. Filing it so #6 can close honestly rather than closing with an unstated limit. ## What #6 now guarantees, precisely - Per-consumer residency is measured from the GPU, not from broker bookkeeping (#4). All four consumers report real numbers. - A hold the broker did not allocate is a legitimate eviction victim and gets reclaimed under pressure (#6). - Eviction of whisper-server really stops the unit and the scheduler restarts it on the next request (#8). - **Not guaranteed:** that VRAM stays available to a running service. A direct client, or any non-broker GPU user, can still take it. ## Provenance Found by tesla (mgpumanager/gitster) while verifying #8 on 2026-07-29, when its own pressure test crash-looped whisper. Full statement of the limit, with the fixture: #6 comment 17196. Left unassigned — this is a scope decision for m before it is work for anyone. Related: #6 (reclaim, shipped), #4 (residency measurement, shipped), #8 (whisper stop lever, shipped), #12 (what a VRAM budget can express).

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: m/mGPUmanager#13

The broker can reclaim VRAM it did not allocate, but nothing stops a direct client taking it — a bare ollama run still OOM-kills whisper #13