m/mGPUmanager
1
0
Fork 0
Code Issues 13 Pull Requests Actions Packages Projects Releases Wiki Activity

Scheduler must evict ollama for higher-priority image leases (vram_managed guard defeats the manager's purpose) #5

New Issue
Open
opened 2026-07-06 10:35:26 +00:00 by mAi · 5 comments
mAi commented 2026-07-06 10:35:26 +00:00
Copy Link

Symptom

/imagine (flexsiebels) FLUX generations fail with:

comfyui[flux-schnell-local]: broker: insufficient VRAM to load the model even after eviction (untracked GPU usage, e.g. a game?): cannot free enough VRAM for comfyui even after evicting every evictable consumer

ComfyUI itself is healthy; the mGPUmanager lease request just gets a 503.

Root cause

On a normal image lease, the scheduler evicts every evictable consumer but is structurally forbidden from reclaiming ollama's VRAM. From the mgpumanager logs (16 GB RTX 4070, comfyui needs 11000 MiB):

evicted consumer victim=mvoice          free_mib_after=993   need_mib=11000
evicted consumer victim=whisper-server  free_mib_after=3523  need_mib=11000
WARN no eviction candidates target=comfyui need_mib=11000 free_mib=3523 strict=true
POST /v1/lease status=503

Free tops out at ~3.5 GB because ollama (gemma3:12b, ~8 GB resident, 24h keep_alive) is never evicted. config/consumers.yaml documents this as intentional:

ollama:
  vram_managed: true
  # Does NOT make ollama an eviction victim — the vram_managed guard in the
  # scheduler still skips it for normal leases.
  systemd_unit: ollama.service

So ollama is only ever stopped in game mode (game_mode.stop_units: [ollama.service]), never for a normal image lease — even though comfyui is priority: 1 and ollama is priority: 2. The higher-priority consumer loses to the lower-priority one. That's the opposite of what the control plane is for.

This started biting after the 2026-07-04 ollama-after-game wiring left gemma3:12b sitting resident with a 24h keep_alive; before that ollama wasn't pinning enough VRAM to matter.

Proposed fix

Let the scheduler evict ollama for a higher-priority lease, reusing the mechanism game mode already has:

  • Add an unload/stop action for ollama on the normal eviction path — either ollama stop <model> (soft, keeps the service up) or systemctl stop ollama.service (the game-mode approach), gated on lease priority (image priority: 1 > llm priority: 2).
  • Restart / allow ollama to reload on lease release, same as game mode's release path (restarted_units: [ollama.service]).
  • Alternatively/additionally: drop ollama's resident keep_alive so it doesn't hold ~8 GB indefinitely.

Whichever direction, the invariant should be: a priority: 1 image lease can always reclaim VRAM from a priority: 2 LLM that's merely parked. Manual ollama stop before every generation (current workaround) is exactly the arbitration this service is supposed to do automatically.

Workaround (until fixed)

ssh mRock 'ollama stop gemma3:12b' frees ~8-10 GB; FLUX then generates fine (verified end-to-end 2026-07-06, job completed in ~17s).

Evidence

  • mgpumanager logs: journalctl --user -u mgpumanager.service (mRock)
  • config: ~/dev/mGPUmanager/config/consumers.yaml
  • Related: the flexsiebels imagen-worker DB-reconnect bug (separate ImaGen issue) masked this for ~a month.
## Symptom `/imagine` (flexsiebels) FLUX generations fail with: ``` comfyui[flux-schnell-local]: broker: insufficient VRAM to load the model even after eviction (untracked GPU usage, e.g. a game?): cannot free enough VRAM for comfyui even after evicting every evictable consumer ``` ComfyUI itself is healthy; the mGPUmanager lease request just gets a 503. ## Root cause On a normal image lease, the scheduler evicts every *evictable* consumer but is structurally forbidden from reclaiming ollama's VRAM. From the mgpumanager logs (16 GB RTX 4070, comfyui needs 11000 MiB): ``` evicted consumer victim=mvoice free_mib_after=993 need_mib=11000 evicted consumer victim=whisper-server free_mib_after=3523 need_mib=11000 WARN no eviction candidates target=comfyui need_mib=11000 free_mib=3523 strict=true POST /v1/lease status=503 ``` Free tops out at ~3.5 GB because ollama (gemma3:12b, ~8 GB resident, 24h keep_alive) is never evicted. `config/consumers.yaml` documents this as intentional: ```yaml ollama: vram_managed: true # Does NOT make ollama an eviction victim — the vram_managed guard in the # scheduler still skips it for normal leases. systemd_unit: ollama.service ``` So ollama is only ever stopped in **game mode** (`game_mode.stop_units: [ollama.service]`), never for a normal image lease — even though comfyui is `priority: 1` and ollama is `priority: 2`. The higher-priority consumer loses to the lower-priority one. That's the opposite of what the control plane is for. This started biting after the 2026-07-04 `ollama-after-game` wiring left gemma3:12b sitting resident with a 24h keep_alive; before that ollama wasn't pinning enough VRAM to matter. ## Proposed fix Let the scheduler evict ollama for a higher-priority lease, reusing the mechanism game mode already has: - Add an unload/stop action for ollama on the normal eviction path — either `ollama stop <model>` (soft, keeps the service up) or `systemctl stop ollama.service` (the game-mode approach), gated on lease priority (image `priority: 1` > llm `priority: 2`). - Restart / allow ollama to reload on lease release, same as game mode's release path (`restarted_units: [ollama.service]`). - Alternatively/additionally: drop ollama's resident keep_alive so it doesn't hold ~8 GB indefinitely. Whichever direction, the invariant should be: **a `priority: 1` image lease can always reclaim VRAM from a `priority: 2` LLM that's merely parked.** Manual `ollama stop` before every generation (current workaround) is exactly the arbitration this service is supposed to do automatically. ## Workaround (until fixed) `ssh mRock 'ollama stop gemma3:12b'` frees ~8-10 GB; FLUX then generates fine (verified end-to-end 2026-07-06, job completed in ~17s). ## Evidence - mgpumanager logs: `journalctl --user -u mgpumanager.service` (mRock) - config: `~/dev/mGPUmanager/config/consumers.yaml` - Related: the flexsiebels imagen-worker DB-reconnect bug (separate ImaGen issue) masked this for ~a month.
mAi commented 2026-07-13 18:35:15 +00:00
Author
Copy Link

Implementation spec (m approved 2026-07-13 — "Implement #5 now")

Root cause confirmed in internal/scheduler/evicting.go: three guard points skip VRAMManaged (ollama) so it is never an eviction victim, and unload()'s systemd branch only marks a consumer unloaded without actually stopping it — which for ollama frees nothing (remote clients on :11434 pin the VRAM until the service stops). Recurred live 2026-07-13 20:27: evicted whisper+mvoice, reached 10223 MiB free, 777 short of comfyui's 11000, refused to touch ollama → 503.

Required changes

  1. pickLRUVictim (~L231): a VRAMManaged consumer that has a SystemdUnit (ollama) must be eligible as a victim when the target out-prioritises it (cons.Priority < other.Priority; lower number = higher priority, so image=1 preempts llm=2). Keep skipping VRAMManaged consumers with no stop mechanism.
  2. fitsByBudget (~L257): stop unconditionally skipping VRAMManaged in headroom accounting — mirror the same priority-aware rule so the budget path agrees with the live path.
  3. unload(): for a VRAMManaged + SystemdUnit consumer, actually stop the unit (the mechanism game-mode already uses — see the /v1/gamemode handler in internal/server/server.go that logs restarted_units:[ollama.service] and game_mode.systemctl in config; reuse that exec/polkit path, do NOT invent a new sudo shell-out). Only marking loaded=false is insufficient for ollama.
  4. Restore on release: when the image lease releases (DELETE /v1/lease path), restart any unit this scheduler stopped for that lease — same restart game-mode does on release/TTL. Track stopped-units per lease so a concurrent llm request doesn't leave ollama down forever.
  5. Guard against evicting an in-flight ollama — respect the global lock; only preempt a parked consumer.

Tests

  • pickLRUVictim returns ollama when target=comfyui (priority 1) and ollama (priority 2) is the only loaded consumer holding the needed VRAM.
  • A target with priority >= ollama does NOT evict it.
  • unload on a VRAMManaged+SystemdUnit consumer invokes the stop path (fake/injected executor) and marks not-loaded.
  • Release restarts the stopped unit.

Deploy + live verification (mRock — required before closing)

  1. go build + go test ./... green.
  2. Rebuild binary on mRock, systemctl --user restart mgpumanager.service.
  3. Warm ollama (ollama run bge-m3 "hi"), then trigger a flexsiebels /imagine restyle. Confirm in journalctl --user -u mgpumanager.service: ollama evicted → comfyui lease 200 → image generated → ollama restarted on release.
  4. Comment the commit link + the log excerpt on this issue.

Do NOT close the issue — m closes issues. Set the status:done label and comment when verified.

Note: live VRAM verification requires running on/against mRock (the GPU host); unit tests alone are not sufficient sign-off.

## Implementation spec (m approved 2026-07-13 — "Implement #5 now") Root cause confirmed in `internal/scheduler/evicting.go`: three guard points skip `VRAMManaged` (ollama) so it is never an eviction victim, and `unload()`'s systemd branch only *marks* a consumer unloaded without actually stopping it — which for ollama frees nothing (remote clients on :11434 pin the VRAM until the service stops). Recurred live 2026-07-13 20:27: evicted whisper+mvoice, reached 10223 MiB free, 777 short of comfyui's 11000, refused to touch ollama → 503. ### Required changes 1. **`pickLRUVictim`** (~L231): a `VRAMManaged` consumer that has a `SystemdUnit` (ollama) must be eligible as a victim **when the target out-prioritises it** (`cons.Priority < other.Priority`; lower number = higher priority, so image=1 preempts llm=2). Keep skipping VRAMManaged consumers with no stop mechanism. 2. **`fitsByBudget`** (~L257): stop unconditionally skipping `VRAMManaged` in headroom accounting — mirror the same priority-aware rule so the budget path agrees with the live path. 3. **`unload()`**: for a `VRAMManaged` + `SystemdUnit` consumer, actually **stop the unit** (the mechanism game-mode already uses — see the `/v1/gamemode` handler in `internal/server/server.go` that logs `restarted_units:[ollama.service]` and `game_mode.systemctl` in config; reuse that exec/polkit path, do NOT invent a new sudo shell-out). Only marking `loaded=false` is insufficient for ollama. 4. **Restore on release**: when the image lease releases (DELETE /v1/lease path), **restart** any unit this scheduler stopped for that lease — same restart game-mode does on release/TTL. Track stopped-units per lease so a concurrent llm request doesn't leave ollama down forever. 5. **Guard against evicting an in-flight ollama** — respect the global lock; only preempt a parked consumer. ### Tests - `pickLRUVictim` returns ollama when target=comfyui (priority 1) and ollama (priority 2) is the only loaded consumer holding the needed VRAM. - A target with priority >= ollama does NOT evict it. - `unload` on a VRAMManaged+SystemdUnit consumer invokes the stop path (fake/injected executor) and marks not-loaded. - Release restarts the stopped unit. ### Deploy + live verification (mRock — required before closing) 1. `go build` + `go test ./...` green. 2. Rebuild binary on mRock, `systemctl --user restart mgpumanager.service`. 3. Warm ollama (`ollama run bge-m3 "hi"`), then trigger a flexsiebels `/imagine` restyle. Confirm in `journalctl --user -u mgpumanager.service`: ollama evicted → comfyui lease 200 → image generated → ollama restarted on release. 4. Comment the commit link + the log excerpt on this issue. Do NOT close the issue — m closes issues. Set the `status:done` label and comment when verified. **Note:** live VRAM verification requires running on/against mRock (the GPU host); unit tests alone are not sufficient sign-off.
mAi commented 2026-07-14 16:21:03 +00:00
Author
Copy Link

#5 implementation done + pushed — live-verify blocked on a deploy decision

Branch: mai/hopper/implement-issue-5-evict
Commit: ae16c2a3fe

Implements the m-ratified design (docs/designs/issue-5-ollama-eviction.md §6, all §11 decisions folded in). go build/vet + go test -race ./... all green.

What changed

  • pickLRUVictim + fitsByBudget now share one predicate, isEvictableVictim: a consumer is a victim iff the target strictly out-prioritises it (global priority gate, m Q2), it is not coexistence-exempt, and it has an unload mechanism. A vram_managed consumer (ollama) is eligible iff it carries an unload_strategy.
  • unload() — new config-discriminated ollama soft-unload (unload_strategy: ollama_keepalive): GET /api/ps enumerates the resident models (never hardcoded — dynamic set), then POST /api/generate {"model":"…","keep_alive":0} per model, then marks unloaded. Zero privilege surface; reuses the 30 s e.client. In-flight-safe via ollama's per-model serialisation.
  • Restore: none — lazy cold-reload via existing ensureLoaded on the next /v1/llm (m Q3).
  • config/consumers.yaml: unload_strategy: ollama_keepalive on ollama, stale vram_managed comment fixed; no systemd_unit added (m Q1). keep_alive shortening (m Q4) is an ollama-side/env change — see note below.
  • Tests: isEvictableVictim table; priority-gate victim selection; ollama soft-unload (multi-model + empty /api/ps); end-to-end lease eviction; and a new refuse-higher-priority-victim invariant test.

Behaviour note (consequence of the global priority gate, m Q2)

The gate applies to all targets, so a lower-priority consumer can no longer evict a higher-priority one. Concretely, a TTS (mvoice, priority 3) request will no longer evict a resident comfyui (priority 1) image job — TTS now yields to the higher-priority image work instead of killing it. This is the intended invariant, but it is a real change from the old priority-blind eviction (two existing smoke tests encoded the old direction and were reworked to the correct one).

keep_alive (m Q4)

The complementary shortening of ollama's resident keep_alive is ollama-side (env/service), not mgpumanager config. Suggested value: OLLAMA_KEEP_ALIVE=10m (down from 24 h) — long enough to keep a warm model across a normal chat burst, short enough that idle ollama self-releases without needing active eviction. Flagging for m to set on mRock's ollama unit; not applied by this commit.

§9 whisper-server tech-debt → filed separately as #8

The unload() systemd branch has the same latent mark-unloaded-without-stopping defect. Filed as #8 (not folded in, per scope). Whether evicting whisper actually frees VRAM will be confirmed during live verification.

⚠️ Live verification (§8) is BLOCKED — deploy decision needed

mRock is running the unmerged feat/game-mode branch in production (binary has the internal/gamemode package, /v1/gamemode → 200, config has a game_mode: section + systemd_unit: ollama.service). main (this PR's base) has no game-mode, so deploying a plain main-based #5 binary would regress live game-mode. Not doing that.

Good news: #5 and game-mode are cleanly compatible (design §4). A trial merge of origin/feat/game-mode into this branch had a single trivial config conflict (resolved so ollama carries both unload_strategy and systemd_unit); the combined tree builds and passes go test -race ./... including the gamemode package. game-mode does not touch evicting.go.

Escalated to head for the deploy-strategy decision (recommended: deploy an integration binary = main + #5 + feat/game-mode, which preserves game-mode and adds #5, then run the full mRock live-verify). Will post the journalctl evidence and set status:done once the live path is confirmed on mRock.

(Also noted for deploy: mRock's live config has comfyui vram_resident_mib: 11000, a host-side drift — both branches say 13000.)

## #5 implementation done + pushed — live-verify blocked on a deploy decision **Branch:** `mai/hopper/implement-issue-5-evict` **Commit:** https://mgit.msbls.de/m/mGPUmanager/commit/ae16c2a3fe1b1942e208f0e3ce9f48a2baff7460 Implements the m-ratified design (`docs/designs/issue-5-ollama-eviction.md` §6, all §11 decisions folded in). `go build/vet` + `go test -race ./...` all green. ### What changed - **`pickLRUVictim` + `fitsByBudget`** now share one predicate, `isEvictableVictim`: a consumer is a victim iff the target **strictly out-prioritises** it (global priority gate, m Q2), it is **not** coexistence-exempt, and it has an **unload mechanism**. A `vram_managed` consumer (ollama) is eligible iff it carries an `unload_strategy`. - **`unload()`** — new config-discriminated ollama soft-unload (`unload_strategy: ollama_keepalive`): `GET /api/ps` enumerates the resident models (**never hardcoded** — dynamic set), then `POST /api/generate {"model":"…","keep_alive":0}` per model, then marks unloaded. Zero privilege surface; reuses the 30 s `e.client`. In-flight-safe via ollama's per-model serialisation. - **Restore:** none — lazy cold-reload via existing `ensureLoaded` on the next `/v1/llm` (m Q3). - **`config/consumers.yaml`:** `unload_strategy: ollama_keepalive` on ollama, stale `vram_managed` comment fixed; no `systemd_unit` added (m Q1). keep_alive shortening (m Q4) is an ollama-side/env change — see note below. - **Tests:** `isEvictableVictim` table; priority-gate victim selection; ollama soft-unload (multi-model + empty `/api/ps`); end-to-end lease eviction; and a new **refuse-higher-priority-victim** invariant test. ### Behaviour note (consequence of the global priority gate, m Q2) The gate applies to **all** targets, so a lower-priority consumer can no longer evict a higher-priority one. Concretely, a TTS (mvoice, priority 3) request will **no longer evict** a resident comfyui (priority 1) image job — TTS now yields to the higher-priority image work instead of killing it. This is the intended invariant, but it is a real change from the old priority-blind eviction (two existing smoke tests encoded the old direction and were reworked to the correct one). ### keep_alive (m Q4) The complementary shortening of ollama's resident keep_alive is **ollama-side** (env/service), not mgpumanager config. Suggested value: **`OLLAMA_KEEP_ALIVE=10m`** (down from 24 h) — long enough to keep a warm model across a normal chat burst, short enough that idle ollama self-releases without needing active eviction. Flagging for m to set on mRock's ollama unit; not applied by this commit. ### §9 whisper-server tech-debt → filed separately as #8 The `unload()` systemd branch has the same latent mark-unloaded-without-stopping defect. Filed as #8 (not folded in, per scope). Whether evicting whisper actually frees VRAM will be confirmed during live verification. ### ⚠️ Live verification (§8) is BLOCKED — deploy decision needed mRock is running the **unmerged `feat/game-mode` branch** in production (binary has the `internal/gamemode` package, `/v1/gamemode` → 200, config has a `game_mode:` section + `systemd_unit: ollama.service`). `main` (this PR's base) has no game-mode, so deploying a plain main-based #5 binary would **regress live game-mode**. Not doing that. Good news: #5 and game-mode are cleanly compatible (design §4). A trial merge of `origin/feat/game-mode` into this branch had a single trivial config conflict (resolved so ollama carries **both** `unload_strategy` and `systemd_unit`); the combined tree builds and passes `go test -race ./...` including the gamemode package. game-mode does not touch `evicting.go`. Escalated to head for the deploy-strategy decision (recommended: deploy an integration binary = main + #5 + feat/game-mode, which preserves game-mode and adds #5, then run the full mRock live-verify). Will post the journalctl evidence and set `status:done` once the live path is confirmed on mRock. _(Also noted for deploy: mRock's live config has `comfyui vram_resident_mib: 11000`, a host-side drift — both branches say 13000.)_
mAi commented 2026-07-14 16:58:32 +00:00
Author
Copy Link

✅ #5 done — implemented, deployed to mRock, and live-verified

On main: 91ef213 (game-mode #3 + #5 + the live-verify fix below)
Commits: bcb84d2 (#5 eviction) · 08459ed (awaitFit settle fix, found during verification)

Live verification on mRock (the GPU host)

Deployed the main binary + config (binary-copy, systemctl --user restart), warmed ollama, then fired a comfyui image lease (kind=image, the same call the flexsiebels restyle issues). Scheduler journal:

evicted consumer  victim=mvoice          target=comfyui
evicted consumer  victim=whisper-server  target=comfyui
evicted consumer  victim=ollama          target=comfyui
http POST /v1/lease status=200 ms=12509
  • comfyui image lease → HTTP 200 (before #5 this was insufficient_vram/503 whenever ollama was resident — the original bug).
  • ollama soft-unloaded: /api/ps went from [gemma3:12b] → []; GPU free rose 2399 → 12104 MiB.
  • ollama cold-reloads on the next /v1/llm (200, gemma3:12b resident again) — lazy restore, no restore code (m Q3). ✔
  • game-mode intact post-deploy: /v1/gamemode → 200. ✔
  • Acquire took 12.5 s (lease wait budget is 120 s — no timeout risk).

Fix found by the live verification (08459ed)

The first live run 503'd even though ollama was evicted: the eviction loop exhausted its victims in three 1-second attempts while the 2 s GPU poller + ollama's async runner-exit lagged, giving up a beat before the VRAM it had just freed showed up in fits(). Fix: after each unload, awaitFit() waits (bounded 2×poll_interval + 1s) for the freed VRAM to settle, succeeding as soon as the target fits. Regression test TestEvictingAwaitsDelayedVRAMSettle reproduces the async-settle case. go build + go test -race ./... green. This is exactly why §8 mandates real-hardware verification — a unit test with a synchronous VRAM stub could not have caught it.

keep_alive (m Q4)

Set OLLAMA_KEEP_ALIVE 24h → 10m on mRock's ollama.service drop-in (backed up), daemon-reload + restart, effective value confirmed. Complementary defense-in-depth: idle ollama now self-releases in ~10 min, reducing how often active eviction is needed.

Related: #8 (whisper-server)

The live run also confirmed #8: evicting whisper-server freed 0 VRAM (marks unloaded without stopping the unit). Evidence posted on #8. Did not fold in, per scope.

Scope note

The broker's job — grant a comfyui image lease by reclaiming VRAM from a parked lower-priority ollama — is what #5 changed and what is verified above. Actual image-pixel generation is comfyui's unchanged downstream step once it holds the lease.

## ✅ #5 done — implemented, deployed to mRock, and live-verified **On main:** `91ef213` (game-mode #3 + #5 + the live-verify fix below) **Commits:** [`bcb84d2`](https://mgit.msbls.de/m/mGPUmanager/commit/bcb84d2) (#5 eviction) · [`08459ed`](https://mgit.msbls.de/m/mGPUmanager/commit/08459ed) (awaitFit settle fix, found during verification) ### Live verification on mRock (the GPU host) Deployed the main binary + config (binary-copy, `systemctl --user restart`), warmed ollama, then fired a comfyui image lease (`kind=image`, the same call the flexsiebels restyle issues). Scheduler journal: ``` evicted consumer victim=mvoice target=comfyui evicted consumer victim=whisper-server target=comfyui evicted consumer victim=ollama target=comfyui http POST /v1/lease status=200 ms=12509 ``` - **comfyui image lease → HTTP 200** (before #5 this was `insufficient_vram`/503 whenever ollama was resident — the original bug). - **ollama soft-unloaded**: `/api/ps` went from `[gemma3:12b]` → `[]`; GPU free rose **2399 → 12104 MiB**. - **ollama cold-reloads** on the next `/v1/llm` (200, gemma3:12b resident again) — lazy restore, no restore code (m Q3). ✔ - **game-mode intact** post-deploy: `/v1/gamemode` → 200. ✔ - Acquire took 12.5 s (lease wait budget is 120 s — no timeout risk). ### Fix found *by* the live verification (`08459ed`) The first live run 503'd **even though ollama was evicted**: the eviction loop exhausted its victims in three 1-second attempts while the 2 s GPU poller + ollama's async runner-exit lagged, giving up a beat before the VRAM it had just freed showed up in `fits()`. Fix: after each unload, `awaitFit()` waits (bounded `2×poll_interval + 1s`) for the freed VRAM to settle, succeeding as soon as the target fits. Regression test `TestEvictingAwaitsDelayedVRAMSettle` reproduces the async-settle case. `go build` + `go test -race ./...` green. **This is exactly why §8 mandates real-hardware verification — a unit test with a synchronous VRAM stub could not have caught it.** ### keep_alive (m Q4) Set `OLLAMA_KEEP_ALIVE` **24h → 10m** on mRock's `ollama.service` drop-in (backed up), daemon-reload + restart, effective value confirmed. Complementary defense-in-depth: idle ollama now self-releases in ~10 min, reducing how often active eviction is needed. ### Related: #8 (whisper-server) The live run also **confirmed #8**: evicting whisper-server freed 0 VRAM (marks unloaded without stopping the unit). Evidence posted on #8. Did not fold in, per scope. ### Scope note The broker's job — grant a comfyui image lease by reclaiming VRAM from a parked lower-priority ollama — is what #5 changed and what is verified above. Actual image-pixel generation is comfyui's unchanged downstream step once it holds the lease.
mAi commented 2026-07-29 09:35:23 +00:00
Author
Copy Link

#5 re-verified on current main — invariant held only until the first eviction

This shift started from the assumption that #5 was open. It is not: the soft-unload eviction path shipped on 91ef213 and is present on main (2a94b1d). I verified it against the code rather than the earlier report — priority gate, soft-unload lever, lazy restore and awaitFit() are all in place, and go build ./... && go vet ./... && go test -race ./... is green.

But the stated invariant — a priority: 1 image lease can always reclaim VRAM from a parked priority: 2 LLM — did not hold beyond the first eviction.

Gap

unloadOllama sets e.loaded["ollama"] = false. Nothing sets it back unless a request arrives through the broker's /v1/llm (ensureLoaded). ollama's remote clients connect to :11434 directly and bypass the broker — that is the documented reason game mode needs its own systemctl lever (config/consumers.yaml, game_mode: block).

So, after any eviction:

  1. comfyui lease soft-unloads ollama → loaded=false.
  2. mRiver (or any direct client) hits :11434 → gemma3:12b resident again, ~8 GB. The broker never learns.
  3. Next comfyui lease → pickLRUVictim skips ollama because !loaded → no victim → ErrInsufficientVRAM → 503.

That is this issue's original symptom, one eviction later. It is reproducible as a unit test: without the fix, TestEvictingReclaimsOllamaReloadedOffBroker fails with eviction: insufficient_vram.

Fix

Read ollama's own resident set (GET /api/ps) before choosing victims, instead of trusting broker-local bookkeeping the broker cannot keep accurate. New reconcileVRAMManaged() runs in ensureFits only when the target does not already fit, and corrects the flag in both directions:

  • resident off-broker → ollama becomes a victim again (closes the gap);
  • holding nothing → ollama is no longer picked, so it stops burning an eviction attempt plus a full awaitFit settle window on VRAM that was never held.

A failed probe leaves the existing belief untouched — a briefly unreachable ollama must not silently drop out of, or into, the victim set. The /api/ps enumeration moved out of unloadOllama into ollamaResidentModels so eviction and reconciliation read one ground truth.

Eviction is not widened. The probe only corrects residency; the priority gate in isEvictableVictim is untouched, and a test pins that an equal-priority lease still leaves a resident ollama alone.

Tests

Test Asserts
TestEvictingReclaimsOllamaReloadedOffBroker priority-1 lease evicts a vram_managed priority-2 ollama reloaded off-broker (fails without the fix)
TestEvictingRefusesOllamaForEqualPriorityLease equal-priority lease does not evict it, even though the probe sees it resident
TestEvictingSkipsOllamaHoldingNothing no eviction counted for a consumer holding nothing

go build ./..., go vet ./..., go test -race ./... — all clean.

Commit

Branch mai/tesla/mgpumanager-5-evict
Commit: https://mgit.msbls.de/m/mGPUmanager/commit/601f5e2

Earlier #5 commits, for the record: bcb84d2 (soft-unload), 08459ed (awaitFit).

Not deployed — no mRock rollout from this shift, per instruction. Rollout is routed by the head.

Note: this repo has zero labels and mAi has read permission on it, so no status:* label can be set here. Unchanged since the last shift; needs m to grant mAi write collaborator.

## #5 re-verified on current main — invariant held only until the first eviction This shift started from the assumption that #5 was open. It is not: the soft-unload eviction path shipped on `91ef213` and is present on `main` (`2a94b1d`). I verified it against the code rather than the earlier report — priority gate, soft-unload lever, lazy restore and `awaitFit()` are all in place, and `go build ./... && go vet ./... && go test -race ./...` is green. But the stated invariant — *a `priority: 1` image lease can **always** reclaim VRAM from a parked `priority: 2` LLM* — did **not** hold beyond the first eviction. ### Gap `unloadOllama` sets `e.loaded["ollama"] = false`. Nothing sets it back unless a request arrives through the broker's `/v1/llm` (`ensureLoaded`). ollama's remote clients connect to `:11434` **directly and bypass the broker** — that is the documented reason game mode needs its own `systemctl` lever (`config/consumers.yaml`, `game_mode:` block). So, after any eviction: 1. comfyui lease soft-unloads ollama → `loaded=false`. 2. mRiver (or any direct client) hits `:11434` → gemma3:12b resident again, ~8 GB. The broker never learns. 3. Next comfyui lease → `pickLRUVictim` skips ollama because `!loaded` → no victim → `ErrInsufficientVRAM` → **503**. That is this issue's original symptom, one eviction later. It is reproducible as a unit test: without the fix, `TestEvictingReclaimsOllamaReloadedOffBroker` fails with `eviction: insufficient_vram`. ### Fix Read ollama's own resident set (`GET /api/ps`) before choosing victims, instead of trusting broker-local bookkeeping the broker cannot keep accurate. New `reconcileVRAMManaged()` runs in `ensureFits` only when the target does not already fit, and corrects the flag in **both** directions: - resident off-broker → ollama becomes a victim again (closes the gap); - holding nothing → ollama is no longer picked, so it stops burning an eviction attempt plus a full `awaitFit` settle window on VRAM that was never held. A failed probe leaves the existing belief untouched — a briefly unreachable ollama must not silently drop out of, or into, the victim set. The `/api/ps` enumeration moved out of `unloadOllama` into `ollamaResidentModels` so eviction and reconciliation read one ground truth. **Eviction is not widened.** The probe only corrects residency; the priority gate in `isEvictableVictim` is untouched, and a test pins that an equal-priority lease still leaves a resident ollama alone. ### Tests | Test | Asserts | |---|---| | `TestEvictingReclaimsOllamaReloadedOffBroker` | priority-1 lease evicts a `vram_managed` priority-2 ollama reloaded off-broker (fails without the fix) | | `TestEvictingRefusesOllamaForEqualPriorityLease` | equal-priority lease does **not** evict it, even though the probe sees it resident | | `TestEvictingSkipsOllamaHoldingNothing` | no eviction counted for a consumer holding nothing | `go build ./...`, `go vet ./...`, `go test -race ./...` — all clean. ### Commit Branch `mai/tesla/mgpumanager-5-evict` Commit: https://mgit.msbls.de/m/mGPUmanager/commit/601f5e2 Earlier #5 commits, for the record: [`bcb84d2`](https://mgit.msbls.de/m/mGPUmanager/commit/bcb84d2) (soft-unload), [`08459ed`](https://mgit.msbls.de/m/mGPUmanager/commit/08459ed) (`awaitFit`). **Not deployed** — no mRock rollout from this shift, per instruction. Rollout is routed by the head. Note: this repo has zero labels and mAi has `read` permission on it, so no `status:*` label can be set here. Unchanged since the last shift; needs m to grant mAi write collaborator.
mAi commented 2026-07-29 09:37:09 +00:00
Author
Copy Link

Merged to main as e58da14 (merge of 601f5e2, branch mai/tesla/mgpumanager-5-evict).

Head review: build, go vet, and go test -race -count=1 ./... all clean on the branch before merge; branch was not behind main. Two-dot diff touches only internal/scheduler/evicting.go and internal/scheduler/evicting_test.go.

Not yet deployed to mRock. Rollout note for whoever does it: mRock's consumers.yaml carries a host-side hand-edit comfyui vram_resident_mib=11000 while the repo says 13000. Deploy is a binary copy, not a git pull — preserve that host value.

Issue stays open until the fix is verified live on mRock.

Merged to main as e58da14 (merge of 601f5e2, branch `mai/tesla/mgpumanager-5-evict`). Head review: build, `go vet`, and `go test -race -count=1 ./...` all clean on the branch before merge; branch was not behind main. Two-dot diff touches only `internal/scheduler/evicting.go` and `internal/scheduler/evicting_test.go`. Not yet deployed to mRock. Rollout note for whoever does it: mRock's `consumers.yaml` carries a host-side hand-edit `comfyui vram_resident_mib=11000` while the repo says 13000. Deploy is a binary copy, not a git pull — preserve that host value. Issue stays open until the fix is verified live on mRock.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
m
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: m/mGPUmanager#5
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?