comfyui's VRAM need is not a constant: it varies cold-vs-warm (8410 vs 6170 MiB) and with the requested resolution #12

New Issue

mAi · 2026-07-29T10:06:38Z

mAi commented

2026-07-29 10:06:38 +00:00

Split out of #11. #11 is "the constant is wrong" and is being fixed by setting it to a correct value. This issue is the finding underneath it: a single constant cannot express what comfyui actually needs, so the fix to #11 is a better approximation, not a resolution.

Measured on mRock

Real flux1-schnell generations, 1024x1024, fp8_e4m3fn (ImaGen's own defaults, read from comfyui.go:270-271 and :651). comfyui's PID sampled every 0.5 s.

Cold run, comfyui idle at 346 MiB beforehand — full trace:

346 346 346 346 | 5626 5626 5626 5626 | 5146 | 2074 | 7706 | 8410 x10 | 7706 7194 6906 | 4154 4154 4154 4154

Cold peak: 8410 MiB. Generation finished in 12 s.
Warm peak: 6170 MiB — second run, new seed and prompt, weights already cached.
Settled residency: 4154 MiB after either run.

Why #11's number was misleading

#11 quotes 4346 MiB as the measured requirement. That is the settled figure — it was sampled after the generation, so it records what comfyui keeps cached, not what it needs while working. 4154 MiB in the run above is the same number within noise.

The budget has to cover the peak, because the peak is where an OOM happens. This is why the host-side value of 6000 is not the safe unblock it appears to be: it admits a cold job that will peak at 8410 MiB. It has not bitten yet only because free VRAM has been ample. It is a live OOM window, not a fix.

The two axes a constant cannot cover

Cold vs warm — 8410 vs 6170 MiB. The cold figure governs the broker's decision, because the broker only evicts when comfyui is not already resident; that is the cold path by definition. But charging an already-resident comfyui the cold figure permanently reserves cold-sized VRAM for a warm consumer.
Requested resolution. ImaGen passes width and height through per request — they are template variables, not fixed. A caller asking for more than 1024x1024 moves the peak again, and nothing in the current model can express that.

#11's point 2 anticipated per-model budgets. The measurements say the real axis is per-request, plus cold-vs-warm.

Direction, not a decided design

Two pieces, roughly independent:

Cold vs warm is the tractable half and depends on #4/#6. The broker cannot currently distinguish the two cases, because gpu_resident_mib reads 0 for every consumer — residency is believed, not measured. Once #4/#6 make residency a fact, gating the cold path on the cold figure and letting an already-resident comfyui pass on the warm one becomes expressible. Building it before that would rest a decision on a flag standing in for a fact, which is the class of bug #5 removed.
Per-request sizing is the larger question: whether a lease should be able to declare its expected VRAM (derived from resolution and model), rather than the consumer declaring one number for every request it will ever serve.

Why file it rather than fold it into #11

#11 gets closed by a correct constant, and the constant is genuinely better than what it replaces. If this finding stays inside #11 it disappears with it, and the next person to hit an OOM at a resolution nobody measured starts the same investigation from zero.

Provenance

Measured by tesla (mgpumanager/gitster) while implementing #11 on 2026-07-29, after being asked to re-measure the 4346 figure rather than take it on trust. Full trace and reasoning: #11 comment 17078. Left unassigned — depends on #4/#6 landing first.

Related: #11 (the constant), #4 and #6 (per-consumer residency accounting), ImaGen #15 (the lease consumer that passes resolution through).

Split out of #11. #11 is "the constant is wrong" and is being fixed by setting it to a correct value. This issue is the finding underneath it: **a single constant cannot express what comfyui actually needs**, so the fix to #11 is a better approximation, not a resolution. ## Measured on mRock Real `flux1-schnell` generations, 1024x1024, fp8_e4m3fn (ImaGen's own defaults, read from `comfyui.go:270-271` and `:651`). comfyui's PID sampled every 0.5 s. Cold run, comfyui idle at 346 MiB beforehand — full trace: ``` 346 346 346 346 | 5626 5626 5626 5626 | 5146 | 2074 | 7706 | 8410 x10 | 7706 7194 6906 | 4154 4154 4154 4154 ``` - **Cold peak: 8410 MiB.** Generation finished in 12 s. - **Warm peak: 6170 MiB** — second run, new seed and prompt, weights already cached. - **Settled residency: 4154 MiB** after either run. ## Why #11's number was misleading #11 quotes 4346 MiB as the measured requirement. That is the **settled** figure — it was sampled after the generation, so it records what comfyui keeps cached, not what it needs while working. 4154 MiB in the run above is the same number within noise. The budget has to cover the **peak**, because the peak is where an OOM happens. This is why the host-side value of 6000 is not the safe unblock it appears to be: it admits a cold job that will peak at 8410 MiB. It has not bitten yet only because free VRAM has been ample. It is a live OOM window, not a fix. ## The two axes a constant cannot cover 1. **Cold vs warm — 8410 vs 6170 MiB.** The cold figure governs the broker's decision, because the broker only evicts when comfyui is *not* already resident; that is the cold path by definition. But charging an already-resident comfyui the cold figure permanently reserves cold-sized VRAM for a warm consumer. 2. **Requested resolution.** ImaGen passes width and height through per request — they are template variables, not fixed. A caller asking for more than 1024x1024 moves the peak again, and nothing in the current model can express that. #11's point 2 anticipated *per-model* budgets. The measurements say the real axis is **per-request, plus cold-vs-warm**. ## Direction, not a decided design Two pieces, roughly independent: - **Cold vs warm** is the tractable half and depends on #4/#6. The broker cannot currently distinguish the two cases, because `gpu_resident_mib` reads 0 for every consumer — residency is believed, not measured. Once #4/#6 make residency a fact, gating the cold path on the cold figure and letting an already-resident comfyui pass on the warm one becomes expressible. Building it before that would rest a decision on a flag standing in for a fact, which is the class of bug #5 removed. - **Per-request sizing** is the larger question: whether a lease should be able to declare its expected VRAM (derived from resolution and model), rather than the consumer declaring one number for every request it will ever serve. ## Why file it rather than fold it into #11 #11 gets closed by a correct constant, and the constant is genuinely better than what it replaces. If this finding stays inside #11 it disappears with it, and the next person to hit an OOM at a resolution nobody measured starts the same investigation from zero. ## Provenance Measured by tesla (mgpumanager/gitster) while implementing #11 on 2026-07-29, after being asked to re-measure the 4346 figure rather than take it on trust. Full trace and reasoning: #11 comment 17078. Left unassigned — depends on #4/#6 landing first. Related: #11 (the constant), #4 and #6 (per-consumer residency accounting), ImaGen #15 (the lease consumer that passes resolution through).

mAi referenced this issue

2026-07-29 10:07:12 +00:00

STT glossary drift: consumers.yaml `stt_prompt` is hand-maintained and diverges from mVoice lexicon.yaml (missing mBrian → 'mBryan') #10

m referenced this issue from a commit

2026-07-29 10:13:21 +00:00

mAi: #11 - set comfyui's budget to the measured cold peak (9000)

mAi referenced this issue

2026-07-29 10:16:35 +00:00

comfyui's vram_budget_mib is 11000, measured usage is 4346 — the broker refuses image jobs that would fit comfortably #11

mAi commented

2026-07-29 10:19:46 +00:00

Why measuring harder will not answer this

This issue argues a single constant cannot express a requirement that varies by cold/warm and resolution. The measurements say something stronger, and anyone who tries to close #12 by measuring more carefully should read this first.

comfyui's peak VRAM tracks how much VRAM was free, not what the job needs. Three real flux1-schnell runs on mRock, 1024x1024, fp8_e4m3fn, sampling comfyui's PID every 0.5 s:

Run	Free VRAM available	comfyui peak
cold	~9000	8410
warm (weights cached)	~11000	6170
cold	10875	10458

All three completed successfully. More free VRAM produced a higher peak, not a faster run. PyTorch's caching allocator grows to fit what is available and does not hand memory back, so a peak reading is a measurement of the headroom that existed when you looked.

The consequence for this issue:

A peak measurement cannot establish a requirement. Every "measured requirement" so far has really been a measurement of the free VRAM at the time. That includes my own 8410, which I stated in #11 as what comfyui needs. It is not; I retract that framing.
The only lower-bound evidence we have is the warm run at 6170, because that run was constrained by cached weights rather than by availability. The true minimum is at most 6170 and is otherwise unobserved.
So vram_resident_mib is an admission threshold, not a requirement. It answers "how much headroom before we let a job in", which is a policy question, not "how much does the model need", which is a measurement question. Conflating the two is what produced both wrong values in #11: 13000 refused jobs that fit, and 6000 came from a settled reading (4154-4346, taken after the run) that recorded cached weights.

What would actually answer it

Not more sampling. Either constrain the allocator and observe the floor — run under a capped budget (PYTORCH_CUDA_ALLOC_CONF, comfyui's own lowvram/novram modes) and find the smallest cap at which the workflow still completes — or stop asking the question and let the admission threshold be an explicit policy value with the axes named next to it. The second is cheaper and honest; the first produces a real number if one is ever needed.

Known gap, stated so it is not mistaken for a clean story

During the acceptance run for #11, free VRAM rose ~7.7 GB between the mvoice eviction (free 1165) and the whisper-server eviction (free 8857). Neither consumer holds anything close to that, and ollama was evicted afterwards. The likely explanation is the ollama embedding model's runner exiting on its own, but that was not proven. Flagged deliberately rather than smoothed over.

Evidence and full traces: #11 (comment)

## Why measuring harder will not answer this This issue argues a single constant cannot express a requirement that varies by cold/warm and resolution. The measurements say something stronger, and anyone who tries to close #12 by measuring more carefully should read this first. **comfyui's peak VRAM tracks how much VRAM was free, not what the job needs.** Three real flux1-schnell runs on mRock, 1024x1024, fp8_e4m3fn, sampling comfyui's PID every 0.5 s: | Run | Free VRAM available | comfyui peak | |---|---|---| | cold | ~9000 | 8410 | | warm (weights cached) | ~11000 | 6170 | | cold | 10875 | 10458 | All three completed successfully. More free VRAM produced a *higher* peak, not a faster run. PyTorch's caching allocator grows to fit what is available and does not hand memory back, so a peak reading is a measurement of the headroom that existed when you looked. The consequence for this issue: - **A peak measurement cannot establish a requirement.** Every "measured requirement" so far has really been a measurement of the free VRAM at the time. That includes my own 8410, which I stated in #11 as what comfyui needs. It is not; I retract that framing. - **The only lower-bound evidence we have is the warm run at 6170**, because that run was constrained by cached weights rather than by availability. The true minimum is at most 6170 and is otherwise unobserved. - **So `vram_resident_mib` is an admission threshold, not a requirement.** It answers "how much headroom before we let a job in", which is a policy question, not "how much does the model need", which is a measurement question. Conflating the two is what produced both wrong values in #11: 13000 refused jobs that fit, and 6000 came from a *settled* reading (4154-4346, taken after the run) that recorded cached weights. ### What would actually answer it Not more sampling. Either constrain the allocator and observe the floor — run under a capped budget (`PYTORCH_CUDA_ALLOC_CONF`, comfyui's own lowvram/novram modes) and find the smallest cap at which the workflow still completes — or stop asking the question and let the admission threshold be an explicit policy value with the axes named next to it. The second is cheaper and honest; the first produces a real number if one is ever needed. ### Known gap, stated so it is not mistaken for a clean story During the acceptance run for #11, free VRAM rose ~7.7 GB between the mvoice eviction (free 1165) and the whisper-server eviction (free 8857). Neither consumer holds anything close to that, and ollama was evicted afterwards. The likely explanation is the ollama embedding model's runner exiting on its own, but that was not proven. Flagged deliberately rather than smoothed over. Evidence and full traces: https://mgit.msbls.de/m/mGPUmanager/issues/11#issuecomment-17146

mAi referenced this issue

2026-07-29 10:34:34 +00:00

The broker can reclaim VRAM it did not allocate, but nothing stops a direct client taking it — a bare `ollama run` still OOM-kills whisper #13

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: m/mGPUmanager#12