mAi
ca9bb1773f
scheduler.Evicting wraps the Locked scheduler with the design's LRU-with-coexistence eviction loop. main.go switches to it. Per-job flow: 1. ensureFits — compare cons.vram_resident_mib + 256 MiB cushion against the live nvidia-smi free reading. If insufficient, pick the LRU loaded consumer NOT in cons.can_coexist_with, NOT VRAM-managed (ollama is excluded from eviction by design — it runs its own LRU), and NOT the target itself, then call its unload route. Wait 1s for VRAM to actually free. Repeat up to 5 times. 2. ensureLoaded — if the target was previously unloaded, call its /api/admin/load (mvoice). Consumers without a load route are assumed to cold-start implicitly on first request. 3. inner.Run — global GPU lock + job execution. State: - scheduler-local 'loaded' map + scheduler-local 'lastUsed' map. The registry's health-derived Loaded field is the source of truth for consumers that report it, but we need our own state for the seconds between an unload call and the next probe. - Stats.Evictions counts successful unload calls and surfaces through /v1/status. LRU pick order: - Scheduler-local lastUsed (set on successful Run completion) takes precedence over registry.LastUsed (set on health probes) because the former reflects real GPU work, not health chatter. Zero-time consumers (never used) lose first. Tests: - Already-resident target: no eviction calls. - 13 GiB comfyui evicted to fit 2.8 GiB mvoice → 1 unload + 1 load, Stats.Evictions = 1. - Coexistent consumer (ollama, in mvoice.can_coexist_with) is never picked even if it's the LRU candidate; the non-coexistent comfyui is unloaded instead. Race detector clean. Refs: m/mGPUmanager#1 (Schritt 5).
mGPUmanager
GPU-Inference-Control-Plane für mRock — Scheduler vor TTS/STT/LLM/Image-Gen mit globalem GPU-Lock + LRU-Eviction + einheitlicher /v1-Fassade. Konsumenten: mVoice, whisper-server, Ollama, ComfyUI/FLUX, später Furbotto. Go.
Full design: docs/design.md — Bestandsaufnahme, 10-Alternativen-Survey, Eviction-Algorithmus, Migrationspfad.
Was es macht
Auf mrock:8770 sitzt ein Go-Daemon, der:
/v1/tts,/v1/stt,/v1/llm,/v1/imageals einheitliche Konsumenten-Fassade exponiert,- jede Anfrage durch einen globalen GPU-Scheduler schleust (seriell, Queue),
- bei VRAM-Druck LRU-Eviction über die deklarierten Coexistenz-Gruppen aus
config/consumers.yamlfährt, - in
/v1/statusLive-GPU-Belegung + Consumer-Health + Scheduler-Statistiken zeigt, - niemals stille Fallbacks zurückgibt — Fehler kommen als strukturiertes
{error,message,consumer,retryable}.
Konsumenten-Registry
config/consumers.yaml deklariert pro Consumer:
url,health.{method,path}für Liveness-Probingpaths.<kind>.{method,path}— wie der Broker zu seinem TTS/STT/LLM/Image-Endpoint kommtvram_resident_mib— für die Scheduler-Mathe (Schritt 5)unload.{method,path,body}und optionalload.{method,path}— wie der Broker den Consumer aus dem VRAM räumt / wieder hochfährtcan_coexist_with: [..]— wer parallel resident sein darfpriority(0=low, 4=urgent),max_concurrency
Build + Deploy
make build # ./bin/mgpumanager
make test # go test ./...
make run # lokal gegen ./config/consumers.yaml
make deploy HOST=mrock # rsync + systemd reload + restart
Auf mRock läuft der Daemon als System-Unit (/etc/systemd/system/mgpumanager.service).
Endpoints
| Verb | Pfad | Verhalten |
|---|---|---|
| POST | /v1/tts |
Proxy zu routing.tts-Consumer (default: mvoice /api/synthesize) |
| POST | /v1/stt |
Proxy zu routing.stt-Consumer (default: mvoice /api/transcribe) |
| POST | /v1/llm |
Proxy zu routing.llm-Consumer (default: ollama /api/generate) |
| POST | /v1/image |
Proxy zu routing.image-Consumer (default: comfyui /prompt) |
| GET | /audio/* |
Proxy zu audio_proxy-Consumer (wa.sh fetcht generiertes Audio so) |
| GET | /v1/status |
Live-Snapshot: GPU + Consumer-Health + Scheduler-Stats |
| GET | /healthz |
Broker-Liveness (200 OK) |
Fehler-Schema
Jeder Broker-eigene Fehler hat die Form:
{
"error": "consumer_unreachable",
"message": "upstream mvoice last probe failed: connection refused",
"consumer": "mvoice",
"retryable": true
}
Codes: consumer_unreachable, no_consumer, scheduler_error, bad_consumer_url, bad_request. Pass-through-4xx/5xx vom Consumer landet unverändert beim Client.
Phase 1 Status (Issue #1)
- ✅ Schritt 0 — ComfyUI persistent (
systemd: comfyui.service) - ✅ Schritt 1 —
mvoice /api/admin/{load,unload}(mai/knuth/admin-load-unload @ mVoice) - ✅ Schritt 2 — Routing-Façade +
/v1/status - ✅ Schritt 3 — wa.sh auf Broker umgestellt (m/mAi
mai/knuth/wa-tts-broker) - ✅ Schritt 4 — Queue + globaler GPU-Lock
- ✅ Schritt 5 — Coexistenz-Gruppen + LRU-Eviction