Dylan Banta
← Case studies

Case study 05 · 2 min read

GPU / AI host

Shared GPUs for local AI, voice and media on one homelab VM

Role
Built and run it for my own use
When
2022 – present
Stack
XCP-ngNVIDIA CDIDocker ComposeOllamaOpen WebUIComfyUIn8npgvectorUnslothfaster-whisper

A fuller write-up of this host is coming. This page covers what it runs and why.

TL;DR
  • A dedicated VM with GPU passthrough: an RTX 3060 Ti for AI workloads and two Tesla M60s for media transcoding.
  • Containers get GPUs through NVIDIA CDI. A lock file coordinates which service is using a card.
  • Siggy-bot, a Discord voice assistant, runs its speech-to-text and agent workflow on this host.

The problem

I wanted to run AI workloads locally instead of renting them: chat models, image generation, speech-to-text and small fine-tuning jobs. I also needed hardware video transcoding for my media server.

Those workloads all want a GPU, they run in different containers, and some of them will happily take all of a card's memory. I also didn't want to give every other VM on the hypervisor access to the GPUs.

Architecture

All of it runs on one VM on my XCP-ng host, with the GPUs passed through to it. Everything else on that hypervisor stays GPU-free.

Who uses which card
RTX 3060 Ti
then
Ollama + Open WebUI local LLM chat
then
ComfyUI with an autoheal container
then
Unsloth fine-tuning
2× Tesla M60
then
Media server hardware transcoding

Containers get access through NVIDIA CDI (the Container Device Interface), and I keep a script that enables it. The M60s are older datacenter cards, so I also keep a script for tuning their clocks.

Several services can want the same card, so a GPU lock file coordinates access between them. ComfyUI has an autoheal container that restarts it when its health check fails, plus a guard config for the GPU. Some stacks also have OpenTelemetry compose files for tracing.

Siggy-bot is a Discord voice bot built as separate services:

Siggy-bot
Wake word
then
Speech-to-text faster-whisper service
then
n8n agent workflow
then
Text-to-speech reply

Speech-to-text is its own small API service, so other projects can use it too. The bot only handles the Discord side. The thinking happens in an n8n workflow, backed by Postgres with pgvector and Redis.

Key decisions

Pass the GPUs to one VM, not to everything. One VM owns the cards, and anything that needs a GPU lives there, whichever of my domains it belongs to. A lock file coordinates jobs so they don't fight over the cards.

Split the bot into services. Separating speech-to-text, the agent and the bot means I can swap the model or workflow without touching the Discord code.

Self-heal the flaky piece. Image generation is the workload most likely to wedge, so it gets automatic restarts instead of me noticing hours later.

Training a model on my own likeness

The one image-generation project worth mentioning: in 2023–24 I fine-tuned early Stable Diffusion models on photos of myself, so they could generate images with my likeness. I trained the full model, not a LoRA. It worked, with results typical of models from that era.

If I did it today, I'd train a LoRA instead. It's a small add-on file rather than a whole new model, it trains faster, and it can be combined with other LoRAs and base models.

Results

  • Local LLM chat, image generation and fine-tuning run on my own hardware, alongside media transcoding.
  • This started in 2022 with a single AUTOMATIC1111 Stable Diffusion Web UI container. It grew into a dedicated VM with GPU passthrough, CDI and coordinated access.
  • Siggy-bot answers voice questions in Discord using my own speech-to-text service and an n8n workflow.

I actually use n8n more at work than at home. There it drives SIS imports, device workflows and IT automation, and an MCP server lets Claude build and edit workflows with me. That's covered in the WM.Tools case study.