AI Server Workstation

Objective

Stand up a dedicated, separate physical machine (not the existing homelab Proxmox host) to run Proxmox as a platform for AI workloads hosting local LLMs through Ollama and Open WebUI in one container, alongside a second VM with its own dedicated GPU. That second slot was originally a Windows VM and has since been reassigned to a GPU-accelerated Hackintosh macOS VM.

Current status

The AI half is deployed and stable. The failed original GPU has been replaced with a 24 GB Tesla P40, now installed with ECC enabled and confirmed running local models fully on GPU. An Ubuntu Server LXC runs Ollama natively alongside Open WebUI in Docker, and the container-start GPU race condition is now permanently fixed and verified across a full host reboot.

Architecture

Completed work

• Chose Proxmox on bare metal over Windows + Docker Desktop, for near-zero container overhead and a clean one-GPU-per-role split.
• Selected and confirmed a motherboard with the slot layout needed for two full-length GPUs without lane contention.
• Deployed a dedicated Ubuntu Server LXC with GPU device passthrough (cgroup rules and device bind mounts rather than full VFIO) for the inference container.
• Worked through several non-obvious deployment issues: a driver-install method that conflicted with Proxmox's own packages, a power-management setting that was silently capping inference speed by more than an order of magnitude, and a device-initialisation race between container start and GPU driver readiness.
• Diagnosed a genuine GPU hardware failure from driver and kernel logs distinguishing a real defect from a configuration problem and made the call to replace the card rather than keep troubleshooting it.
• Confirmed two local models (a coding-focused model and a general-purpose model) running with GPU-accelerated inference at expected speeds.
• Installed the replacement Tesla P40 and enabled ECC. Diagnosed a probe failure caused by the newest NVIDIA driver branch having dropped support for older GPU architectures, and fixed it by moving to the legacy branch that still supports them.
• Permanently fixed the container/GPU-readiness race with a systemd unit that creates the GPU device nodes before the container starts, verified across a full host reboot with no manual steps.
• Confirmed community models pulled from Hugging Face run on the new GPU, and tuned per-model context length so longer conversations no longer overflow.

Risks and open items

• Installed RAM is about half the original target, enough for inference but not for the planned macOS VM; an upgrade is needed first.
• The planned second GPU isn't sourced yet, and the hardware in that slot doesn't match the plan.
• A higher-power second GPU leaves the current PSU with thin headroom; a larger unit is worth considering before both cards run together.
• Integration with the existing homelab's identity and reverse-proxy layer is an open decision.

Next actions

1. Confirm the second-GPU plan, then source and install it.
2. Upgrade host RAM and review PSU sizing before the second GPU goes in.
3. Install the second storage drive.
4. Build the macOS VM with full GPU passthrough.
5. Decide how this host integrates with the existing homelab's identity and networking layers.

Related