I have a server colocated at a datacenter — a Dell PowerEdge with dual E5-2650 v2 Xeons and 256 GB of RAM. It might be older silicon, but it still packs a punch for CI/CD builds, smaller personal Docker apps, and general compute that doesn't need the latest and greatest. Plus 256 GB of RAM — in this market? Every gigabyte of DRAM I'm not paying cloud rates for is a small win on its own. Inside it are two Tesla P4 GPUs that have been along for the ride for years — installed originally for remote gaming and GPU-accelerated desktop streaming, where the P4's hardware NVENC and 8 GB of VRAM are honestly more than enough to push a few HEVC streams across the wire. That use case stayed light over time, and the cards spent more hours idle than encoding frames. Then everyone else in the world figured out how to run language models locally, the tooling matured, and the bar to actually make use of the cards in their off-hours got low enough that "this weekend" felt realistic.

The problem hasn't been "do I want to use them" — it's been "can I actually pass them through Proxmox cleanly." Tesla cards in a homelab Proxmox host historically meant a weekend of fighting IOMMU groups, vBIOS dumps, and the dreaded Code 43 errors that NVIDIA used to throw at any consumer card touched by virtualization. I'd convinced myself the project would eat too much time. Turns out the modern story is much better, and the whole stack came together in an afternoon.

This post is what that afternoon looked like.

What's a Tesla P4 and Why Does It Matter

The Tesla P4 is a server-class GPU that NVIDIA released in 2016, originally marketed for INT8 inference workloads. It's a half-length, single-slot, 75-watt passively-cooled card with 8 GB of GDDR5 memory and 2,560 CUDA cores on the GP104 die — the same silicon as a GTX 1070 / 1080, but designed to be cooled by the host chassis's fans rather than its own. That's the form factor's whole point: in a 1U / 2U rack chassis with proper front-to-back airflow these cards are happy and silent, and you can stack two or four of them in a single host without a single GPU fan spinning.

Two interesting things about this card in late 2025 / early 2026:

The price. P4s are old enough now that they sell for $80-120 used. For an 8 GB CUDA-capable card with reasonable memory bandwidth, that's astonishingly cheap.

The TDP. 75 watts means no auxiliary power connector — the card runs on PCIe slot power alone. Two of them sustained pulls maybe 150 W under inference load. That's lower than a single mid-range gaming GPU at idle.

What you give up versus modern cards: no Tensor Cores (those came with Volta), no FP8, no NVENC for video transcoding (P4s have a different chip variant for that), and you cap out around 8 GB of VRAM per card. For training, this hardware is a non-starter. For inference of small-to-medium quantized language models, it's surprisingly fine.

The Plan

I wanted local LLM inference for some of my own apps — specifically, the contact-card scanner I'd been hacking on for random.clusterlabs.dev. That tool reads business cards via Tesseract OCR, then tries to extract structured contact details (name, title, organization, phones, emails) from the OCR output. The extraction half was originally pure regex/heuristic parsing. It worked OK on simple cards. It was bad at anything weird: multi-line organisations where a parent and a division both belonged in the org field, job titles that didn't match the handful of keywords I'd hard-coded, names with diacritics or particles, anything where the layout strayed from a clean five-line stack of name / title / org / email / phone.

A small instruction-tuned LLM eats this kind of structured-extraction problem for breakfast. The question was just where to run it. I'd tried running ollama on my k3s cluster's CPU nodes — it worked, but a contact card extraction took ~50 seconds on a qwen2.5:3b-instruct model running on Ivy Bridge-tier CPUs split between other workloads. That's slow enough to feel broken. Sub-3-second extraction would feel instant.

The P4s, sitting idle in a server I was already paying to keep colocated, were the obvious answer. So.

A Previous Attempt — and Why I Went Smaller This Time

This isn't actually my first run at hosting an LLM stack. A while back I stood up OpenWebUI — pretty, polished frontend with auth, multi-user support, conversation history, RAG pipelines, the whole batteries-included experience — on these same two Tesla P4s in this same PowerEdge. And it worked. For a while.

Then it stopped, in the way that the combination of Docker, OpenWebUI, and the host's NVIDIA driver tends to stop — each one updating on its own cadence and making different assumptions about what the others are doing. A driver update on the host would invalidate the nvidia-container-toolkit binding inside the container. A new OpenWebUI image would bump a CUDA baseline that the existing toolkit hadn't caught up to. A Docker daemon update would shift the runtime config just enough that the container would launch fine but nvidia-smi inside it would show "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver." Each round was technically fixable, but each fix was a couple of hours of "why does the host see the cards perfectly but the container can't?" By the third or fourth round, I'd lost interest in playing whack-a-mole. The setup quietly retired itself.

The lesson I took into this round: pick the smallest layer that still does what I need. ollama already exposes a clean HTTP API. nginx as a Bearer-token gate is fifteen lines of config. Cloudflare Tunnel is one daemon with one token. There's no database in the path, no frontend bundle to rot, no compose-file sprawl. If I want a fancier UI later I can build it as a separate concern that talks to the same uniform API — and if that rots, the inference layer keeps working unaffected.

That's also part of why this post is mostly about ollama, nginx, and a handful of qm arguments rather than a fifteen-service stack. The whole appeal of the minimal version is that there's not much surface to maintain.

Proxmox Host Setup

The Dell box runs Proxmox VE 9 as the hypervisor. Verifying the foundations took about five minutes.

First, IOMMU has to be enabled (this is what isolates a PCIe device into its own DMA address space, which is what makes passthrough possible at all):

dmesg | grep -e DMAR -e IOMMU

I was looking for DMAR: IOMMU enabled and Intel(R) Virtualization Technology for Directed I/O. Both showed up — VT-d enabled in BIOS, kernel sees it. If they don't, your fix is in the BIOS / firmware setup, not in software.

Next, IOMMU group layout — does each P4 sit in its own group, with no other devices it has to drag along?

for d in /sys/kernel/iommu_groups/*/devices/*; do
  n=${d#*/iommu_groups/*}; n=${n%%/*}
  printf 'IOMMU Group %3s ' "$n"
  lspci -nns "${d##*/}"
done | sort -V

On the Dell, each P4 was alone in its own group. That's the textbook clean case — Dell's enterprise boards have proper ACS support so each PCIe slot is isolated. If the cards had been in the same group as the chipset or the RAID controller, I'd have needed the ACS-override kernel patch (which is a whole other can of worms).

To bind the cards to vfio-pci so they're available to a guest VM, four small files:

# Make IOMMU options explicit in GRUB
sed -i 's|^GRUB_CMDLINE_LINUX_DEFAULT=.*|GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"|' /etc/default/grub
update-grub

# Load vfio modules at boot
cat >> /etc/modules <<'EOF'
vfio
vfio_iommu_type1
vfio_pci
EOF

# Bind both P4s to vfio-pci by their PCI device ID (covers any future card swap)
cat > /etc/modprobe.d/vfio.conf <<'EOF'
options vfio-pci ids=10de:1bb3 disable_vga=1
EOF

# Belt-and-braces: blacklist nouveau and the host nvidia driver
cat > /etc/modprobe.d/blacklist-nvidia.conf <<'EOF'
blacklist nouveau
blacklist nvidia
EOF

update-initramfs -u -k all
reboot

After reboot, lspci -nnk -d 10de:1bb3 should show Kernel driver in use: vfio-pci for each P4. That was the milestone — the host was no longer touching the cards, and they were available to be claimed by a guest VM.

The VM

I deliberately kept this simple: one Ubuntu Server 24.04 LTS VM with both P4s attached. Could have done Debian, but Ubuntu's NVIDIA tooling story is meaningfully smoother and that mattered for getting going quickly.

Created via Proxmox's qm CLI so the spec was reproducible:

qm create 200 \
  --name ollama-gpu \
  --machine q35 \
  --bios ovmf \
  --efidisk0 local-lvm:1,format=raw,efitype=4m,pre-enrolled-keys=0 \
  --cpu host \
  --cores 12 --sockets 1 \
  --memory 32768 --balloon 0 \
  --scsihw virtio-scsi-single \
  --scsi0 local-lvm:100,iothread=1,discard=on,ssd=1 \
  --net0 virtio,bridge=vmbr0 \
  --hostpci0 0000:41:00.0,pcie=1 \
  --hostpci1 0000:42:00.0,pcie=1 \
  --ostype l26 \
  --agent enabled=1

Three things matter here. q35 machine type and OVMF (UEFI) firmware are required for clean PCIe passthrough — older i440fx doesn't expose enough PCIe topology to the guest. cpu host passes Ivy Bridge's actual feature flags through (AVX, AES-NI), which matters because some software gates on CPU detection. And balloon 0 disables memory ballooning entirely — GPU drivers and DMA passthrough hate it when the kernel tries to reshuffle the guest's memory mid-session.

I also enabled the VM's TPM and Secure Boot, which led to one of the more confusing-but-routine moments of the whole setup. Installing the NVIDIA driver in Ubuntu via apt install nvidia-driver-575-server works fine — but on next reboot, you hit a blue screen called MOK Manager asking you to enroll a Machine Owner Key. The driver was built and signed by DKMS using a self-generated key; Secure Boot won't load an unsigned kernel module, so the public half of that key needs to be enrolled into the firmware's MOK store. You set a one-time password during the apt step, then on the next boot you type it back in to confirm you're enrolling the cert.

After that step, nvidia-smi showed both P4s, 8 GiB of memory each, driver 575.x. The hardware was now fully usable from inside the VM.

ollama in Two Lines

The actual LLM-serving install is famously simple:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5:3b-instruct

The installer detects nvidia-smi and configures GPU support automatically. By default, ollama listens on 127.0.0.1:11434, which is fine if everything that calls it runs in the same VM but useless if anything else on the LAN needs access. So a small systemd drop-in:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_KEEP_ALIVE=5m"
Environment="OLLAMA_NUM_PARALLEL=2"
Environment="OLLAMA_FLASH_ATTENTION=1"

OLLAMA_NUM_PARALLEL=2 lets ollama serve two concurrent inference requests — with two P4s, ollama distributes the requests across the cards rather than queuing them. OLLAMA_KEEP_ALIVE=5m unloads models from VRAM after 5 minutes of idle, which is mostly a courtesy to anything else that might want to run on the GPU. OLLAMA_FLASH_ATTENTION=1 is a free perf win on Pascal+ — lower KV-cache memory and slightly faster inference.

A first smoke test from inside the VM:

curl -s http://localhost:11434/api/generate \
  -d '{"model":"qwen2.5:3b-instruct","prompt":"Say hi in 5 words.","stream":false}' \
  | jq -r '.response'

Cold start, with the model loading from disk into VRAM the first time: 5 seconds. Every subsequent request: under 700 milliseconds. That's the GPU doing its job.

Auth + Public Endpoint via Cloudflare Tunnel

I wanted the API reachable from outside the LAN — specifically, from local-dev sessions where I'm running my apps on my laptop and don't want to round-trip through a VPN. The tradeoff is obvious: any LLM-serving endpoint exposed publicly is a target. I needed Bearer-token auth on the way in.

The pattern I settled on uses three pieces:

  1. nginx, running natively on the VM, listening on :8080. Acts as a reverse proxy between the public side and ollama on :11434, with a map-based directive that checks the Authorization: Bearer <key> header against a regex of valid keys.
  2. A Cloudflare Tunnel, running as a systemd service via the official cloudflared apt package. The tunnel terminates at Cloudflare's edge and forwards https://llm.example.com traffic to localhost:8080 on the VM.
  3. No open ports. The VM has no inbound port forwards. All public traffic enters through the outbound tunnel that cloudflared keeps open to Cloudflare. (I already wrote about why I do this everywhere).

The auth-gate nginx config is short enough to read in one screenful:

# Pipe-delimited regex alternation. Multiple keys = independent revocation.
map $http_authorization $is_authorized {
    default 0;
    "~^Bearer (8c3f...replace-me-please...|d1a2...also-replace-me...)$" 1;
}

server {
    listen 8080;
    server_tokens off;

    # Cloudflare loves to cache things. LLM responses are unique per request
    # and must never be cached. `always` is the magic word — without it,
    # add_header skips error responses.
    add_header Cache-Control "no-store, no-cache, must-revalidate, max-age=0" always;

    location / {
        if ($is_authorized = 0) { return 401; }

        proxy_pass http://127.0.0.1:11434;
        proxy_http_version 1.1;
        proxy_set_header Host $host;

        # Generation calls can take 30+ seconds. Bump the proxy timeouts
        # well above the 60s default or long replies get truncated.
        proxy_read_timeout 5m;
        proxy_send_timeout 5m;
        proxy_buffering off;
    }
}

The keys themselves are generated with openssl rand -hex 32. I keep two active at any time so I can hand one to a new consumer (a laptop, a CI job, an app) and rotate the other independently — revoking a single key means deleting it from the regex alternation and reloading nginx. The whole rotation is two commands.

API Strategy

A few things I deliberately built in from the start:

Multiple keys for multiple consumers. Each app or environment gets its own key. Production apps get one, staging gets another, my laptop gets a third, future apps get fourth/fifth. If any one key leaks, I revoke just that one without disturbing the others. The auth regex is just a pipe-delimited alternation — adding or removing a key is a one-line edit.

Generic Bearer-token interface. Any HTTP client that can set a header can talk to the API. No SDKs, no library lock-in. This matters because in a year I might want to call this API from something I haven't built yet — a Python notebook, a shell script, a different programming language entirely. All they need is Authorization: Bearer <key>.

Same Authorization shape as commercial LLM APIs. Anyone who's worked with OpenAI / Anthropic / Mistral knows the pattern. Reusing it means apps can be written once with OLLAMA_API_KEY env var, and pointed at the local cluster or any commercial provider depending on the day. I haven't actually written that abstraction yet, but the door is open.

Defence in depth on what gets called. ollama exposes /api/pull, /api/delete, /api/copy, /api/push — all of which an attacker holding a valid key could use to mess with my model cache. The next thing on the list is locking those down at the nginx layer (location /api/pull { return 403; }) so the only operations a remote consumer can perform are /api/generate, /api/chat, /api/embeddings, and /api/tags. Pulling and deleting models stays a side-channel operation done via SSH.

No-cache headers everywhere. Cloudflare's edge will happily cache anything with a 200 status and no explicit Cache-Control — which would be catastrophic for an inference endpoint that returns unique content per request. I learned this the hard way during a configuration mistake earlier in the day, when CF cached an nginx-default 404 page and kept serving it for 15 minutes after the underlying issue was fixed. The always flag on add_header is critical: without it, error responses (401, 5xx) skip the no-cache header and stay cacheable.

Performance

The numbers, before and after.

Path Hardware Latency (3B model)
Cluster CPU (Ivy Bridge, 4 vCPU) Storage-tier worker nodes ~50 sec / contact card
GPU (single Tesla P4) The Dell, after this stack ~700 ms / contact card

That's roughly a 70× speed-up on identical work. The card-extraction tool in random.clusterlabs.dev went from "barely usable" to "feels like a feature you'd pay for." Because the cards are old, this hardware is essentially free. Because ollama's interface is so plain, I got from nvidia-smi working to a public API serving real requests in about an hour.

The current model lineup on the box: qwen2.5:3b-instruct for the latency-sensitive contact-card path, qwen2.5:7b-instruct as the default for everything else — still well under five seconds per call on these cards, and the few-shot adherence on structured-output tasks is meaningfully better than 3B in ways that matter when the prompt cares which fields end up where. And qwen2.5-coder:7b for ad-hoc coding completions when I don't want to round-trip through a paid API. Two Pascal cards aren't going to run a 70-billion-parameter model, but for the 3B-14B range — which covers basically every "smart enough to be useful for narrow tasks" model right now — they're more than adequate.

Right-sizing the VM

The qm create command in the section above provisions the guest with 12 cores and 32 GiB of RAM. That was my first cut, and it's the wrong size — too much of both. The reasoning is worth writing down because it's specific to GPU-passthrough VMs and doesn't generalise from "what does a normal VM need."

Models live in VRAM, not host RAM. Once a model is loaded, host RAM holds the ollama daemon (a few hundred MB of RSS), the OS, nginx, and cloudflared — call it 1.5 GiB combined. The one moment the host needs more is the brief window when a model is being mmapped off disk during a load: peaks around the file size, so for qwen2.5:14b (~9 GB on disk in Q4) you want at least that much free, but it's transient. 16 GiB of guest RAM covers every model that fits on these cards, with headroom. The other 16 GiB is locked away from the rest of the colocated PowerEdge for nothing — and because the VM was provisioned with balloon 0 (which is correct for GPU passthrough, since drivers and DMA hate the kernel reshuffling memory underneath them), the host literally can't reclaim it under pressure.

CPU is the same story. With every layer running on the GPU, the cores are doing tokenization, prompt prep, and the bookkeeping around request-response — bursty, brief, never sustained. 6 cores is plenty. Twelve made me feel reassured at allocation time and have spent every minute since at single-digit utilisation. The remainder belongs to CI/CD builds and other workloads that actually need cycles.

The corrected spec, applied without re-creating the VM:

qm shutdown 200
qm set 200 --cores 6 --memory 16384
qm start 200

Keep balloon 0 regardless of pool size. The lesson is simple but easy to miss the first time: GPU-passthrough VMs are sized by what the host can't put on the GPU, which on this stack is almost nothing. The instinct to "give it a fat allocation just in case" defaults to wasteful here.

The Invisible Queue

There's one footgun in this stack that took building a second consumer to notice. OLLAMA_NUM_PARALLEL=2 doesn't mean "the third request gets rejected" — it means the third request gets quietly queued inside ollama, waiting for a slot to open. The queue is FIFO, defaults to 512 deep, and is completely invisible: there's no /api/queue endpoint, no header, no log line that says "you're 4th in line." A queued non-streaming request just emits zero bytes until its turn arrives.

That's a problem because of what's sitting in front of it. Cloudflare's edge gives any non-streaming response about 100 seconds before it returns a 524 to the client. With stream: false (which is what most tool-use callers send — anything using format: "json" for structured output) there are no intermediate bytes either, just the single final JSON. So a queued request has to wait for a slot and finish generating, all within 100 seconds, or the user sees a timeout that looks like the server is broken when in fact it's just busy.

How much that bites depends entirely on your model + token-budget combination:

Model + budget Per-call wall time What happens at 3+ concurrent
qwen2.5:3b-instruct 0.5–3 s Queue clears well inside 100 s. Users see latency, not errors.
qwen2.5:7b-instruct, numPredict ≤ 400 10–20 s Usually OK up to ~4 concurrent before the 3rd starts racing the edge.
qwen2.5:7b-instruct, numPredict 1600 50–75 s The 3rd concurrent call will almost certainly 524. Don't fan out.
qwen2.5:14b-instruct 2–3 min if it fits Single-request only; concurrent callers will time out.

There are mitigations if it ever does start mattering. The cheap one is stream: true, which makes first-token latency race the 100-second ceiling instead of total latency — once any byte starts flowing, CF stops counting. The next one up is bumping OLLAMA_NUM_PARALLEL to 4 (each additional slot wants ~1.5 GiB of VRAM headroom; check nvidia-smi under load before doing it). The serious answer is putting a small queue-aware shim in front of ollama that exposes a /queue JSON endpoint and returns 429 Retry-After instead of pretending to accept a request that's about to die — but at single-digit-requests-per-day load, that's overkill.

The real lesson here is the meta one: a stack works at the load you tested it at. Test it at the next 5×.

What's Next

A few things on my list:

  • Try moondream or qwen3.5:4b, which are small vision models. (My first attempt at this was ollama pull qwen2-vl:7b — which 404s, because the Qwen vision lineage is now qwen3.5 and the older standalone qwen2-vl tags have been retired from ollama's library.) The contact-card path currently runs Tesseract OCR before the LLM extraction — a vision model could potentially skip the OCR step entirely and just read the card image directly. May or may not be better than what I have; worth one evening of A/B testing.
  • Wire up GPU monitoring. Right now I have no visibility into VRAM use, GPU utilisation, or temperatures while the cards are working. The PowerEdge's chassis cooling is more than adequate for two passively-cooled P4s, but I'd rather catch a fan failure or an unexpected thermal trend from a Prometheus alert than from an email saying the host fell off the network.
  • Explore embedding models like nomic-embed-text. Embeddings open up semantic search over my own personal knowledge base, RAG over my wiki, and a few other things I've been wanting to mess with.

Most of those are afternoon-class projects on top of a stack that's already running. The hard part — actually getting two old GPUs out of "I should do something with these eventually" purgatory and into something that's serving real requests — is done.

The lesson, again, was the same one I keep relearning: the fastest way to validate that a hardware project is actually feasible is to spend an afternoon trying. The two cards that had been quietly riding along in a colocated server for years went from "I should do something with these" to serving inference for my apps in less time than it would take to read a book about the same topic. The modern stack — Proxmox passthrough, ollama, cloudflared, nginx — is friendlier than the same stack would have been three years ago. The right time to start is now.