Skip to content

Self Hosted

Peer-to-peer got its verdict on Rocket Lake, and the swap device that ate my RAM got mine

Last week I wrote up the PCIe atomics bug that made two Radeon R9700s look broken on my Z490 AORUS MASTER / i7-11700 box. Blocking AtomicOps fixed that, and the machine ran well for days on two cards doing tensor-parallel inference under Radiance. Then two separate nights of trouble produced two findings worth writing down: what actually happens when you force PCIe peer-to-peer on this platform, and why my inference engine kept getting OOM-killed at seemingly random moments. Both answers are mechanical, and neither was the obvious suspect.

What a Gen4 board costs: three engines, two R9700s, one benchmark ladder

Two Radeon AI PRO R9700s on a Z490 board with 62 GiB of DDR4 is not the machine the local-AI crowd writes about. The reference rigs in most engine docs are Gen5 AM5 boxes with fast DDR5. I wanted to know how much my Gen4 Rocket Lake board actually gives up, so I ran the same benchmark ladder against three engine/model combinations and put the results next to the published reference-rig numbers. Every figure below is engine-reported (not wall-clock math from me), and the raw JSONL sits in git- adjacent form on the box if anyone wants the per-request records.

Two R9700s, one Rocket Lake board, and the PCIe atomics that went missing

I moved my i7-11700 out of the Dell XPS and onto a used Z490 AORUS MASTER late one night, bolted in two Sapphire Radeon AI PRO R9700 TOPs, and got exactly what I paid for: every device on the bus, both cards reporting 32 GB of VRAM, idle temps in the mid-20s. Then every single attempt to load a real model into VRAM killed the machine. It took the entire night to find out why, and the answer was not in any driver, engine, or firmware log I expected.

The manual snapshot habit that ate 400 GB

I run two Hermes agent VMs on my Fedora workstation — libvirt/KVM, not Proxmox. For months the backup story for those VMs was me, clicking "snapshot" in virt-manager every so often when I was about to do something risky. That worked, in the sense that the snapshots existed. What I didn't notice is what they were doing to my disk.

The 8.5 GB that showed up as free

The fix in the last post was two more matching 32 GB sticks, and the sticks arrived. Four slots filled, all four the same part number, negotiated speed 4800 MT/s at 2DPC — right in the predicted range. With 128 GB installed I figured the GPU VM could take 114 and still leave the host its 2 GB. The host OOM'd.

The 401 spam that wasn't HA

Gotify kept pinging with 401s. The natural read is "an app's token is broken" or "gotify is down." Neither was true. The gotify server is healthy, and the 401 was content inside a forwarded log line. The thing posting the alert was loggifly, the log-monitor container, matching a keyword in a container's access log and forwarding it through Apprise.

This is the class of work: gotify ping, who actually posted it, what container logged it, who the real client is, what the true cause is.

The 56 GB that wasn't there

The GPU VM on the Powerspec box runs a 27B model that keeps its experts resident in RAM, so I wanted more RAM for it. The host has 64 GB, the VM had 50, and I figured I had room to grow. I bumped the VM to 56 GB and the host OOM'd. Twice.

This is the story of why, and it's mostly a story about assumptions I didn't check.

The false alarm that became the watchdog

The first run of the health watchdog reported pbs-backup-local as inactive, a CRITICAL on a backup path that was perfectly fine. The storage was healthy the whole time. What the watchdog actually caught was a momentary auth handshake, and it treated one bad sample as ground truth.

That false alarm is why the watchdog works the way it does now.

The one-byte bug

The voice assistant on the HA box has three legs: speech-to-text on the GPU box, an LLM brain, and text-to-speech back out. For weeks, a full pipeline run over the websocket died in about 30 milliseconds. The whisper bridge log showed a client connecting and immediately disconnecting with zero audio. Read that as "HA can't reach the bridge" and you'll chase it for days.

It wasn't HA. It wasn't the GPU. It was my own test client, and the whole thing came down to one byte.