Peer-to-peer got its verdict on Rocket Lake, and the swap device that ate my RAM got mine¶
Last week I wrote up the PCIe atomics bug that made two Radeon R9700s look broken on my Z490 AORUS MASTER / i7-11700 box. Blocking AtomicOps fixed that, and the machine ran well for days on two cards doing tensor-parallel inference under Radiance. Then two separate nights of trouble produced two findings worth writing down: what actually happens when you force PCIe peer-to-peer on this platform, and why my inference engine kept getting OOM-killed at seemingly random moments. Both answers are mechanical, and neither was the obvious suspect.
Why I went hunting for P2P at all¶
With two cards tensor-parallelized, every transformer layer ends in an all-reduce, and the cards have to exchange data with each other. With peer-to-peer working, one card reads the other's VRAM directly over the PCIe fabric. Without it, every exchange stages through pinned host memory — out to the CPU and back twice per hop. Radiance tells you exactly which world you're in at every startup:
note p2p-off: the cards cannot reach each other's memory directly (0->1, 1->0)
why: the collectives exchange through pinned host memory, which crosses each card's link twice
On AMD platforms the kernel makes this mostly automatic: drivers/pci/p2pdma.c permits peer-to-peer for AMD root complexes (Zen and newer) essentially unconditionally, because AMD has validated that fabric. Everywhere else — including every Intel consumer platform — the kernel only whitelists hierarchies it has evidence for, and the p2pdma docs say it out loud: peer-to-peer is only enabled "when the platform is known to support it." My Z490 was not in that club, which is why the whole x99-7900xtx-p2p project exists: a small kernel module that sticks a kretprobe on pci_p2pdma_distance_many and forces the kernel's answer to "yes" for two AMD cards.
The interesting part is that the gate is software. Nobody ever proved the Z490 fabric can't route GPU-to-GPU traffic — the kernel simply refuses to try. An X99-forcing precedent exists and works, so "Rocket Lake can't do P2P" had been folklore on this box, not fact. I decided to make it fact.
The experiment, and the two runs¶
Setup steps that worked, in order:
- The preflight script in that repo demands IOMMU groups. My kernel cmdline had a build-day relic pair —
intel_iommu=off iommu=pt— which contradicts itself; I removed both and setintel_iommu=onviagrubby --update-kernel=ALL. DMAR came up in translated mode, no faults, both cards in their own IOMMU groups. - Built the module against the running kernel, installed it to
/lib/modules/.../extra/, and pinned it into the initramfs with asoftdep amdgpu pre: p2p_forcemodprobe config so it loads before the GPU driver. - Reboot. For the first time on this machine, KFD reported peer-to-peer links between the two cards. The kernel gate was open.
- Correctness probe:
torchrun --nproc-per-node 2running an all-reduce over the forced P2P path (torch 2.13 + ROCm 7.1; note that ROCm 7.1 needsHSA_ENABLE_IPC_MODE_LEGACY=0orhipIpcGetMemHandlefails with "invalid argument" — a known regression, wasted ten minutes on it).
First run: passed numerically. All-reduce results bit-exact against reference (bad=0), about 3.2 GB/s through the forced path. The fabric looked alive. Then I reran it with verbose debugging enabled and the machine answered the question for me:
amdgpu: HW Exception by GPU node-1 ... GPU Hang
pcieport 0000:00:01.0: AER: CmpltTO Requester 00:01.0 Transaction Layer
CmpltTO is a completion timeout on the root port — the root complex stopped returning completion packets for peer reads. GPU→GPU memory reads were being silently dropped until timeout, which is precisely the failure family of the atomics bug from the previous post: this platform's root ports lose PCIe transactions under certain cross-device traffic patterns, log nothing until it's a hang, and the GPU eats it. The first run passing is not counter-evidence; lost-completion behavior is traffic-pattern dependent, and "correct when I happened to look" is the worst possible property for a memory path.
Verdict
PCIe P2P between two R9700s on Rocket Lake is now closed with evidence, not folklore. The kernel's refusal is forceable; the fabric underneath is not trustworthy. Full revert: module, softdep, dracut config, and initramfs rebuild all removed; kernel cmdline back to the original flags; the kernel I'd updated for the build uninstalled. Nothing survived except the answer. If you're on X99 or any AMD platform, that repo is legitimate — this verdict is about Rocket Lake root ports, not the technique.
The kills that weren't P2P¶
Independent of all that, Radiance kept dying. Over one night the engine got OOM-killed three times from three different directions: at 55k tokens mid-session, minutes after a session finished, and once two minutes after startup with zero requests served. Every misdirection you can imagine showed up: a VM that autostarts on reboot, a kernel 7.3.3 regression where the amdgpu driver stopped exposing mem_info_vram_vis_total (the engine then mis-detects the BAR size and falls into a degraded staging path — pinned to 7.2.15 until that's fixed upstream), and a BIOS that had silently reset itself and wiped Resizable BAR. All real, all side-shows.
The three kills shared one number in the kernel's dump: anon-rss: 33.2 GB on the victim, always. Add the engine's pinned GPU buffer pages and its host footprint is roughly 43 GiB — by design, for a 262k-context configuration. The machine has 62 GiB total. So the engine plus kernel, desktop, browser, and a small VM lands within spitting distance of full, and whether the last straw comes at load time or session end is just page-cache roulette. That explains "why so much RAM," and it explains the load-time kill once I found the line below — but the session-end kill kept its own counsel until later in the night, and it turned out to be a completely separate mechanism. For the kills that did belong to zram, the answer was sitting in one line of the OOM report:
zram: the swap device that is made of RAM¶
Fedora has shipped zram enabled by default since Fedora 38 — a generator creates an 8 GiB swap device at boot whose backing store is memory: the kernel compresses evicted pages and stores the compressed blobs in RAM. It's a great idea for a laptop with an idle margin. My box no longer has that margin.
The failure loop: memory pressure builds as the engine loads → the kernel dutifully compresses idle pages into the zram pool to "relieve" pressure → the pool itself is unreclaimable RAM (7-8 GiB by the time I caught it) → the accounting says pressure is relieved, so more allocations get approved → the pool keeps growing toward its ceiling → when the ceiling is hit, swap-out fails, pages that were evicted fault back in, and the kernel discovers several gigabytes at once that it doesn't have. global_oom, biggest victim dies — the engine, every time, for the crime of being large. The 46 GiB container memory cap I'd set can't help here, and that's worth stating plainly: the cap fires when the container exceeds its limit, and the engine never did. It was the system running dry around it. The engine's own startup notes had been printing the fix for weeks:
note swap-in-use: 5.82 GiB of swap is in use
fix: free memory, or turn swap off on a machine that only serves
Disabling it for the running system is two commands — swapoff /dev/zram0 then zramctl -r to reset the device — and the effect is immediate: 7 GiB of RAM returned, engine restarted, steady state now 43 used / 18 available with zero swap devices, which has stayed stable under real sessions.
The generator pitfall
Making it stay off is the part that bit me twice. zram units are produced by a systemd generator, which runs at boot and on every daemon-reload. My first instinct — systemctl mask on the generated unit — was wrong on both counts: generated units can't be masked meaningfully, and the mask command itself triggers a daemon-reload, which re-ran the generator and resurrected zram mid-session, which OOM-killed my engine a fourth time by my own hand. The documented kill switch is a kernel command-line flag, which disables the generator's built-in default:
One command to undo if you ever want it back: remove the flag, reboot. There is no vendor config file on Fedora to null out — the default is compiled into the generator's behavior, which is exactly why systemd.zram=0 is the right lever.
The kill that outlived the zram fix¶
With swap gone, the box held steady at 43 used / 18 available under real sessions — and then the engine died again. This time the shape was unmistakable and still unexplained: it died idle, with no requests in the log for ten minutes, about nine minutes after the conversation had finished. "Minutes after completion" again. I had one good tool left, so I armed it: a shell loop writing container memory, system memory, GPU-pinned memory, and dirty-page counts to a CSV every two seconds, and I went back to work.
The capture settles it. Here is the death in the sampler's own rows — two seconds apart, nothing running:
00:41:55 cgroup=32 sys=51 avail=11 gpu-pinned=0.4 GiB (idle)
00:41:57 cgroup=32 sys=60 avail=1 gpu-pinned=9.4 GiB (nine gigabytes pinned in two seconds)
00:42:01 kernel kills the engine (global OOM)
The trigger is housekeeping, not a workload. Radiance keeps a finished conversation's KV in VRAM so follow-ups hit cache, and the help text for --prefix-cache-ttl has the operative clause: entries are kept (default: an hour) "until the memory is wanted." What wants it is the expert pool — 22 GiB of routed experts that live in the same elastic VRAM as the KV cache, and which the engine happily reclaims from any conversation no live request holds. When it reclaims, the conversation is demoted out of VRAM into a pinned host-memory tier — the KV, the linear-attention state checkpoints, and the staging buffers for the move, all in one burst, into memory that is pinned by definition and therefore invisible to every reclaim mechanism the kernel has. The engine prints the motive in plain words: the host copy exists "so their VRAM can go to experts and a return restores instead of re-prefilling." My 100k+ token sessions cost about 9 GiB to demote. The machine had 11 GiB of room. The offload landed within two gigabytes of the wall and the OOM fired on the next page anyone needed.
That also retroactively explains the original "dies minutes after a session finishes" kill: the eviction isn't scheduled, it's opportunistic — the engine tidies the idle conversation out of VRAM whenever it next wants the space, which is minutes after the conversation goes quiet. A timer I initially blamed (prefix-cache-ttl, default 3600 seconds) turned out to be a cap, not a fuse; raising it changes nothing, because the eviction happens whenever the VRAM is wanted, not when the clock says so. The fix is one of two trades — shrink another pinned allocation to make room for the demotion, or turn the demotion off. I turned it off: --prefix-cache-mib 0 makes finished conversations simply disappear, and the engine's own startup budget confirmed the tier is gone (its host allocation plan dropped by exactly the 4 GiB the tier used to reserve). The cost is real and I'll pay it knowingly: returning to a finished conversation now re-prefills from scratch, which at my measured prefill rate is half a minute or more of dead time per big session. A predictable tax beats a booby trap. If the box ever grows to 128 GiB, the tier goes back on.
What's left standing on the box¶
The machine is boring now, which is the point. P2P stays off — the root ports made that decision for me, with a completion timeout as the signature. Swap stays off — systemd.zram=0 rides on every kernel entry. The resume tier stays off — finished conversations are dropped rather than demoted, and the half-minute re-prefill tax is the cheapest thing I buy all week. The engine runs 262k-context tensor-parallel work under a 46 GiB container cap with roughly 15 GiB of honest headroom on a 62 GiB box, and the cap is the only guardrail because nothing else is pretending. The durable upgrade is the one everything keeps converging on: more DIMMs. The ~43 GiB host footprint is a design property of a 262k configuration, the 9 GiB demotion burst is a design property of keeping finished conversations, and neither is a leak — the box is simply being asked to be 128 GiB.
Three nights, three shapes of the same lesson: the exotic theory (root complex transaction loss) needed a forced experiment to be proven; the mundane theory (swap that isn't really swap) needed me to stop reading free as if it lied; and the last one — a feature doing exactly what it advertises, at the worst possible second — needed a two-second sampler and the patience to let a timer go off on camera. The atomics post covers the first bug in this rig's family; the p2p-off, swap-in-use, and prefix-cache notes quoted above cover the rest, printed by the engine itself, at every boot, waiting to be read.