Running even larger LLMs on small hardware

In a previous post [GM4] we ran gemma4 12B on a 16GB RAM budget; today we’ll push the envelop and run deepseek-r1:14b-qwen-distill-q4_K_M on the same budget. While last time we got away by fiddling with the model definition, today we’ll have to reconfigure the OS itself (in addition to fiddling with the model definition). While this post is heavy on details for running on a Radeon GPU, just omit them if interested in running on a CPU-only platform.

The BOM:
– my (t)rusty Acer Nitro V15 on Ubuntu 24.04 [ACN]. Rhyzen/Radeon, 16GB RAM, 2 SSDs
– swap spread over two SSDs
– custom kernel parameters & settings
– a tweaked model file
– ollama

Boot kernel parameters

Kernel parameters in the grub command line:

			
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=off ttm.pages_limit=2097152 amdgpu.gttsize=8192 zswap.enabled=1 zswap.compressor=zstd zswap.max_pool_percent=25"

amd_iommu [IOM] I don’t pretend to understand what exactly this is, but it seems related to AMD virtualisation. Disabling it yields half a token/s and I’ve not noticed any drawbacks. If you don’t run on an AMD platform, this isn’t useful and can be omitted.

ttm.pages_limit [TPL] allows the graphics driver to use more VRAM. This seems to be related to shared memory architectures, thus unless you have one (eg. Rhyzen with integrated GPU), this won’t help either.

amdgpu.gttsize caps the shared memory size between driver and OS. Making this too large results in out-of-memory failures. I suspect making it too small harms performance. I tweaked it because of the former reason.

zswap.enabled activates swap compression; in my experiments this doesn’t speed the model up but makes the system a bit more responsive.

zswap.max_pool_percent is like a “cache” for compressed memory pages. The particular value was chosen after monitoring zswap stats (grep -r . /sys/kernel/debug/zswap/) over a few runs. Far from scientifically rigorous, my intuition is that my consumer SSDs (as implied my mdstat) choke when swapped to; so even if they read nearly 6GB/s, once pages are swapped out to them, read speed tanks. mdstat regularly showed read speeds < 200 MB/s while running the model. Since the CPU is idling during swapping, the time it allocates to (de)compression is well-spent. It doesn’t seem to compress model memory well, but swapped-out OS/application memory.

Run-time kernel parameters

			
/etc/sysctl.conf  settings:
 
vm.overcommit_memory = 1   
vm.swappiness = 60
vm.page-cluster=0

vm.overcommit_memory allows allocating more memory than available; will work if the requested memory wasn’t use, will crash the system otherwise.

vm.swappiness clears out memory for the LLM faster.

vm.page-cluster I’m conflicted over this. It’s a “block read/write ahead” of sorts for memory pages. Large(r) values seem to speed up model startup, smaller values seem to reduce CPU load during token generation.

Docker

			
The actual run script:
#!/bin/bash
docker stop ollama || true
docker rm ollama || true
group_id_video=$(getent group video | cut -d: -f3)
group_id_render=$(getent group render | cut -d: -f3)
cat << 'EOF' > mock_meminfo
MemTotal:       67108864 kB
MemFree:        60000000 kB
MemAvailable:   60000000 kB
Buffers:          100000 kB
Cached:          2000000 kB
SwapTotal:      67108864 kB
SwapFree:       67108864 kB
EOF
docker run -d \
--privileged \
--device /dev/kfd \
--device /dev/dri \
--device /dev/dri/card1 \
--device /dev/dri/renderD128 \
--memory-swap -1 \
--shm-size=2gb \
--memory=48g \
--volume "./ollama-volume:/root/.ollama" \
--volume "./mount:/tmp/mount:ro" \
--volume "./mock_meminfo:/proc/meminfo:ro" \
--group-add $group_id_video --group-add $group_id_render \
-p 11435:11434 \
-e OLLAMA_KV_CACHE_TYPE=q4_0 \
-e OLLAMA_FLASH_ATTENTION=1 \
-e OLLAMA_MAX_LOADED_MODELS=1 \
-e OLLAMA_DEBUG=1 \
-e OLLAMA_CONTEXT_LENGTH=4096 \
-e "OLLAMA_ORIGINS=*" \
-e OLLAMA_IGPU_ENABLE=1 \
-e HSA_OVERRIDE_GFX_VERSION=10.3.0 \
-e OLLAMA_NUM_PARALLEL=1 \
--name ollama ollama/ollama:rocm

		

A few notes:

mock_meminfo: tricks conservative model checks against available memory. We know we’re massively OOM here and made peace with the fact that there’ll be swap. The model doesn’t know that, but we do.

privileged/device mounts: required for rocm device access.

various memory switches: override docker defaults to make sure all (virtual) memory is used.

OLLAMA_IGPU_ENABLE: ollama won’t use the GPU without it, not sure why. The linked post is quite instructive; I understand there are two “ways” of using the Radeon GPU, one with AMD’s proprietary rocm, another one through Vulkan. I have not tried Vulkan, I can confirm that rocm is a pain to set up and get right. I have, however, benchmarked [CLP] rocm vs Vulkan and found the former to be somewhat faster.

OLLAMA_KV_CACHE_TYPE overrides the KV cache precision. Shrinks the model footprint at the cost of precision.

OLLAMA_FLASH_ATTENTION should be a default setting by now. Google it or ask your friendly AI.

OLLAMA_CONTEXT_LENGTH reduces memory use at the cost of working memory size.

Model layers on the GPU

This is turning into a science of its own. Previously [GM4] we found the modelfile num_gpu setting to be a life-saver which allows off-loading too large LLMs into the system RAM when it doesn’t fit into VRAM. But it turns out to be more than that. The particular model (14b) has 49 layers (according to container logs). If ran like that, it barely fits into VRAM, creating massive thrashing between VRAM/system RAM/swap. Loading the model alone takes more than 5 minutes. By reducing num_gpu, the model loads much faster, runs faster and avoids a swapocalypse. When set to 0, the model runs entirely on the CPU. I’ve found values between 2 and 8 to work best for my setup.

Power

Make sure the GPU runs at full power. Eg. I’m running [TLP] which, among other things, can reduce GPU (memory) frequency by over 50% to save power and reduce heat dissipation. Model run times halve between the battery and AC profiles, so it’s well worth checking with nvtop/radeontop what state the GPU is in.

Was it worth the trouble?

The model runs at 5,5 tokens/sec which is about as fast as I read. The context window is tiny, so it can’t process large text. It doesn’t leave much RAM for anything else, but a browser with a few web pages won’t crash the system. It’s smart enough for text analysis and simple reasoning, so a good candidate for non-critical batch/asynchronous tasks on a CPU-only VM with 16GB RAM. Totally worth it.