Ollama CUDA Out of Memory on Linux: Fixing VRAM Allocation Errors

The 'CUDA out of memory' or 'cudaMalloc failed' error in Ollama occurs when model weights combined with the context KV cache exceed the physical VRAM capacity of your NVIDIA GPU. The fastest solutions are enabling Flash Attention (OLLAMA_FLASH_ATTENTION=1), constraining the context window (num_ctx), and tuning GPU layer offloading (num_gpu).

Why Does Ollama Fail With CUDA Out of Memory?

When loading an LLM (such as Llama-3, Mistral, or DeepSeek-Coder), Ollama calculates how many neural network layers fit inside your GPU VRAM before offloading remaining layers to system CPU RAM. Allocation failures happen when:

• The KV cache expands rapidly during extended multi-turn conversations or large document processing.
• Background processes (desktop GUI, browser sessions, Docker containers) consume VRAM unmonitored.
• Ollama attempts to serve multiple concurrent slots (OLLAMA_NUM_PARALLEL > 1) without sufficient headroom.

4 Engineering Fixes for Linux Environments

1. Enable Flash Attention (Saves 30–50% VRAM)

Flash Attention significantly optimizes memory layout during attention computation, reducing peak VRAM requirements on modern NVIDIA cards.

Create or edit your systemd override file (/etc/systemd/system/ollama.service.d/override.conf):

[Service]
Environment="OLLAMA_FLASH_ATTENTION=1"

Reload and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart ollama

2. Restrict Context Window (num_ctx) in Modelfile

Modern models default to 8k or 32k token contexts, which consume gigabytes of VRAM just for buffer allocation. If you only process short prompts, constrain the context to 4096 tokens:

FROM deepseek-coder-v2:16b
PARAMETER num_ctx 4096

Build your optimized custom image:

ollama create custom-coder -f Modelfile

3. Restrict Concurrent Request Slots

Modern Ollama versions can allocate memory for up to 4 parallel requests by default. On 6GB–12GB VRAM cards, this causes instant CUDA allocation panic.

Lock execution to a single concurrent model and session:

[Service]
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_MAX_LOADED_MODELS=1"

4. Manual Layer Offloading via API (num_gpu)

If full GPU execution fails, force Ollama to offload fewer layers to GPU while keeping the remainder in system RAM:

curl http://localhost:11434/api/generate -d '{
  "model": "deepseek-coder-v2:16b",
  "prompt": "Refactor this SQL query:",
  "options": {
    "num_gpu": 20,
    "num_ctx": 4096
  }
}'

Live Telemetry & Verification

Monitor your active VRAM allocation directly in the terminal:

watch -n 1 nvidia-smi
ollama ps

ollama ps reveals the exact percentage of the model loaded into GPU VRAM versus CPU RAM.