Best GPU for Local AI in 2025: Real-World Benchmarks

When it comes to running local AI models, VRAM is the single most important spec — more than CPU clock speed, more than system RAM bandwidth, more than PCIe generation. The entire model (or as much of it as possible) needs to live inside your GPU's video memory during inference. When it doesn't fit, Ollama falls back to CPU RAM, and performance drops by 3–10×.

This guide benchmarks the most common consumer GPUs for local AI workloads and helps you decide what to buy — or whether the hardware you already own is already good enough. All token/s figures are measured running Qwen3 14B Q4_K_M unless otherwise noted, which is one of the most common "serious" model sizes for home users.

Quick picks by budget

Prices reflect mid-2025 retail. Used market prices are typically 20–35% lower. The M4 Pro tokens/s figure is for the 48 GB unified memory configuration; the 24 GB variant runs the same model but with slightly less headroom for context.

* Unified memory is shared between CPU and GPU. Effective "VRAM" equals total system memory minus OS overhead (~4 GB).

VRAM vs RAM: what actually matters

System RAM (DDR5, DDR4) has a peak bandwidth of roughly 50–90 GB/s on modern platforms. GPU VRAM (GDDR6X on a 4090) delivers 1 TB/s — about 15–20× faster. Since transformer inference is heavily bandwidth-bound (the GPU has to read all model weights for every generated token), VRAM bandwidth directly translates to tokens per second.

When a model doesn't fit in VRAM, Ollama splits it between GPU and CPU memory. Layers that fit in VRAM run at full GPU speed; the rest run on CPU. A 14B model that's 1 GB over your VRAM limit might still achieve 20 tok/s; the same model 8 GB over the limit might run at 4–6 tok/s. More offloaded layers = slower inference, not a hard failure.

AMD and Apple Silicon

AMD RX 7000 series

AMD's RX 7900 GRE (16 GB, ~$500) and RX 7900 XTX (24 GB, ~$900) offer competitive VRAM at lower prices than NVIDIA equivalents, but the software story is less mature. Ollama uses ROCm for AMD acceleration, which is only supported on Linux as of mid-2025 — Windows AMD users fall back to CPU inference. ROCm support has improved significantly in 2024–2025 and is now stable for the major model architectures, but expect occasional driver quirks that NVIDIA/CUDA users don't encounter.

If you're on Linux and primarily run open-source models, an RX 7900 GRE at $500 for 16 GB of VRAM is one of the best value propositions in the market. If you're on Windows or need reliable plug-and-play, stick with NVIDIA for now.

Apple Silicon (M3 / M4 series)

Apple's unified memory architecture is uniquely suited to local AI. Because CPU and GPU share the same physical RAM, a 48 GB M4 Max has 48 GB of "VRAM" available — something that would cost $3000+ on a discrete GPU. The memory bandwidth (~400–800 GB/s on M4 Pro/Max) is between GDDR6 and GDDR6X, resulting in solid tokens/s for large models.

An M4 Pro 48 GB Mac Mini (~$2000) can run Qwen3 32B at full Q4_K_M — a task that requires an RTX 4090 on the PC side. It does it silently, with low power draw, and without a dedicated GPU. The trade-off: for smaller models that fit in VRAM, a $600 RTX 4070 Super will be 20–40% faster at raw tokens/s.

The used GPU market

The RTX 3090 (24 GB VRAM) is the most compelling used GPU for local AI. It was the consumer VRAM king of its generation, and units now trade for $400–600 on eBay and local marketplaces — putting 24 GB of VRAM within reach of a mid-range budget. At 24 GB you can run Qwen3 14B fully in VRAM at high quantization, or attempt 32B models with some CPU offloading.

The RTX 3080 Ti (12 GB) and RTX 3080 (10 GB) are also worth considering for their price-to-performance on 7–8B models, where their fast GDDR6X bandwidth still delivers 30–45 tok/s at a fraction of the cost of a new 40-series card.

Monitor VRAM usage in real time

While Ollama is running a model, you can watch VRAM consumption and utilization to verify the model is fully GPU-loaded and diagnose performance issues:

# NVIDIA — live VRAM usage (Linux)
watch -n 1 nvidia-smi

# NVIDIA — live VRAM usage (Windows PowerShell)
nvidia-smi -l 1

# Apple Silicon — GPU utilization + memory pressure
sudo powermetrics --samplers gpu_power -i 1000 | grep -iE "gpu|memory"

Look for MiB Used in the nvidia-smi output. If it's close to your card's total VRAM, the model fits. If the GPU utilization (GPU-Util %) spikes to 100% during generation, your GPU is the bottleneck — which is the ideal situation.