Running Gemma 4 Locally - Gathered Notes 2
With 32GB of system RAM and an 8GB VRAM RTX 2060, the best
overall options for your setup are gemma4:12b for a completely smooth,
GPU-accelerated experience, or gemma4:26b if you want to push for max
intelligence using partial RAM offloading. [1, 2, 3, 4, 5]
Recommended Models for Your Specs
|
Model Tag |
Download Size |
Best For |
Speed on Your Setup |
|
gemma4:e4b |
~9.6 GB |
Ultimate speed & daily tasks |
Fastest (~35+ tokens/sec) — Fits completely into
your 8GB VRAM. |
|
gemma4:12b |
~7.6 GB |
Best Balance (Coding & Logic) |
Fast (~25 tokens/sec) — Squeezes almost entirely
into VRAM with minimal RAM spillover. |
|
gemma4:26b |
~18 GB |
High-tier reasoning & writing |
Moderate (~8–12 tokens/sec) — Uses partial
offloading to your 32GB system RAM. |
Why These Models Fit Your Hardware
1. The Sweet Spot: gemma4:12b
This model size bridges the gap perfectly. A quantized 4-bit
version only requires about 6.7 GB of memory. This leaves just enough room for
your 8GB VRAM to handle the model weights, giving you high-speed performance
along with advanced multimodal and coding abilities. [6, 7, 8, 9]
ollama run gemma4:12b
2. Squeezing Out Max Quality: gemma4:26b
The 26B model uses a Mixture of Experts (MoE) design. It has
the knowledge of a massive model but only activates 4 billion parameters per
token. While its 18 GB footprint exceeds your 8GB VRAM, Ollama will
automatically split the layers—putting ~6GB onto your GPU and offloading the
rest to your 32GB system RAM. Because it only evaluates 4B parameters at a
time, it generates text much faster than traditional large dense models. [3, 4,
6, 8, 10, 11, 12, 13]
ollama run gemma4:26b
3. The Pure Speed Option: gemma4:e4b
If you need instant responses for simple tasks, lookups, or
automation, the e4b (Effective 4B) variant fits entirely inside your VRAM with
plenty of room left over for giant conversation histories (context windows).
[6, 14]
ollama run gemma4:e4b
With 64GB of system RAM and a 24GB VRAM RTX 3090, your
system is a top-tier local AI workstation. You have enough memory to run the
flagship gemma4:31b or gemma4:26b completely within your GPU's
VRAM for lightning-fast, uncompromised performance. [1, 2, 3, 4]
Recommended Models for Your Specs
|
Model Tag |
Download Size |
Best For |
Speed on Your Setup |
|
gemma4:31b |
~17.5 GB |
Flagship Choice (Max Reasoning/Coding) |
Blazing Fast (~50+ tokens/sec) — Fits 100% in VRAM
with room for long context. |
|
gemma4:26b |
~14.4 GB |
High-throughput agent workflows |
Extremely Fast (~80+ tokens/sec) — Uses ultra-fast
MoE sparse activation. |
|
gemma4:12b |
~7.6 GB |
Low-latency audio & vision multitasking |
Max Speed (~110+ tokens/sec) — Instantly processes
huge context windows. |
Why These Models Fit Your Hardware
1. The Flagship Experience: gemma4:31b
The 31B dense model is the workstation crown jewel of the
Gemma 4 lineup. At its default 4-bit quantization, it only occupies 17.5 GB
of memory. Because this fits entirely within your 24GB VRAM with ~6.5GB to
spare, you avoid slow system RAM bottlenecks completely. This model handles
deep architectural coding, advanced logic parsing, and multi-turn complex
reasoning effortlessly. [3, 5, 6, 7, 8, 9]
ollama run gemma4:31b
2. High-Throughput Efficiency: gemma4:26b
The 26B model uses a sparse Mixture of Experts (MoE) design
that only activates 4 billion parameters per token. The full model fits
seamlessly inside your VRAM at 14.4 GB. Since it only computes a
fraction of the parameters per token while retaining the full knowledge of a
26B model, it delivers a massive speed jump over dense models, routinely
pushing 80+ tokens per second on a 3090. [1, 5, 7, 10, 11]
ollama run gemma4:26b
3. Extra Capabilities Note: gemma4:12b
While the 31B and 26B variants excel at raw intelligence,
text reasoning, and image parsing, the 12b Unified model includes native
end-to-end audio encoders. If you need a model that can natively listen to or
process live speech alongside massive context windows (up to 256K tokens), keep
the 12B variant in your toolkit. [5, 6, 9]
ollama run gemma4:12b
To handle both complex software development and heavy
document analysis on your RTX 3090 (24GB VRAM) and 64GB RAM workstation, gemma4:31b
is your best primary model, paired with gemma4:26b as a high-speed
alternative. [1, 2, 3]
When analyzing massive PDFs and coding full repositories, Context
Window (KV Cache) memory exhaustion becomes your biggest bottleneck. If a
model spills over your 24GB VRAM into system RAM, text generation speeds
instantly tank. [4, 5]
Hardware Breakdown for Context & Coding
|
Model |
Size (Q4) |
Safe Context Window on 24GB VRAM |
Use Case Strength |
|
gemma4:31b |
~17.5 GB |
Up to ~32K–64K tokens |
Heavy logic, deep software architecture, code debugging. |
|
gemma4:26b (MoE) |
~14.4 GB |
Up to 128K+ tokens |
Gigantic multi-PDF analysis and high-speed iterative
coding. |
Strategic Recommendation Based on Your Tasks
1. For Deep Architecture & Complex Coding: gemma4:31b
The flagship 31B dense variant is the most intelligent model
in the family. Its advanced reasoning excels at tracking complex code
dependency trees, syntax generation, and logic structures. Because the model
base takes up 17.5 GB, it leaves you about 6.5 GB of free VRAM. This remaining
space comfortably holds a solid context history (plenty for medium-to-large
code scripts) while maintaining maximum GPU execution speeds. [1, 2, 4, 5, 6]
ollama run gemma4:31b
2. For Massive PDFs & Repository Analysis: gemma4:26b
If you need to feed the AI entire source code repos or
hundreds of pages of documentation at once, use the 26B Mixture of Experts
(MoE) model. It only requires 14.4 GB of base VRAM, leaving nearly 10 GB of
free VRAM purely for your context window. Since it only activates a portion
of its network (4B parameters) per token, it reads through giant inputs at
lightning speeds, allowing you to stretch deep into Gemma 4's massive native
context limits without suffering hardware slowdowns. [1, 3, 6]
ollama run gemma4:26b
Pro-Tip for Document Processing
When running your documents through Ollama, make sure to
adjust the environment variables or configuration files to maximize the num_ctx
setting (e.g., set it to 32768 or 65536) to match the size of your files,
ensuring the model reads the entire document rather than cutting it off.
Community Sentiment on the 26B MoE
Developers running this tier of hardware find the
architectural trade-offs of the MoE variant highly effective:
“The 26B MoE variant is the best pick for most developers.
It activates only 4B parameters per token, runs at near-4B speeds, but delivers
quality close to a 13B model.”
