General setup notes on my running Gemma 4 locally, using the hardware l have:
The file size of gemma-4-e4b-it-BF16.gguf is approximately 15.1 GB (or about 15.05 GB to 15.1 GB on disk).
Model Details
- Precision: Unquantized 16-bit bfloat16 (
BF16) - Architecture: Gemma 4 E4B (Effective 4.5B parameters, higher with embeddings)
- Hosting: Available for download on Hugging Face repositories such as the Unsloth Gemma 4 E4B GGUF Repository
My available VRAM is 24Gb on the RTX3090 and my PC RAM is 64Gb
I can easily run the BF16 version entirely in your GPU VRAM without any quantization loss.
Performance Breakdown
- VRAM Fit: 24 GB is plenty. The 15.1 GB model leaves ~9 GB of free VRAM.
- Context Space: This remaining 9 GB is more than enough for a large KV cache (context window).
- Speed: Running entirely on the RTX 3090 VRAM ensures maximum generation tokens per second. The 64 GB of system RAM acts as a massive cushion, meaning this system won't slow down.
Quantization is unnecessary for this setup unless one plans to run multiple heavy applications or models simultaneously.
The native context window for Gemma 4 E4B is 128,000 tokens (128K)
By default, Ollama often restricts context sizes to 4,096 tokens unless you explicitly tell it to use more. With the RTX 3090 (24 GB VRAM), one has enough memory to scale up dramatically.
VRAM and Context Windows
The unquantized 15.1 GB model leaves one with roughly 8.9 GB of free VRAM. This extra VRAM determines how large your context window can safely scale before risking out-of-memory (OOM) crashes:
- 8K to 16K Context: Uses minimal extra VRAM. Perfectly safe and fast. Great for general chatting, short documents, and coding scripts.
- 32K Context: Consumes roughly 3–4 GB of additional VRAM for the KV cache. This leaves a comfortable buffer and is ideal for long coding sessions or reading full articles.
- 64K to 128K Context: Will max out or exceed your remaining 8.9 GB of VRAM. Attempting a full 128K context window with an unquantized BF16 model will likely cause Ollama to offload layers to your system RAM, heavily slowing down performance.
How to Set It Up in Ollama
You can easily adjust the window using Ollama's
num_ctx parameter. A 32,000 (32K) token window is the sweet spot forthiw hardware to ensure blazing fast speeds without crashing.
Method 1: On-the-Fly via the REPL
If you are running the model in the command line terminal, type this command before sending your first prompt
bash
/set parameter num_ctx 32768
Method 2: Permanent Custom Model fileTo avoid typing it every time, you can bake it directly into a custom model:Create a blank text file named Modelfile. Paste the following lines (point it to your downloaded GGUF file path):dockerfileFROM /path/to/gemma-4-e4b-it-BF16.gguf
PARAMETER num_ctx 32768
Save the file, open your terminal, and build the model:bash
ollama create gemma4-32k -f ./Modelfile
Run your brand new custom model:bash
ollama run gemma4-32k
For a laptop with 8GB VRAM and 32GB RAM, the absolute best choice is a 7B or 8B parameter model quantized to 4-bit (Q4_K_M), or a smaller 3B to 4B parameter model in high precision.The 8GB VRAM is the primary bottleneck. To keep generation speeds fast, the entire model plus its context memory must fit inside that 8GB window. If a model exceeds 8GB, Ollama will split it into your system RAM, causing speeds to drop drastically.Top Model Recommendations- Llama 3.1 8B (Q4_K_M variant): This is the best all-rounder for an 8GB GPU. At 4-bit quantization, the model takes up roughly 4.8 GB of VRAM, leaving you over 3 GB of room for a comfortable 8K to 16K context window. It excels at reasoning, coding, and general conversation.
- Gemma 2 9B (Q4_K_M variant): An incredibly powerful model that often punches above its weight class. The file size is around 5.5 GB, which fits safely in your VRAM. However, because its architecture uses more memory during generation, you will need to cap your context window at 4K to 8K to avoid spilling over into system RAM.
- Phi-3.5-mini 3.8B (High precision / FP16): If you need to process massive documents or long codebases, this is your best option. The model is tiny (~7.6 GB unquantized or under 3 GB if quantized), allowing you to scale the context window up to 32K or higher without running out of VRAM
Performance Outlook for This Setup
Model Selection VRAM Usage Speed Expectation Best Use Case 8B Models (Q4_K_M) ~4.8 GB – 5.5 GB Fast (40+ tokens/sec) Coding, logic, everyday tasks 3B–4B Models (Q8 / FP16) ~3.0 GB – 7.5 GB Blazing Fast (60+ tokens/sec) Long documents, low battery usage 14B+ Models (Any format) Exceeds 8 GB Very Slow (Spills to RAM) Not recommended for daily fluid use
