Building an LLM from scratch

 Great video from Syntax Youtube channel on building an LLM from scratch.

Here is a link to a copy/fork of the code he developed for this exercise.

Running Gemma 4 Locally - Notes

General setup notes on my running Gemma 4 locally, using the hardware l have:

The file size of gemma-4-e4b-it-BF16.gguf is approximately 15.1 GB (or about 15.05 GB to 15.1 GB on disk).

Model Details

• Precision: Unquantized 16-bit bfloat16 (BF16)
• Architecture: Gemma 4 E4B (Effective 4.5B parameters, higher with embeddings)
• Hosting: Available for download on Hugging Face repositories such as the Unsloth Gemma 4 E4B GGUF Repository

My available VRAM is 24Gb on the RTX3090 and my PC RAM is 64Gb

I can easily run the BF16 version entirely in your GPU VRAM without any quantization loss.

Performance Breakdown

• VRAM Fit: 24 GB is plenty. The 15.1 GB model leaves ~9 GB of free VRAM.
• Context Space: This remaining 9 GB is more than enough for a large KV cache (context window).
• Speed: Running entirely on the RTX 3090 VRAM ensures maximum generation tokens per second. The 64 GB of system RAM acts as a massive cushion, meaning this system won't slow down.

Quantization is unnecessary for this setup unless one plans to run multiple heavy applications or models simultaneously. 

The native context window for Gemma 4 E4B is 128,000 tokens (128K)

By default, Ollama often restricts context sizes to 4,096 tokens unless you explicitly tell it to use more. With the RTX 3090 (24 GB VRAM), one has enough memory to scale up dramatically.

VRAM and Context Windows

The unquantized 15.1 GB model leaves one with roughly 8.9 GB of free VRAM. This extra VRAM determines how large your context window can safely scale before risking out-of-memory (OOM) crashes:

• 8K to 16K Context: Uses minimal extra VRAM. Perfectly safe and fast. Great for general chatting, short documents, and coding scripts. 
• 32K Context: Consumes roughly 3–4 GB of additional VRAM for the KV cache. This leaves a comfortable buffer and is ideal for long coding sessions or reading full articles.
• 64K to 128K Context: Will max out or exceed your remaining 8.9 GB of VRAM. Attempting a full 128K context window with an unquantized BF16 model will likely cause Ollama to offload layers to your system RAM, heavily slowing down performance.

How to Set It Up in Ollama

You can easily adjust the window using Ollama's num_ctx parameter. 

A 32,000 (32K) token window is the sweet spot forthiw hardware to ensure blazing fast speeds without crashing. 

Method 1: On-the-Fly via the REPL

If you are running the model in the command line terminal, type this command before sending your first prompt

bash

/set parameter num_ctx 32768

Method 2: Permanent Custom Model file
To avoid typing it every time, you can bake it directly into a custom model:
Create a blank text file named Modelfile.
Paste the following lines (point it to your downloaded GGUF file path):
dockerfile
FROM /path/to/gemma-4-e4b-it-BF16.gguf
PARAMETER num_ctx 32768

Save the file, open your terminal, and build the model:bash
ollama create gemma4-32k -f ./Modelfile
Run your brand new custom model:bash
ollama run gemma4-32k

For a laptop with 8GB VRAM and 32GB RAM, the absolute best choice is a 7B or 8B parameter model quantized to 4-bit (Q4_K_M), or a smaller 3B to 4B parameter model in high precision.
The 8GB VRAM is the primary bottleneck. To keep generation speeds fast, the entire model plus its context memory must fit inside that 8GB window. If a model exceeds 8GB, Ollama will split it into your system RAM, causing speeds to drop drastically.
Top Model Recommendations
Llama 3.1 8B (Q4_K_M variant): This is the best all-rounder for an 8GB GPU. At 4-bit quantization, the model takes up roughly 4.8 GB of VRAM, leaving you over 3 GB of room for a comfortable 8K to 16K context window. It excels at reasoning, coding, and general conversation.
Gemma 2 9B (Q4_K_M variant): An incredibly powerful model that often punches above its weight class. The file size is around 5.5 GB, which fits safely in your VRAM. However, because its architecture uses more memory during generation, you will need to cap your context window at 4K to 8K to avoid spilling over into system RAM.
Phi-3.5-mini 3.8B (High precision / FP16): If you need to process massive documents or long codebases, this is your best option. The model is tiny (~7.6 GB unquantized or under 3 GB if quantized), allowing you to scale the context window up to 32K or higher without running out of VRAM
Performance Outlook for This Setup
Model Selection
VRAM Usage
Speed Expectation
Best Use Case
8B Models (Q4_K_M)
~4.8 GB – 5.5 GB
Fast (40+ tokens/sec)
Coding, logic, everyday tasks
3B–4B Models (Q8 / FP16)
~3.0 GB – 7.5 GB
Blazing Fast (60+ tokens/sec)
Long documents, low battery usage
14B+ Models (Any format)
Exceeds 8 GB
Very Slow (Spills to RAM)
Not recommended for daily fluid use

Nvidia Jetson Orin

 Below are some great videos on using the Nvidia Jetson Orin

1)

NVIDIA Jetson Orin Nano: Secret to Running Ollama on the GPU on Jetpack 7.2, by Paul McWhorter of toptechboy.com

2)

How to Run YOLO 26 Detection Models on the NVIDIA Jetson Orin Nano, by Edje Electronics Channel

Nvidia - Exploring LLMs with Nvidia Jetson Nano, Orin

 Links for Nvidia - Exploring LLMs with Nvidia Jetson Nano, Orin:

1)

Getting Started with Edge AI on NVIDIA Jetson: LLMs, VLMs, and Foundation Models for Robotics

2)

Chitoku Yato - Technical Blog

3)

Khalil BenKhaled - Technical Blog

4_

Nvidia Developer Website Home page

Nvidia Jetson Orin

Models for JEtso  ORin

https://www.jetson-ai-lab.com/models/

How To Think SO Clearly People Assume You're Brilliant

How To Think SO Clearly People Assume You're Brilliant, by Sandeep S.

Understand AI in 14 minutes

 Understand AI in 14 minutes,  with Anthropic's Chloe Lubinski at ARC 2026 event.

What to teach when AI writes the code

 What to teach when AI writes the code by Rainer Stropek via TEDxLinz