Understanding Model Sizes and Quantization (GGUF): Run Bigger Local AI on Modest Hardware

A 70B parameter model compressed into 40 GB. Running on a laptop. No cloud. No API key. No monthly bill. That's not a marketing claim — that's what GGUF quantization actually delivers for people who know how to use it.

The problem is that most guides stop at "just download the Q4_K_M file." They don't explain what that label means, why it matters for your specific machine, or what to do when your model crawls or crashes. This guide does all three.

You'll learn what parameters and quantization actually are, what 4-bit and 8-bit mean in plain terms, how to calculate whether a model fits your RAM before you download it, and how to set things up in both LM Studio and Ollama. There's also a troubleshooting section for the three failure modes that catch most people off guard.

Parameters vs Quantization: What's Actually Different

These two terms get collapsed together constantly. They're not the same thing.

Parameters are the model's knowledge. When a model trains on text, it adjusts billions of numerical weights. Those weights encode everything the model has learned about language, reasoning, and facts. A 7B model has roughly 7 billion of them. A 70B model has ten times that. More parameters generally means a more capable model, but also a much larger file and much higher memory requirements at runtime.

Quantization doesn't touch the count. It changes how each parameter is stored.

Think of it like this: imagine a high-resolution TIFF photograph and a compressed JPEG of the same image. The JPEG doesn't remove objects from the scene. It reduces the precision used to represent colors and edges. The image is recognizably the same, but the file is a fraction of the size. Quantization does the same thing to model weights — it rounds each number to fit inside fewer bits.

The format that makes all this practical for local use is GGUF. It's the file format developed by llama.cpp that packages a quantized model into a single portable file, with metadata that tools like LM Studio and Ollama can read directly. Before GGUF, running quantized models locally required more manual setup. Now it's a download and a click.

What 4-bit, 5-bit, and 8-bit Actually Mean

Every weight in a model is a number. The bit depth tells you how many binary digits are used to represent that number.

• 8-bit can represent 256 distinct values (2⁸)
• 5-bit uses more precision than 4-bit, storing each weight with a finer range of possible values
• 4-bit is a coarser approximation, trading precision for a smaller footprint
• 2-bit can represent only 4 distinct values (2²) — the most aggressive compression, with real quality cost

Full-precision models typically use 16-bit or 32-bit floats. Quantization compresses those down. Less precision means smaller file, faster load, lower memory footprint — but also a coarser approximation of each weight.

The Naming Conventions Explained

When you browse Hugging Face or the Ollama library, you'll see filenames like Q4_K_M, Q5_K_S, Q8_0. Here's how to read them:

• The number after Q is the bit depth (4, 5, 6, 8).
• _K means the file uses a newer quantization method called k-quants, which is more accurate than older integer quantization at the same bit depth. If you see a K variant, prefer it over the plain version.
• _M means "medium" within that family. _S is "small" (slightly smaller file, slightly lower quality). _L is "large" (slightly bigger, slightly better).
• Q8_0 uses 8-bit integer quantization without the k-quant method. It's very close to full quality and commonly used as a high-fidelity baseline.

The practical ranking from smallest/fastest to largest/best quality runs roughly: Q2_K → Q4_K_S → Q4_K_M → Q5_K_M → Q6_K → Q8_0.

For most hardware setups, Q4_K_M and Q5_K_M are the practical sweet spot. Both fit in reasonable RAM and run at usable speed, with a modest quality drop from full precision on instruction-following tasks. Q4_K_M is the better pick when memory is tight. Q5_K_M is worth the extra memory if you're doing anything that requires precise reasoning or structured output.

A Real Quality Comparison

To make this concrete: take Mistral-7B-Instruct and run the same prompt through Q4_K_M and Q8_0.

Prompt:

Write a short story about a cat who learns to play the piano.

Q4_K_M output:
The cat, Mittens, enjoyed playing the piano. She liked pressing keys with her paws. Sometimes she would get confused and meow at the music. It was fun!

Q8_0 output:
Mittens was a curious cat. One day, she discovered an old piano in the attic. Intrigued, she leaped onto the keys, producing a jumble of sounds. Soon she was practicing diligently, her paws surprisingly nimble as she learned to play simple melodies. The house filled with the charming sound of a cat's musical journey.

The Q8_0 output is more coherent, more specific, and handles narrative structure better. For creative writing, Q8_0 is noticeably better. For summarizing a document or answering a factual question, Q4_K_M is often indistinguishable from Q8_0 and loads significantly faster.

The takeaway: match your quant to the task, not just the hardware.

How to Pick the Right Quant for Your Hardware

Stop guessing. The formula below takes 60 seconds and tells you exactly what your machine can run.

The RAM/VRAM Formula

Required Memory (GB) =
  (Parameters in billions × Bits per weight / 8)
  + (Context tokens × KV cache bytes per token / 1,073,741,824)
  + OS and app overhead (roughly 1–2 GB)

Breaking that down:

• Parameters × bits / 8 gives you the model weights in bytes, then converted to GB. A 7B model at 4-bit: 7,000,000,000 × 4 / 8 = 3.5 GB.
• KV cache is the memory used to store the attention state as the model generates tokens. It grows with context length and model architecture. A rough estimate for many 7B models is around 256 MB to 512 MB at 2048-token context, but this varies by architecture. Use 0.5 GB as a safe buffer for typical context windows.
• OS overhead: add 1.5–2 GB for the OS, the runtime, and whatever else is running. More if you have a browser with 40 tabs open.

Quick Reference by Hardware Tier

A note on Apple Silicon: Macs with M-series chips use unified memory, meaning the same physical RAM serves both the CPU and the GPU. This means you can run larger models on Apple Silicon than a dedicated GPU with the same memory spec on a Windows machine would typically allow. It's a genuine advantage for local AI work. For more on hardware choices and local model setups, see the Cosmos 3 Edge local AI setup breakdown.

One Worked Example

You have 16 GB of RAM. You want to run a 13B model at Q4_K_M. A 13B model at 4-bit weighs roughly 7–8 GB. Add 0.5 GB for KV cache at a 2048-token context, and 2 GB for OS overhead. Total: around 10 GB. That leaves 6 GB of headroom, so it fits. If you want Q5_K_M instead (roughly 9–10 GB for the weights), you're cutting it closer but still viable if you close other apps.

Running Quantized Models in LM Studio

LM Studio is a desktop app with a GUI. It's the fastest way to get started if you're not comfortable with a command line.

1. Download and install LM Studio from lmstudio.ai. It's free and runs on Mac, Windows, and Linux.
2. Open the Model Hub (the magnifying glass icon in the left sidebar). Search for any model by name, for example "Mistral 7B Instruct" or "Llama 3".
3. Pick your quant. LM Studio shows you every available GGUF variant for each model, with file sizes listed. For an 8 GB machine, filter to files under 5–6 GB. The filename will tell you the quant: look for Q4_K_M or Q5_K_M for a good balance of size and quality.
4. Download and load. Click the download icon, then switch to the Chat tab and select your model. LM Studio will load it into memory.
5. Adjust context window. In the model settings panel (right sidebar in Chat), you can set the context size. Start at 2048 tokens. Increasing it improves coherence on longer tasks but uses more memory.
6. Run your prompt. LM Studio shows token generation speed in the bottom bar. If it's below 5–6 tokens/sec, that's a signal your quant is too large for comfortable use.

If you want to get more out of your models once they're running, structured prompt templates make a significant difference. Ultra Prompt's local LLM category has templates built specifically for models you run offline.

For a deeper walkthrough of running specific models in LM Studio and Ollama, the guide on running Inkling locally covers the full setup process with real examples.

Running Quantized Models in Ollama

Ollama runs from the command line and handles model management cleanly. It's better than LM Studio for scripting, automation, and integration with other tools.

1. Install Ollama from ollama.com. On Mac, it's a drag-to-Applications install. On Linux, one curl command. On Windows, an installer.
2. Pull a model. Open your terminal and run:

   ollama pull mistral

   This pulls the default Mistral variant from Ollama's library. Ollama automatically picks a quantization level that fits typical hardware.
3. Pull a specific quant. To get a specific quantization, use the tag syntax:

   ollama pull mistral:7b-instruct-q5_K_M

   Browse available tags on the model's page at ollama.com/library.
4. Run it:

   ollama run mistral

   This opens an interactive chat session. Type your prompt and press Enter.
5. Use it from a script or app. Ollama exposes a local REST API at http://localhost:11434. You can call any loaded model from Python, Node, or any HTTP client without changing your code when you swap models.

One thing worth knowing: Ollama manages model storage and GPU/CPU allocation automatically. If you have a GPU, it'll use it. If the model is too large for VRAM, it offloads layers to RAM. You can see which layers are on GPU vs CPU by running ollama ps.

If you're new to Ollama entirely, the step-by-step guide to installing and running your first local model covers the full setup from zero.

Common Problems and What's Actually Causing Them

Out of Memory Errors

Your model file is loading fine but the process crashes or throws an OOM error partway through generation. This usually means the KV cache is pushing you over your RAM/VRAM limit, not the weights themselves. The fix: reduce your context window size first (try 1024 or 512). If that doesn't work, drop to a lower quant. If you're on a GPU and still getting OOM errors, check whether layer offloading is configured in LM Studio (under GPU settings) or whether Ollama is trying to load more layers to GPU than it can hold.

Generation Is Too Slow to Be Useful

If you're getting under 3–4 tokens per second, the model is probably hitting RAM bandwidth limits rather than compute limits. This happens when the model is too large to fit in VRAM and has to stream weights from system RAM. Dropping from Q5_K_M to Q4_K_M on a 7B model meaningfully reduces file size, which can improve throughput when you're bandwidth-constrained. Closing other applications helps. Restarting your machine before a long session also clears fragmented memory that can throttle performance.

Output Quality Dropped Noticeably

If outputs suddenly got worse after switching models or quants, check three things. First, are you at Q2_K or Q3_K? Those tend to degrade significantly on tasks requiring structured reasoning or precise formatting. Second, is your system prompt still correct for the model you switched to? Different model families (Llama 3, Mistral, Qwen, etc.) use different chat templates, and mismatching the template against the model can hurt output quality more than quantization does. Third, try a Q5_K_M or Q6_K variant if your hardware allows — it often recovers the quality loss without a huge memory increase.

Quantization and Structured Output

This one's underdocumented. If you're using a local model to produce JSON, code, or any other structured format, lower quants are disproportionately unreliable at following format constraints. A Q4_K_M model that handles conversational text well may start producing malformed JSON or dropping closing brackets at Q2_K. For structured output tasks, don't go below Q4_K_M, and prefer Q5_K_M or Q8_0 if your hardware can hold it. Pair structured output prompts with explicit format instructions and, where possible, use a grammar-constrained generation mode if your tool supports it (LM Studio's JSON mode, for example).

FAQ

How much RAM do I need to run a 7B or 13B model locally?

For a 7B model at Q4_K_M, expect roughly 4–5 GB of memory for the weights, plus 0.5–1 GB for KV cache and OS overhead. 8 GB total RAM is workable with other apps closed; 16 GB gives comfortable headroom. A 13B model at Q4_K_M needs roughly 7–8 GB for the weights. 16 GB total RAM handles it, though you'll want to keep context windows modest. Use the formula in the "How to Pick the Right Quant" section above to calculate your specific case.

What's the difference between Q4_K_M and Q5_K_M?

Q5_K_M uses 5 bits per weight instead of 4, which means it stores each parameter with more precision. In practice, Q5_K_M is noticeably better for tasks involving structured output, multi-step reasoning, and creative writing. The tradeoff is roughly 15–25% more memory and somewhat slower generation. For everyday summarization, Q&A, and chat, Q4_K_M is usually good enough. For code generation or anything requiring exact formatting, Q5_K_M is worth the extra cost if your hardware can handle it.

Why is my model running slowly after quantization?

Lower bit depth reduces file size but doesn't always speed up generation. The bottleneck is usually memory bandwidth. If the model doesn't fit in VRAM, the GPU has to stream weights from system RAM each time it generates a token, which is slow regardless of quant. If you're already in VRAM, a lower quant can help because smaller weights load faster. Check whether your model is GPU-resident by running ollama ps or checking LM Studio's GPU usage indicator. If most layers are in RAM, that's your bottleneck, not the quant level itself.

Can I run bigger models on a Mac with unified memory?

Yes, and this is one of the strongest reasons to run local AI on Apple Silicon. Because the CPU and GPU share the same physical memory pool on M-series chips, the full RAM allocation is available to the model. A 32 GB machine can run larger models than a typical 32 GB discrete GPU setup would allow, though the exact ceiling depends on which model family you're using. Check the GGUF file size for your target model and compare it against the formula in this guide before downloading. The memory bandwidth on M-series chips is also competitive, so generation speed is reasonable even on larger quants.

How do I convert a model to GGUF myself?

The conversion process uses the convert_hf_to_gguf.py script from the llama.cpp repository, followed by quantization with the llama-quantize tool. You need the original model in Hugging Face format, Python, and the llama.cpp build. The full process is covered in the llama.cpp README on GitHub and is well-documented for most major architectures. It's not quick to set up the first time, but it's the right path if you want to quantize a model that isn't already available in GGUF format on Hugging Face.

Does quantization affect how a model follows system prompts?

At Q4_K_M and above, the effect is minimal for most instruction-following tasks. Below Q4, there's a real degradation in how reliably the model follows complex or multi-part instructions. The model's ability to stay "in character" with a persona, maintain a specific output format, or honor negative constraints (things you've told it not to do) erodes faster than its raw language quality. If your workflow depends heavily on precise instruction-following, treat Q4_K_M as your floor.

The Part Most People Miss

Picking the right quant gets you a model that runs. But a model that runs is just the starting point. The difference between mediocre and genuinely useful local AI is almost always the prompt, not the quantization level. A Q4_K_M model with a well-structured prompt will outperform a Q8_0 model with a vague one on virtually any task that matters.

Hardware lets you run the model. The prompt tells it what to do. Both have to be right.

If you want structured prompt templates built for local models, Ultra Prompt has a full library covering technical, creative, and business use cases — designed to get useful output on the first try, not the fifth.