A local AI model needs enough memory to hold its weights, plus extra room for the conversation you're having with it. The weights are easy to estimate. Multiply the model's parameters by the number of bits each one is stored in and divide by eight, and you get gigabytes. An 8-billion-parameter model stored at 4 bits needs about 4 GB for its weights, and the conversation, called context, costs memory on top of that. Memory is the number to shop for. A model that doesn't fit won't run at all, and a model that barely fits will choke the first time you hand it a long file.
If you're still deciding whether running AI on your own machine makes sense, what a local LLM is and how it works covers the basics, and whether your business needs private AI at all covers the case for one. This post assumes you've decided and want to know what to buy.
Bits, gigs, and context in plain English
Spec sheets for local models throw three terms at you and assume you know them. Each one is simpler than it sounds.
A model is a large set of numbers it learned during training, called parameters. Sizes are quoted in billions of them, so "8B" means 8 billion numbers. Bits are how precisely each of those numbers is stored. Full-size models keep 16 bits per number. Squeezing them down to 8 or 4 bits is called quantization, and the Hugging Face guide to quantization describes the trade plainly: Lower precision cuts the memory a model needs while trying to hold on to as much accuracy as it can. For drafting and summarizing, 4-bit is the normal way to run a model on an ordinary computer, and the drop in quality is small.
The arithmetic follows directly. Eight bits make a byte, so parameters times bits divided by eight gives bytes, and billions of bytes are gigabytes. Run it for that same 8B model and every halving of the bits halves the memory its weights need, so 4-bit takes a quarter of what 16-bit does.
The numbers
| Precision | Memory |
|---|---|
| 16-bit | 16 GB |
| 8-bit | 8 GB |
| 4-bit | 4 GB |
Weights only, before context.
Source: Calculated as parameters × bits ÷ 8, weights only, before context
Gigs are just that memory. The weights have to sit in it the whole time the model is working. On a recent Mac the processor and graphics chip share one pool, so the system memory is what counts. On a Windows or Linux desktop with a graphics card, the card's own memory is what counts, and consumer cards commonly carry 16 or 24 GB.
Context is the separate cost, and it's the one that surprises people. A model's context window is how much text it can work with at once, meaning your question, any documents you paste in, and every earlier message in the conversation. It's measured in tokens, which are word fragments, and a page of text runs a few hundred of them. While the model reads, it keeps a working record of every token so it doesn't have to start over with each new word. That record lives in memory alongside the weights and grows as the conversation grows. The Hugging Face notes on the KV cache, which is the technical name for that record, warn that it "can occupy a significant portion of memory and become a bottleneck for long-context generation."
So a context window is a ceiling, and using it isn't free. Ollama's listing for Google's Gemma open models, as read in October 2026, shows windows of 128,000 to 256,000 tokens. A laptop that holds the weights comfortably can still run out of room long before it reaches that ceiling. An accountant whose setup handles a two-page engagement letter and stalls on a year of bank statements has usually hit context, not a bad model.
How much memory does each size of model need?
Below, the table applies the bits rule at 4-bit, the common way to run locally. The second column is the weights alone, before any context. The third is the machine memory to look for once you leave room for the conversation and for the computer's own programs.
| Model size | Weights at 4-bit (parameters × 4 ÷ 8) | Machine memory to look for | What it's good for |
|---|---|---|---|
| Small, 2 to 4 billion parameters | 1 to 2 GB | 8 GB or more | Short emails, summarizing a page, sorting and tagging text |
| Mid-size, 12 to 14 billion | 6 to 7 GB | 16 to 32 GB | Long documents, pulling facts out of forms, first drafts in your house style |
| Large, 26 to 32 billion | 13 to 16 GB | 32 GB or more | Closer to the paid apps on writing and reasoning, still behind the top cloud models on hard problems |
| Very large, 70 billion and up | 35 GB and up | 64 GB or more | Workstation territory, and rarely worth it for a small office |
Real downloads run a little above the bare rule. As listed in October 2026, Ollama puts a 31-billion-parameter Gemma model at about 19 to 20 GB, where the arithmetic says 15.5. As far as I can tell, the gap comes from parts of the file kept at higher precision and the extra machinery that ships with the model. Treat the rule as the floor and the listing as the truth for a specific model. Hugging Face also runs a free quantization picker that checks whether a given model fits a given machine.
What hardware does a local model need?
Buy memory first and speed second. A faster chip makes answers arrive sooner, while more memory decides which models you can run in the first place and how long a document they can take.
That points most offices toward one of two machines. A Mac with 32 or 64 GB of shared memory runs mid-size and large models without a separate graphics card, and it's quiet enough to sit on the front desk of an inn. A desktop with a graphics card does the same work faster per dollar if someone on staff is comfortable building or maintaining one. Either way, the card's or the Mac's memory is the line you read on the spec sheet.
Match the machine to the job you actually have. Tidying a salon's appointment notes into client records is a small-model job, and almost any recent laptop handles it. For an HVAC outfit, having a model read a season of service tickets and draft renewal quotes is a mid-size job with long context, and that's where 32 GB stops being optional. A retail shop asking for something closer to the paid apps' writing quality is looking at a large model and the memory that comes with it, and might be better served by weighing frontier against open models before buying anything.
Where any of this sits in the bigger picture is covered in how to use AI in your business. Count how long your own documents run before you pick a machine.