A local LLM is an AI model that runs on a computer you own instead of on an AI company's servers. LLM stands for large language model, which is a program trained on an enormous amount of text until it can predict, one word-piece at a time, what a useful reply looks like. With a local one, you download the model as a file, a free app loads it into your computer's memory, and from then on it answers your questions using only your own processor. No internet connection is needed for the answering, and nothing you type leaves the machine.
The everyday AI most people know, ChatGPT or Claude or Gemini in a browser, works differently. Your question travels to a data center, a very large model runs there, and the reply travels back. The two can look identical on screen, a text box and a scrolling answer, which is why "local" sounds like a setting when it's really a different address for the work.
What is inside a local model?
Mostly numbers. When a company trains a language model, the result is a long list of numerical settings called weights, billions of them, each one a small dial that was turned during training. The weights are the model. Together, those tuned dials turn your words into a likely next word, and then the next one, without looking anything up.
Some companies publish their weights for anyone to download. These are called open-weight models, and they're what makes local AI possible at all. Meta's Llama family, Google's Gemma, Mistral's models, and several from Chinese labs such as Alibaba's Qwen sit on download sites like Hugging Face, a public library where model files are shared. The top models from OpenAI and Anthropic are not published this way, so they can't run on your computer. Frontier vs. open-source AI covers what that trade costs you in capability.
A model file is big, often several gigabytes, sometimes far more. How big a file your computer can handle is its own question, and sizing a local AI model takes it up in plain terms.
Where does a local model run?
On the computer in front of you, or on one in your back office. The model sits on the hard drive like any other large file. When you start the app, it copies the weights into memory, either the computer's main memory or the separate memory on a graphics card, because the math runs much faster there. Then it waits for a question.
You rarely touch the model directly. An app sits around it and does the ordinary work of a chat window. It takes your typing, hands it to the model, and shows the reply as it forms.
Two free apps cover most beginners. Ollama runs quietly in the background and is popular with people comfortable typing a command or two. LM Studio looks more like a regular program, with a search box for finding models and a chat screen for using them. Either runs on Windows, a Mac, or Linux. That surrounding program shapes much of what using a model feels like, as the program around the model explains.
Run it on one shared office computer and the whole team can reach it over the office network, still without anything going outside the building. That setup takes more care than a single laptop, but the principle doesn't change.
How does it answer without the internet?
It never needed the internet to think. A cloud model needs a connection because the model lives somewhere else, and the connection is how your question gets there. Once the weights are sitting in your own computer's memory, the question doesn't have to go anywhere.
Underneath, the answering is arithmetic. Your app breaks your question into small chunks called tokens, roughly word-pieces, and turns each into numbers. The model runs those numbers through its weights and produces a score for every possible next token. The app picks one, adds it to the reply, and runs the whole thing again with that token included. A long answer is that loop repeated a few hundred times. The mechanisms behind the magic walks through why something so mechanical can read as fluent.
Both apps' own documentation describes this. LM Studio's offline guide says the app "can operate entirely offline" once you have the model files, and that nothing you enter in a chat leaves your device. Ollama's FAQ says that when you run locally, the company doesn't see your prompts or data. Read the rest of that page, because Ollama now also offers cloud-hosted models and web search, and those do send your words out.
Ollama has a setting to switch the cloud features off. Check that setting before you trust the app with anything private.
The test I'd run before trusting any setup is simple enough to do in a minute. Turn off the Wi-Fi, unplug the network cable, and ask the model to summarize a page of something. If it answers, the work is happening on your machine.
What the model can't do offline is learn anything new. Its knowledge stops at the date its training ended. Ask a local model about this week's snowfall on Beech Mountain or a fee schedule that changed in September and it will either say it doesn't know or, worse, produce a confident guess. What it does well is work with whatever you give it in the moment, such as the letter you paste in, the contract you attach, or the list of last month's appointments.
| Where it runs | What leaves your machine | Needs internet | |
|---|---|---|---|
| Cloud model | A provider's servers | Your prompt and files | Yes |
| Local model | Your computer | Nothing | No, after download |
What does using one look like day to day?
Less exotic than the name suggests. An accounting office might keep a local model on one machine for summarizing long client letters. A lodge might use one to draft replies to booking questions from its own policy sheet, pasted in at the start. A salon owner could clean up product descriptions for the shop's website. Each is a Level 2 use, the model working from material you hand it, and the work is the same as it would be in a cloud chat window, minus the trip to a data center.
The differences show up in speed and polish. On an ordinary laptop, a local model types its answer noticeably slower than a cloud model does, and smaller models stumble on long or tricky requests more often. On a well-equipped machine the gap narrows a lot.
Whether your business should bother is a separate decision, and it turns on what data you hold and who has a say over it. Do you need private AI? works through it.