But what is an artificial intelligence running on your own computer for?
You can download an AI and use it with no internet, free, with nobody reading your questions. What it takes, what you give up, and the few cases where it is worth it.
- Level
- intermedio
- Time
- 12 minutes
- What you need
- A computer with 16 GB of memory, or 8 GB for the smaller models
Normally, when you write to an AI assistant, your question sets off, crosses half the world, is processed in a company’s data centre and comes back as an answer. It works very well, and it has three consequences: you need a connection, there are limits or subscriptions, and what you write leaves your house.
For a while now there has been an alternative that used to be specialists’ territory: downloading the model and keeping it on your own computer. Free, no counted messages, no network, and nobody reading anything. There are still limits, though, and they are your computer’s: we will go through them. The right question, as always, is who it actually suits.
Model: the very large file holding what the AI has learned. Downloading it is like downloading a program. Local model: a model that runs on your computer rather than over the internet. Parameters (7B, 14B and so on): the measure of how big a model is; the bigger it is, the more memory it needs. How good it is also depends a great deal on how recent it is: a small model released now often beats a big one from two years ago. Quantized: a compressed version, smaller on disk and slightly less precise. RAM: the computer's working memory, which is what counts here.
What it takes, in practice
Less than you might think, but not nothing.
What matters is memory, not processor power. A model of normal size for home use, one of those around 7 or 8 billion parameters in compressed form, takes roughly 4 to 5 GB on disk and about the same in memory while it works. The programs that run them list 16 GB of memory as the recommended configuration, with smaller models working on 8 GB.
A dedicated graphics card is not required, but it changes the speed a great deal. Without one the model still works and writes more slowly, at about the pace of a person typing. Recent Apple computers are a favourable case, because memory is shared between processor and graphics.
Disk space: count ten or twenty GB if you want to keep two or three different models.
How it is done, without getting technical
There are two routes, and neither needs you to write code.
A program with a window, such as LM Studio: you install it, it contains a catalogue of models, you choose, download, and find yourself in front of a chat window identical to the ones you are used to. This is the route for anyone.
A command-line program, such as Ollama: leaner, preferred by people in the trade and by anyone wanting to plug it into other software.
Either way the real work is choosing the model, and the advice is to start with a small one, try it, and go up only if your computer copes.
Downloading a model does not "install an artificial intelligence in your computer" in the unsettling sense. It is a file a program reads when you open it, and it does nothing while you are not using it. Delete the file and that is that, as with any other program.
What you give up, and better to know first
The quality gap with the big online services is real and wide, and anyone telling you otherwise is wasting your time.
A small model running at home does well on bounded tasks: summarising a text you give it, rephrasing a sentence, correcting, pulling data out of a list, answering questions about a document you supplied, writing a first draft you will tidy up.
It does markedly worse on: complex reasoning, mathematics, non-trivial programming, facts about the world and current events (its knowledge stops at training time and it has no internet), and generally languages other than English. That last point is one to take seriously: in any other language, a small model sounds small, far more than it does in English.
The three reasons it is worth it
Documents that must not leave. This is the strongest reason. If you work with medical records, legal papers, clients’ accounts or material under confidentiality, a local model lets you use AI without handing those documents to anyone. It is not paranoia, it is obligations you may already have.
Working with no network. On a train, on a plane, somewhere with a poor connection, in a warehouse or a basement. The local model works just the same.
No limits and no monthly fee. No counted messages, no subscription, and above all no model changing under your feet: the file you downloaded stays as it is and will still be exactly the same one in a year, which is not guaranteed with online services. Individual answers still vary each time, as they do everywhere.
I would add a fourth, less practical and more honest: understanding how it works. Watching a model write on your own computer with the connection unplugged takes a good deal of magic out of the subject, and magic removed is always a gain.
Who can skip it
If you want the best possible quality, local is not the road. If you have a laptop with 8 GB of memory and no graphics card, the experience will be frustrating. If you would use AI twice a month, an online service’s free tier is more than enough. And if downloading multi-gigabyte files, choosing versions and updating programs bores you, this is a hobby that will not repay you.
An AI on your own computer makes sense for someone with a specific reason: confidential data to handle, work without a connection, or the wish to understand how it works. For everything else an online service is better and more convenient, and that deserves saying without snobbery. Trying it anyway costs one afternoon and breaks nothing: if you have 16 GB of memory, it is an afternoon well spent.
If you decide to try
Start with a small model, three or four GB, and give it the things it is good at: paste in a text and ask for a summary, or a rewrite. Judge the quality on that, not by asking who won the last championship, because it cannot know.
First, though, it is worth knowing what the phone in your pocket already does with no connection, because that often covers the real need: I cover it in what artificial intelligence that works without internet is for. And to understand why even the big models sometimes invent things, the explanation is in what a language model is.