How a computer really works, from the ALU to the NPU

The box of a new computer carries four or five acronyms: CPU, core, GPU, NPU, TOPS. They look like the names of unrelated parts thrown together, and instead they tell a single story.

The story goes like this. Every time one kind of calculation became important enough, somebody built a piece of silicon that mostly does just that, and does it on far less power. From the FPU of the 1980s to today’s NPU it is always the same trade. Let’s take it from the start, trying not to say anything that would make a person who does this for a living wince.

1945: the model we still use

In June 1945 a document titled First Draft of a Report on the EDVAC starts circulating, signed by John von Neumann. That is where the name “von Neumann architecture” comes from, although it is a fairly well known historical injustice. The work belonged to the Moore School group, and J. Presper Eckert and John Mauchly had at least as good a claim to sign it. Von Neumann wrote well and his name was on the cover.

The idea, though, is decisive: a single memory, holding both the data and the instructions, in the same format and addressed the same way. Before that, the program was an arrangement of cables and switches. From that moment the program is a number like any other, and a machine can load a different one without being rebuilt.

The rest of the model is a handful of parts, namely the memory, the control unit, the arithmetic unit, the input and the output. In the textbook description, it works as a cycle that repeats.

1. Fetch the next instruction from memory.
2. Decode it, that is, work out what it is asking for.
3. Execute: do the sum, or move a number, or jump to another instruction.
4. Start again.

The clock is the beat that keeps the various parts of the chip agreed on when a result is stable and can be recorded. Three gigahertz means three billion beats a second.

Be careful, though, about what that cycle describes. It describes the behaviour the program sees, not what happens inside the chip. A modern high-end processor does not take one instruction, finish it and move on to the next. It keeps many of them in flight together, executes them in an order other than the one written down, guesses the outcome of branches so as not to sit idle, and when it guesses wrong it throws away the work it has done and starts again. The final result has to be identical to that of an orderly execution, and that is the contract. The route taken to get there is not.

You need the same distinction for another commonplace, the one that says today’s processors are still von Neumann machines. Caches, out-of-order execution and separate paths for instructions and data inside the chip concern how the machine is built; the contract with the programmer is still the one from 1945, a single memory with data and instructions inside it. Von Neumann architecture, with an implementation that borrows heavily from the rival model, the Harvard one. Two different levels of the discussion.

The ALU, and why megahertz are no longer enough

The ALU is the workbench. It adds, subtracts, compares, does AND and OR on whole numbers. It takes the operands from the registers and puts the result back into them. A modern core does not have just one, and alongside it live other specialised units: multiplication, division, floating point, operations on groups of numbers.

For twenty years the normal way to compare two computers was to look at the clock. People said “a 486 at 66 megahertz”, then “a Pentium at 200”, then came the gigahertz, and through the late 1990s advertising was a race to the highest number. For a while that number worked, because the processors being compared were fairly similar to one another: for a given design, more beats a second means more work done.

Then the criterion broke. And it helps to understand why, because it is the same reason you should not trust TOPS today.

The time a program takes depends on three things multiplied together: how many beats a second, how much work goes through on average at each beat, and how many instructions it takes to do that work. The clock is only the first of the three factors. A processor can have more of it and still be slower, if the other two are worse. That is exactly what happened to the Pentium 4, which reached 3.8 GHz and was slower, on many programs, than rival processors running at two. Its internal assembly line had been lengthened precisely to allow high frequencies, and every time a branch prediction went wrong that extra stretch had to be paid for.

Here we should also clear away the other classic simplification, “the processor executes one instruction per clock tick”, which is imprecise in two opposite ways.

First, a single instruction occupies many beats. The work is organised as an assembly line divided into stages, quite a few of them in high-performance designs: while one instruction is being executed, the next is already being decoded and the one after that is being fetched. So it is not the instruction that lasts a beat, it is the line, once it is up to speed, that can deliver one per beat.

Second, there is not just one line. High-performance processors are superscalar, that is, they have several execution units in parallel and can start more work within the same beat. In many architectures, x86 chief among them, the instruction written in the program is first translated into one or more simpler micro-operations, and it is on those that the machine really works. How many of them can be started per cycle depends on the specific design: it is a number that changes from architecture to architecture and from generation to generation, and it is one of those figures that only means something if you name the precise chip.

The curious thing is that this capacity nearly always goes unused. Measurements on ordinary code give low averages, of the order of one or two instructions completed per cycle, with large differences from program to program: a tight, easily predicted loop runs well, while a program that jumps from one structure in memory to another does much worse. The main reason is a single one, and it will come back at the end of this article. The processor is often sitting still, waiting for data to arrive.

The practical moral. Gigahertz still count, but they are for comparing two models in the same family, not two chips picked at random. Between a recent processor at 3 GHz and one from six years ago at 3.5 GHz, the recent one wins almost every time.

The FPU: why the decimal point needed a part of its own

Adding two whole numbers is easy. Adding 1.5 × 10³ and 2.7 × 10⁻² is another story, because you have to compare the exponents, line up the digits, add, renormalise the result and round it. That is a longer and more expensive path through the silicon, and that is why floating point numbers got a department of their own, the FPU.

For years it was a separate chip, bought separately. The Intel 8087, released in 1980, slotted into an empty socket beside the processor, and in most computers that socket stayed empty. With the 486DX in 1989 the FPU ended up on the same piece of silicon as the processor, at least in the PC world. Floating point calculation in hardware, it should be said, had existed long before. What changes in those years is that it arrives on the home computer.

Meanwhile the most important thing had happened, which was the IEEE 754 standard, approved in 1985 and today at its 2019 revision. It helps to say precisely what it standardises, because it is often summarised badly. It defines the formats in which numbers are represented, the operations and their behaviour, the rounding modes, the special values (signed zero, the infinities, and the non-numeric result called NaN) and the handling of exceptions, for instance what should happen on a division by zero or an overflow. The consequence that matters to a person using a computer is that operations carried out according to the standard give the same results on different machines. Before 1985 every manufacturer went its own way, and the same program could answer differently depending on where it was running.

One detail remains that you meet sooner or later in a spreadsheet. Binary floating point numbers cannot represent every decimal exactly. In binary, 0.1 has no finite form, just as 1/3 has none in decimal. That is why in many languages 0.1 + 0.2 does not come to 0.3 but to something like 0.30000000000000004. The computer is not getting it wrong: it is paying the price of representing with a finite number of bits something that is not finite. For the same reason, adding the same numbers in a different order can give slightly different results, which has concrete consequences when the work is divided among many calculation units.

A curiosity worth the detour. In 1994 a mathematician discovered that certain divisions on early Pentiums gave a wrong result from the fourth decimal place onwards, because of a few missing entries in an internal table. Intel ended up replacing the processors for anyone who asked and booked a write-down of about 475 million dollars. It is one of the reasons arithmetic units today are checked with formal methods, that is, by proving mathematically that they behave as they should.

SIMD: one instruction, many results

Towards the end of the 1990s a new problem arrives: video, audio, images. Jobs in which the same operation has to be done on very many numbers. Raise the brightness of every pixel, add a value to every sound sample.

Doing that with one instruction per number is a waste, and the reason runs deeper than it looks. A much cited presentation by Mark Horowitz gives the order of magnitude, for the technology of those years: on a general-purpose processor, the energy spent handling the instruction, that is, fetching it, decoding it and keeping the surroundings running, is much greater than the energy spent doing the sum itself. The sum is the cheap part. What costs is the organisation around it.

The answer is called SIMD, for Single Instruction, Multiple Data: a single instruction operating at the same time on a group of numbers side by side. One fetch, one decode, many results. On Intel and AMD processors it is the family running from MMX (1997) to SSE (1999) and on to AVX (from 2011) and its successors; on Arm processors, the ones in phones and Macs, the equivalent is called NEON.

It is not free, because it works when the data are numerous, independent and laid out regularly, and it yields little when each element calls for a different decision. But keep it in mind, because this pattern returns throughout the rest of this article: narrow the kind of work, cut down the surroundings, gain a great deal in efficiency.

Cores: when frequency stopped climbing

Until the early 2000s every generation of processors went faster in frequency than the one before, and all you had to do was wait. Then, around 2004 to 2006, the climb stopped against a physical limit, because the higher you go, the more power is consumed and the more heat has to be got rid of within a few square millimetres. Intel abandoned the designs aiming at 4 GHz and changed tack: instead of a faster processor, more processors in the same chip. On home computers the first dual cores arrive in 2005; in servers they had existed for some years already, and machines with several separate processors for decades.

Here there are two things that are nearly always confused.

Moore’s law is about the number of transistors that can be put on a chip, and it has carried on, slowing down and becoming much more expensive. Dennard scaling is something else. It said that as transistors shrank the voltage could be lowered too, so power per unit of area stayed constant and frequency rose almost for free. It is this second law that broke, and on Hennessy and Patterson’s account it did so gradually, chiefly between 2007 and 2012. The frequency wall came first, the end of the law that made it free came after.

And are cores useful? Yes, but not in the way people think, because eight cores do not make a program eight times faster. A job speeds up only to the extent that it can be split into independent parts, and almost every program has a part that has to stay in the queue. That is why the speed of a single core still matters a great deal.

A laptop today generally has between six and sixteen cores, and they are often of two kinds, some big and fast, others smaller and more efficient. The small ones are not “slow cores for when the computer is doing nothing”: they take up far less space, so for a given amount of silicon you can fit more of them, and when there is a lot of parallel work to grind through they make a serious contribution. The drawback is that the operating system now has to decide where to send each program, and getting that choice wrong is noticeable.

The GPU: thousands of lanes, no soloists

3D graphics has a convenient property. The colour of a pixel can largely be worked out independently of the others. They are a great many similar calculations, which can be done together.

A machine was built specially on that property. NVIDIA called the GeForce 256 a “GPU” in 1999, and that is the moment the name enters common speech; the claim to be “the first GPU in history” is a generous one, though, because boards with geometry processing in hardware existed before, in workstations and elsewhere. What really changes comes later, in two steps. First the fixed parts of the chip become programmable (the shaders, from the early 2000s), then, in the middle of the decade, they stop being divided by task and become a single pool of general-purpose units assignable to one job or another. At that point, largely as a side effect, the graphics card also becomes a general-purpose parallel computer. In 2007 NVIDIA publishes CUDA and programming it stops being a conjuring trick.

How it works, in two points that count for more than any tally of “cuda cores”.

The first is that the units work in close-knit groups. Threads are organised in groups (32 in NVIDIA architectures) that share the same instruction at the same moment. If inside a group half take one branch of an “if” and half the other, the machine tends to execute first one branch and then the other, keeping the lanes that are not concerned idle. That is why GPUs yield less on code full of decisions that differ from element to element. It has nothing to do with how clever the machine is, and everything to do with its being built so that many lanes do the same thing at the same moment.

The second concerns memory. A GPU does not try to reduce the wait, it covers it with other work. A processor fights the wait with large caches and with out-of-order execution. A GPU instead keeps a great many work teams ready and, when one stalls waiting for a piece of data, it sends another forward. It is a machine designed for overall throughput, more than for finishing one single thing quickly.

Two practical consequences. The first: when you read “16,000 cores” on a graphics card and “16 cores” on a processor, the two numbers are not comparable, because what NVIDIA calls a core is a calculation lane, not a complete processor. The second: on sequential work, made of steps that depend on one another, a GPU starts at a strong disadvantage, because its lanes run at lower frequencies, have fewer mechanisms for getting on by themselves, and on top of that there is the cost of launching the work and of moving the data. That is not a law of nature, it is a question of which work you give it: change the work and you change the winner.

In 2012 a neural network called AlexNet wins the world’s most closely watched image recognition contest, trained on two gaming graphics cards. It was not the first to use a GPU, but it was the one that convinced the field. In 2017 NVIDIA adds tensor units inside the chip, circuits that perform a multiplication between small matrices in one go, with reduced-precision inputs and full-precision accumulation: on the very same chip, on the operations they are made for, they yield far more than the general-purpose units. Once again, narrowing the kind of work and stripping out the superfluous pays.

The NPU: not the fastest, the thriftiest

Simplifying a good deal, and saying so, the bulk of a neural network’s work lies in matrix multiplications, that is, enormous quantities of “multiply and add”, often with reduced-precision numbers. It is not everything a network does, but it is the part that dominates calculation time, and it is enough to explain the hardware.

An NPU is an accelerator designed to run precisely that kind of calculation very efficiently. It is not a general-purpose processor and does not need to be, because it gives up part of a CPU’s flexibility in exchange for much lower power consumption when the work is the work it was made for.

How it achieves this varies from design to design, and generalisations here are dangerous. Some accelerators organise the multiply-and-accumulate units in a grid crossed rhythmically by the data, the arrangement called a systolic array, which has the merit of bringing a number in from memory once only and then reusing it all the way along the grid; the idea dates from 1978, far older than the current fashion. Other designs use different organisations. Almost all of them, though, share three choices: reduced precision (typically 8-bit whole numbers, sometimes fewer, or compact floating point formats), close attention to how much data gets moved, and local memories near the calculation units, often managed by the compiler rather than left to an automatic hierarchy of caches. Saying that “an NPU has no cache” would be too absolute, though: it depends on the chip.

The historical path is well known. Google has used its TPUs in its own data centres since 2015 and published their architecture in 2017; Apple puts a neural engine inside the iPhone in 2017, at first reserved for its own features and opened to developers the year after; Qualcomm gets there by another road, evolving the signal processor from its modems. In 2024 Microsoft sets the threshold at 40 TOPS for calling a laptop a Copilot+ PC, a requirement still cited for several features in that category.

So why fit one, if in many cases the graphics card in the same computer grinds through more work per second? For the energy. Here the earlier figure returns, in its robust form: moving a piece of data in from external memory costs far more than doing an arithmetic operation on it, two or three orders of magnitude in the classic measurements. An accelerator designed to move little and to avoid paying for the surroundings of general-purpose instructions does the same work on a fraction of the watts.

That is why the features you see running on the NPU are the ones that have to stay on for a long time, on battery, without starting the fans: live captions, background blur on video calls, speech recognition. On the kind of work it was designed for, the NPU is the part that spends the least energy, and that, more than speed, is why they fit one.

Where we are now: when the bottleneck is memory

And here is the point that closes the circle, the one announced when we talked about the processor sitting still and waiting.

Over the past twenty years the calculating power of leading chips has grown far faster than the speed at which data is read from memory. By exactly how much depends on which products you take as your reference: the analysis by Gholami and colleagues, comparing successive generations of accelerators, estimates growth in calculation of several orders of magnitude against growth in memory bandwidth that is much slower, of the order of a hundredfold over the same period. The precise numbers should be taken as orders of magnitude, not as measurements. The direction, on the other hand, is beyond dispute, and it is that for years what got optimised was chiefly the thing that cost less.

You can see it clearly in the case that interests a lot of people today, running a language model on your own computer. When the model writes, it produces one token at a time (a token is a piece of a word, the unit these models work in), and to produce one it has to bring the model’s parameters from memory to the calculation units, doing relatively few sums on them. In this mode the speed depends above all on memory bandwidth, and to a first approximation tokens per second ≈ memory bandwidth ÷ bytes of the model. A 4.5 gigabyte model on a laptop that reads around 100 gigabytes a second sits in the region of twenty tokens a second, and the TOPS figure on the case does not change the result much.

It is also the reason quantisation, that is, rewriting the model with fewer bits per parameter, makes it faster in this phase. Not so much because it does fewer sums, but because there are fewer bytes to move.

Put this way, industry moves that otherwise look disconnected fall into place together. The unified memory of Apple’s chips and similar ones, where processor, graphics and neural engine read the same bank without copying data to each other. HBM memory, stacked vertically next to the processor, which in data centres has become one of the most fought-over resources. And chiplets, that is, building a processor out of several separate pieces of silicon rather than one single large block, which would have too much production waste.

Be careful, though, not to turn this into a slogan. “The bottleneck is memory” holds for many workloads, not for all. It holds for the case described above, a model generating text for a single person. The initial processing of the prompt, training, and servers handling many users at once are generally limited by calculation, because there every parameter read is reused for many requests. And outside artificial intelligence there are programs limited by the disk, by the network, or simply by the way they are written.

If you want to carry on

The practical follow-up to this article is But what is a computer “with artificial intelligence” actually for?, which looks at what really changes in the shops. If instead you want to try running a model on your own computer and see the memory argument with your own eyes, there is But what is an artificial intelligence running on your own computer for?.