Chip types

The CPU is built to finish one thing fast.

A central processing unit runs whatever it is given: an operating system, a database, a browser, a spreadsheet. Its job is to keep the latency of a single stream of instructions as low as possible. The instructions it understands are defined by its instruction set architecture, or ISA. Three matter today. x86 runs most PCs and servers. Arm is licensed to many companies and runs nearly every phone and a growing share of laptops and servers. RISC-V is an open standard anyone can implement without a licence fee, and is spreading fastest in embedded chips.

Most of a modern core is not arithmetic. It is machinery for keeping the arithmetic busy. A pipeline splits each instruction into stages so several are in flight at once. Out-of-order execution looks ahead for instructions whose inputs are ready and runs them early. Branch prediction guesses which way each decision will go before it is known, and in typical code guesses right well over nine times in ten. A wrong guess throws away the work done on the wrong path.

The rest of the die is mostly cache: small, fast SRAM close to the core that holds recently used data so the core rarely waits on main memory. Since single cores stopped getting much faster (course 01, lesson 04), CPUs have grown wider instead. Server processors now carry more than a hundred cores per socket.

L1 cache

Tens of KB per core, a few cycles away

L2 cache

Hundreds of KB to a few MB per core

L3 cache

Tens to hundreds of MB, shared

Main memory (DRAM)

Around 100 ns, hundreds of cycles away

Clock speed

Typically 2 to 6 GHz

The GPU is built to finish many things at once.

A graphics processing unit makes the opposite trade from a CPU. It gives up speed on any single task for throughput across thousands of them. Shading a frame means running the same short program on millions of pixels, and each pixel can be worked on independently. So a GPU spends its area on many simple arithmetic lanes and very little on prediction and reordering.

The lanes are grouped. In the model known as SIMT (single instruction, multiple threads), a group of 32 or 64 threads executes the same instruction together, each on its own data. When one thread in the group stalls on memory, the hardware switches to another group that is ready. GPUs hide memory latency by always having other work in hand, where CPUs hide it with caches.

The same structure suits any job made of large, regular arithmetic, and the matrix multiplications at the heart of neural networks are exactly that. Once general-purpose GPU programming became practical in the late 2000s, researchers began training networks on graphics cards. Data-centre GPUs now add matrix units (often called tensor cores) that multiply small blocks of numbers in a single step at low precision. They sit beside stacks of high-bandwidth memory that deliver several terabytes per second.

AI accelerators give up flexibility for efficiency.

An ASIC, an application-specific integrated circuit, is a chip designed for one job. Hard-wiring the job removes everything a general processor needs to decode, schedule and predict. For a fixed task, the result can be many times more efficient than a CPU or GPU.

AI accelerators are the best-known ASICs today. Many are built around a systolic array, a grid of multiply-accumulate units. Data flows across the grid in a steady rhythm, and each unit passes its result to its neighbour instead of writing it back to memory. The first tensor processing unit, deployed in 2015, used a 256 × 256 array of 8-bit units, which is 65,536 multipliers working at once. Smaller versions called NPUs (neural processing units) now sit inside phone and laptop chips to run AI models locally.

Accelerators are usually tuned for one of two workloads. Training builds a model: it needs high-precision accumulation, huge memory and fast links between thousands of chips. Inference runs the finished model: it tolerates 8-bit or even 4-bit numbers and cares most about cost and energy per answer.

The ASIC bargain

The price of efficiency is non-recurring engineering (NRE), the one-off cost of design, verification and masks. At the leading edge this is commonly estimated in the hundreds of millions of dollars. An ASIC pays off only if it ships in volume, or if the efficiency it buys is worth that much. The same logic produced the cryptocurrency mining chip, which does a single hash function and nothing else, and the network switch chip, which forwards tens of terabits per second of packets.

An FPGA is hardware you can rewrite.

A field-programmable gate array is a chip whose logic is defined after it is made. Its fabric is a grid of lookup tables (LUTs), small memories that can act as any logic function of a few inputs, typically six. Each table sits beside a flip-flop and is linked to the others by programmable routing. Hard blocks for arithmetic (DSP slices), memory (block RAM) and high-speed serial links are scattered through the grid.

A designer writes a circuit in a hardware description language, and tools compile it into a bitstream that sets every table and switch. On most FPGAs the configuration is held in SRAM and loaded at power-up, so the same chip can become a different circuit tomorrow.

That flexibility costs area, speed and power. A well-known comparison found an FPGA needs tens of times the area of an equivalent ASIC, runs three to four times slower and uses roughly ten times the dynamic power. FPGAs win where volumes are too small to justify an ASIC, or where the logic must change in the field. Typical cases are prototyping ASICs before tape-out, telecom base stations, test equipment, aerospace, and trading systems where a few nanoseconds of latency matter.

Which chip for which job

Memory is a ladder of trade-offs.

No single memory is fast, dense, cheap and permanent at once, so systems stack several. SRAM (course 01, lesson 03) is the fastest, at six transistors per bit, and lives on the processor die. DRAM stores each bit as charge on a tiny capacitor behind one transistor. That makes it far denser than SRAM, but the charge leaks away, so every cell must be refreshed: read and rewritten, typically every 64 milliseconds. DRAM loses everything when power is cut.

DRAM comes in several packagings for different jobs. DDR modules serve PCs and servers. LPDDR trades some speed for low power in phones and laptops. GDDR feeds graphics cards. HBM (high-bandwidth memory) stacks DRAM dies vertically, joins them with thousands of through-silicon vias, and sits beside the processor in the same package. A very wide interface gives each stack over a terabyte per second in current generations. That bandwidth is why HBM supply now gates AI accelerator output.

NAND flash keeps data with the power off by trapping charge in each cell. Cells now stack vertically in 3D NAND, with layer counts in the hundreds, and each cell stores one to four bits: SLC, MLC, TLC and QLC. More bits per cell means more capacity and lower cost, but slower writes and fewer rewrite cycles before the cell wears out. NOR flash is smaller and costlier but allows fast random reads, so it often holds the boot code a chip runs directly.

Emerging memories

Several newer memories store bits as a physical state rather than as charge. MRAM uses magnetic orientation, ReRAM the resistance of a thin film, and phase-change memory whether a material is crystalline or glassy. All keep data unpowered. Embedded MRAM is already offered by major foundries as a replacement for on-chip flash. None has yet displaced DRAM or NAND in bulk.

Whole systems on one chip.

A system on chip (SoC) puts most of a computer on a single die. A phone SoC typically holds CPU cores of two or three sizes, a GPU, an NPU and an image signal processor for the camera. It also carries video encoders and decoders, a secure enclave for keys, display and memory controllers, and often the cellular modem. Keeping these blocks on one die saves the power and delay of driving signals between chips, which matters most on a battery.

1. CPUBig and little cores
2. GPUDisplay and games
3. NPUOn-device AI
4. ISPCamera pipeline
5. MediaVideo codecs
6. ModemCellular, often on die

A microcontroller (MCU) is the small end of the same idea. It is a modest processor, usually 32-bit and often still 8-bit, with its program flash, SRAM, timers and analog inputs all on one cheap chip. Memory runs from a few kilobytes to a few megabytes. MCUs run washing machines, car windows, keyboards and thermostats, and tens of billions ship every year. They outnumber every other kind of processor.

At the high end the trend now runs the other way. Large processors are split into chiplets, several smaller dies made separately and joined in one package, because one huge die yields poorly and hits the reticle limit. Each chiplet can also use the process that suits it best. Course 03 covers how they are connected.

The chips between computing and the world.

Digital chips deal in ones and zeros. Everything outside them, such as sound, light, radio, motion and power, is continuous. A large family of chips sits at that boundary, and most devices contain more of them than processors.

Old nodes, special processes

Most of these chips do not need the newest node, and often work better on an older one. Analog circuits want high voltages, low noise and precisely matched components rather than tiny transistors. They are made on mature processes, 28 nm and larger, many tuned for one purpose. Examples are BCD processes that combine logic with high-voltage power transistors, and silicon-on-insulator for RF. By count, most chips made each year come off these mature lines. For the highest voltages and frequencies, silicon gives way to gallium nitride and silicon carbide, which are covered in course 05.

Typical nodes

28 nm to 180 nm and larger

Wafer sizes

200 mm and 300 mm, some 150 mm

Design priority

Precision, voltage, noise, cost

Product lifetime

Often a decade or more