Compute-in-Memory vs Von Neumann: Moving the Multiply to the Data

2026-11-20来源 dev.to AIEnglish

Every AI accelerator is, underneath the marketing, a machine for moving numbers. The multiply-accumulate operations that make a transformer work are cheap; getting the weights to the multiplier is not. In a conventional chip, a weight travels from DRAM through several cache levels into a register, gets multiplied, and the partial sum travels back. Compute-in-memory (CIM) asks a blunt question: if the weight already sits in the memory array, why move it at all?

A conventional accelerator can burn sixty to eighty percent of its energy on data movement, while a compute-in-memory array performs the multiply inside the memory cell itself — the savings come from deleting the bus, not from speeding up arithmetic.

Part One: What is being compared

The Von Neumann architecture separates storage from computation. A processor fetches instructions and operands from memory, computes, and writes results back. This separation is the reason a general-purpose computer can run any program, and it is also the reason a neural-network inference pass is memory-bound: for each weight you do one multiply, but that weight crosses a wire that is orders of magnitude more expensive than the multiply itself.

Compute-in-memory collapses the two. In its dominant analog form, a crossbar of non-volatile memory cells — resistive RAM, flash, or phase-change — stores the weight as a conductance. Inputs are applied as voltages along the rows. Kirchhoff's law sums the currents at each column, and that sum is the dot product, computed by physics in a single step. No fetch, no register file, no separate ALU for that operation.

The comparison is not "old versus new." It is a trade between flexibility and energy: a Von Neumann chip can compute anything but pays for every operand's journey; a CIM array does one operation beautifully and is locked to it.

Part Two: The arithmetic

Take a mainstream 7-nanometre accelerator running int8 inference. A DRAM access costs roughly 50 to 100 picojoules per byte, while a multiply-accumulate costs about 0.1 picojoules — a ratio of hundreds to one that is the entire case for moving the multiply into memory. That ratio — hundreds to one — is the entire case for CIM. A 7-billion-parameter model at int8 is 7 gigabytes of weights. Reading those weights once from DRAM costs on the order of 0.5 joules, while the arithmetic for one token is a few millijoules. The machine is a delivery service with a tiny factory attached.

An analog CIM crossbar stores those weights in the array and applies inputs as voltage rows. The multiply and sum happen in the current flowing through the conductors, at maybe 1 to 10 femtojoules per operation — one to two orders of magnitude below the digital path. Published prototypes report energy efficiencies in the tens of tera-operations per watt for the in-array multiply, against single-digit tera-operations per watt for the best digital accelerators on the same process.

The catch appears in the peripheries. Digital-to-analog and analog-to-digital converters sit at the edges of the crossbar, and they are power-hungry and area-hungry. A 128-by-128 array needs converters precise enough to preserve the signal; pushing their resolution up erases the energy win. The array also computes only matrix-vector products, so nonlinearities, normalization and attention softmax still run on conventional logic. Real chips end up hybrid, and the measured system-level gain is usually two to five times, not the hundred times the cell physics suggests.

Part Three: Where the model breaks

Analog computation is noisy in a way digital computation is not. Cell conductance drifts with temperature, with the number of program-erase cycles, and with the accumulated history of inputs. A weight written today is not exactly the weight read next month. On a 4-bit crossbar, device variation can push effective precision below what a quantized model tolerates, and accuracy drops unless the training procedure is taught to expect the noise.

That is the deep asymmetry. A digital multiplier is deterministic: the same inputs always produce the same output, which is why software can be debugged. A CIM array is probabilistic hardware running a deterministic algorithm, so the system must either calibrate constantly — reading weights back and correcting — or train with noise injected so the network is robust to it. Both cost the very energy CIM was meant to save.

There is also a programming-model tax. Digital accelerators run conventional frameworks with minor changes. CIM requires weights to be mapped onto physical cells, with aware placement to balance array utilization and tolerate defective cells. Toolchains exist but are immature, and a model that performs well in PyTorch may need substantial retraining and remapping before it runs correctly on the array.

Part Four: The later cost

The lifecycle differs in kind. A digital accelerator ages gracefully: a transistor slows slightly, timing margins absorb it, and the chip works. An analog CIM array ages as its memory cells age — resistive drift accumulates, and the accuracy curve degrades gradually over months rather than failing at a point. For a consumer device with a two-year life that may be invisible; for a datacentre accelerator expected to serve a decade, it is a maintenance problem nobody has fully priced.

The failure modes also differ. Digital chips fail hard and are replaced. Analog arrays can fail soft — output accuracy quietly slips below spec while the chip still passes basic function checks — which is far more dangerous in a production inference pipeline. Detecting that requires on-chip reference circuits and periodic calibration, which again returns energy to the system.

Manufacturing is the other tax. Resistive RAM and phase-change memory are not standard CMOS steps; adding them to a logic process raises cost and lowers yield, at least until the technology matures. Flash-based CIM avoids a novel process by reusing an established one, but flash is slow to program and wears out, which suits inference better than training.

Part Five: Outside the team

For the person using the product, the difference is battery and latency. A phone running an on-device assistant with a CIM accelerator can keep the model local instead of sending each query to a datacentre — better privacy, better offline behaviour, less network cost. The visible win is that the feature works on a plane and does not drain the battery by lunchtime.

For the engineer, the choice decides what can ship. A digital NPU is boring to deploy and flexible when the model changes next quarter. A CIM block is efficient but opinionated: it rewards stable models, low precision and high batch reuse, and punishes rapid iteration. Teams whose model architecture changes monthly often find the flexibility of digital more valuable than the energy saving.

For the environment, the arithmetic matters at scale. If inference grows to a significant share of global electricity use, a two-to-five-times efficiency gain on the dominant workload is material. But the gain must be measured at the system level, including converters, calibration and the digital logic that CIM cannot replace — not quoted from the crossbar alone.

A fair comparison fixes the model, the precision target, the batch size and the acceptable accuracy loss, then measures joules per token at the wall socket. Quoting cell-level femtojoules against a system-level number is the most common way this comparison gets abused.

Market sizing check

If edge inference grows with the installed base of phones, cars and industrial sensors, and CIM captures even a quarter of the accelerator silicon in battery-powered devices, the addressable market for in-memory compute IP and chips is plausibly $8B to $20B a year by 2030. The swing factor is whether model architectures stabilise enough to justify hard-wiring them, and whether analog noise can be tamed without eating the energy budget.

These figures are a synthesis of published market ranges and a stated model, not a forecast of any specific vendor.

The comparison that matters

The comparison that matters is between moving data and computing in place, priced at the wall socket. Von Neumann wins when the model changes often, precision requirements are high, and flexibility is worth more than efficiency. Compute-in-memory wins when the model is fixed, the precision target is modest, and energy per token is the binding constraint.

The ranking assumes the workload is dominated by dense matrix multiplies. Shift toward attention-heavy or irregular computation and the fraction of work CIM can accelerate shrinks, taking its advantage with it.

Compute-in-memory does not make arithmetic cheaper — it makes the data free by removing the journey, and that only pays when the arithmetic is regular enough to do inside the array.

SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS

原文:https://www.a8q9.cn/en/article/compute-in-memory-vs-von-neumann-moving-the-multiply-to-the-data-6871f9.html

相关文章