Loading Gawtham's portfolio

why AI accelerators use systolic arrays

November 2026

why does AI need matrix multiplication in the first place?

Basic AI knowledge is assumed here, but I still want to start from a simple place. In a feedforward neural network, a neuron does something like:

y = f(w_1 x_1 + w_2 x_2 + w_3 x_3 + b)

The neuron takes a bunch of inputs, multiplies each by a weight, sums them up, adds a bias, and then applies a nonlinearity like ReLU or sigmoid. That part is easy. But once you get to bigger networks with lots of neurons, the math explodes. Instead of writing out giant sets of scalar equations by hand, we package everything into matrices.

If the previous layer has n neurons and the next layer has m neurons, then the weight matrix is W \in \mathbb{R}^{m \times n}. Each entry W[i, j] tells you how much neuron j from the previous layer influences neuron i in the next one. If the input vector is x \in \mathbb{R}^{n}, then the output is basically:

y = Wx + b

This is why matrix multiplication is such a big deal in AI. Neural networks are basically a stack of linear algebra operations, and the bigger the model, the more important it becomes to compute them efficiently.

training vs inference

There are two big phases in machine learning: training and inference.

During training, the network is learning. We feed in inputs, compare the output to the correct answer, compute how wrong we are, and then use backpropagation to figure out how much each weight contributed to that error. We adjust the weights, repeat, and slowly improve the model. This is like studying for a test.

Inference is what happens after training. You give the model new data and it computes an output without changing its weights. This is like writing the test. This is the part that matters when someone asks ChatGPT a question. The model has already been trained; now it is just doing the forward pass as fast as possible.

And one of the biggest bottlenecks in that forward pass is matrix multiplication. A lot of AI work is basically doing a ton of multiply-accumulate operations really fast.

the standard way: nested loops and MACs

The classic way to compute matrix multiplication is:

C[i][j] = \sum_{k=0}^{N-1} A[i][k] \cdot B[k][j]

In code it looks like this:

for (int i = 0; i < N; i++) {
    for (int j = 0; j < N; j++) {
        C[i][j] = 0;

        for (int k = 0; k < N; k++) {
            C[i][j] += A[i][k] * B[k][j];
        }
    }
}

For every output element, you do a bunch of multiply-accumulate operations. A MAC unit is basically a small hardware block that does:

acc += a \cdot b

A basic MAC unit has a multiplier, an adder, and an accumulator. With decent pipelining and careful design, a single MAC can often complete one multiply-accumulate per clock cycle.

For two N \times N matrices, there are N^2 output elements, and each one needsN multiply-accumulate operations. So the total work is roughly O(N^3). That is the core reason matrix multiplication matters so much in AI.

so why not just build a massive number of MAC units?

The obvious idea is: if one MAC unit does one multiply-accumulate per cycle, then just use a ton of them in parallel. That sounds great in theory. If you had a MAC for every output element, a lot of work could happen at the same time.

But this is where hardware reality hits. A fully parallel design is not magic.

First: memory bandwidth. Suppose you have 1000 MAC units, and each MAC needs two inputs. If each input is 8 bits, then in one clock cycle you need:

1000 \times 2 \times 8 = 16000\text{ bits/cycle}

At 1 GHz, that is:

16000 \times 10^9 \approx 2 \times 10^{13}\text{ bits/sec}

That is roughly 2 TB/s of data movement. Feeding that much data into a chip is a huge challenge. The memory system and interconnect can’t keep up, so a ton of MAC units sit idle waiting for data.

Second: data reuse. In matrix multiplication, the same numbers get reused a ton. A naive design keeps fetching the same weight or input values from memory over and over again. That wastes bandwidth and burns power. We want to load a value once and reuse it many times while it is still nearby.

Third: hardware complexity. A giant array of MAC units is hard to design, route, power, and cool. Communication between all those units creates a lot of wiring complexity and power consumption.

Key idea

The challenge is not just “how many MACs can we fit,” but “how do we keep the MACs fed with the data they need without constantly going back to memory?”

enter the systolic array

This is the motivation behind a systolic array. The word “systolic” comes from the human heart. The heart rhythmically contracts and pushes blood through the body. In a systolic array, data is pushed rhythmically through a grid of processing elements in lockstep. Instead of each processing unit grabbing data from memory on demand, the data simply flows through the array like a pulse.

The big idea is simple: keep the arithmetic local and let data flow from one processing element to the next. The same value gets reused across neighboring units instead of being repeatedly fetched from a distant memory system. That reduces bandwidth pressure and improves locality.

A systolic array is basically a structured grid where each processing element does a tiny amount of work and then passes data onward. It is like an assembly line, except the product is a partial sum and the workers are tiny arithmetic units.

worked example: one clock at a time

Let's multiply two small matrices and follow the numbers through the array. Instead of jumping straight to the answer, we'll watch how each clock brings us one step closer.

A = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}
B = \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix}

first, the rules of the array

The systolic array would be set up like this. Each processing element (PE) calculates one entry of the output matrix and starts with an accumulator of zero. Values from A move from left to right. Values from B move from top to bottom. A PE multiplies its two inputs, adds the product to its accumulator, and passes A to its right neighbor and B to its lower neighbor on the next clock. The accumulator stays put.

Before the first clockThe array is ready
A → rightB ↓ downΣ stays in its PE
Cycle 0. Each PE starts at zero and owns one entry of C. The second row of A and second column of B enter one clock later so matching operands meet. Accumulators at the end: PE 1,1: 0; PE 1,2: 0; PE 2,1: 0; PE 2,2: 0.PE11waitingΣ 0no matching pair yetPE12waitingΣ 0no matching pair yetPE21waitingΣ 0no matching pair yetPE22waitingΣ 0no matching pair yet
Each PE starts at zero and owns one entry of C. The second row of A and second column of B enter one clock later so matching operands meet.

There's one timing detail to get right: the second row of A and second column of B enter one clock later than the first. This stagger makes sure the right pairs meet. We'll count a cycle as the inputs arriving and the MAC finishing; those inputs reach the neighboring PEs in the following cycle. The animations slow that down into move, multiply, and accumulate so we can see each part.

clock cycle 1: start with one pair

We feed 1 from A and 5 from B into the top-left PE. It performs the first MAC: 0 + 1 × 5 = 5. Nobody else has a matching pair yet, so the other three accumulators stay at zero.

Clock cycle 1 / 4The first pair arrives
A → rightB ↓ downΣ stays in its PE
Cycle 1. Only PE₁₁ has a matching pair. The other PEs wait; their accumulators stay at zero. Accumulators at the end: PE 1,1: 5; PE 1,2: 0; PE 2,1: 0; PE 2,2: 0.PE11waitingΣ 0inputs on their way15PE12waitingΣ 0no matching pair yetPE21waitingΣ 0no matching pair yetPE22waitingΣ 0no matching pair yet
  1. 1. Move
  2. 2. Multiply
  3. 3. Accumulate
Before clock 1
  • PE110 + 1 × 5 = 5
Only PE₁₁ has a matching pair. The other PEs wait; their accumulators stay at zero.

Keep an eye on that 1 and that 5. We're about to use both of them again, without loading them into the array a second time.

clock cycle 2: pass the inputs to the neighbors

The 1 moves right into PE₁₂, where it meets a fresh 6. The 5 moves down into PE₂₁, where it meets a fresh 3. At the same time, a new pair, 2 and 7, enters PE₁₁. Three PEs are now doing useful work in parallel.

Clock cycle 2 / 4Pass it on, add the next product
A → rightB ↓ downΣ stays in its PE
Cycle 2. The 1 moves right and the 5 moves down. Three PEs work at once, and PE₁₁ completes C₁₁ = 19. Accumulators at the end: PE 1,1: 19; PE 1,2: 6; PE 2,1: 15; PE 2,2: 0.PE11waitingΣ 5inputs on their way27PE12waitingΣ 0inputs on their way16PE21waitingΣ 0inputs on their way35PE22waitingΣ 0no matching pair yet
  1. 1. Move
  2. 2. Multiply
  3. 3. Accumulate
Before clock 2
  • PE115 + 2 × 7 = 19
  • PE120 + 1 × 6 = 6
  • PE210 + 3 × 5 = 15
The 1 moves right and the 5 moves down. Three PEs work at once, and PE₁₁ completes C₁₁ = 19.

Notice what happened to the top-left accumulator. We didn't replace the previous result of 5. We added the new product to it: 5 + 2 × 7 = 19. That's both terms of the first dot product, so PE₁₁ has already finished its answer. The neighboring PEs have only calculated their first terms: 6 and 15.

clock cycle 3: the wave reaches PE₂₂

The top-left PE is done now; its 19 stays in place. Its previous inputs keep moving: 2 goes right into PE₁₂ and 7 goes down into PE₂₁. We also feed the final new values, 4 and 8, into the edges of the array. PE₁₂ finishes with 6 + 2 × 8 = 22, and PE₂₁ finishes with 15 + 4 × 7 = 43.

Clock cycle 3 / 4The wave reaches the last PE
A → rightB ↓ downΣ stays in its PE
Cycle 3. PE₁₁ holds 19. PE₁₂ and PE₂₁ finish, while PE₂₂ receives 3 and 6 from its neighbors. Accumulators at the end: PE 1,1: 19; PE 1,2: 22; PE 2,1: 43; PE 2,2: 18.PE11done ✓Σ 19C11 = 19PE12waitingΣ 6inputs on their way28PE21waitingΣ 15inputs on their way47PE22waitingΣ 0inputs on their way36
  1. 1. Move
  2. 2. Multiply
  3. 3. Accumulate
Before clock 3
  • PE126 + 2 × 8 = 22
  • PE2115 + 4 × 7 = 43
  • PE220 + 3 × 6 = 18
PE₁₁ holds 19. PE₁₂ and PE₂₁ finish, while PE₂₂ receives 3 and 6 from its neighbors.

Meanwhile, PE₂₂ finally gets its first pair. The 3 comes from PE₂₁ on its left, and the 6 comes from PE₁₂ above it. These are the inputs those neighbors used in the previous cycle. PE₂₂ multiplies them and stores 18. So three output entries are complete, and the last one is halfway there.

clock cycle 4: finish the last sum

We don't need to feed in anything new. The final inputs are already inside the array: 4 moves right from PE₂₁ and 8 moves down from PE₁₂. They meet at PE₂₂, which adds their product to its existing 18. That's 18 + 4 × 8 = 50. The entire matrix multiplication is now done.

Clock cycle 4 / 4The final sum lands
A → rightB ↓ downΣ stays in its PE
Cycle 4. No fresh inputs enter. The 4 and 8 make one last hop into PE₂₂, completing C₂₂ = 50. Accumulators at the end: PE 1,1: 19; PE 1,2: 22; PE 2,1: 43; PE 2,2: 50.PE11done ✓Σ 19C11 = 19PE12done ✓Σ 22C12 = 22PE21done ✓Σ 43C21 = 43PE22waitingΣ 18inputs on their way48
  1. 1. Move
  2. 2. Multiply
  3. 3. Accumulate
Before clock 4
  • PE2218 + 4 × 8 = 50
No fresh inputs enter. The 4 and 8 make one last hop into PE₂₂, completing C₂₂ = 50.
C = AB = \begin{bmatrix} 19 & 22 \\ 43 & 50 \end{bmatrix}

Read the four accumulators in their grid positions and you get the output matrix. Each PE held onto its partial sum while the inputs moved rhythmically between neighbors every clock. That's the systolic part.

follow the 5

In cycle 1, the 5 entered PE₁₁. In cycle 2, that same 5 reached PE₂₁. We loaded it into the array once and used it twice. A naive implementation might fetch it again for the second output; here, a short hop to the next PE handles the reuse. Other architectures can reuse data through caches or registers too, but this array makes that reuse a regular part of the data path. On a larger array, each value can serve more PEs as it travels.

why this matters

AI accelerators are not trying to be fancy for fun. They are trying to solve the real engineering problem of doing lots of matrix multiplication quickly, efficiently, and with manageable memory traffic.

A plain array of MAC units looks great on paper, but it runs into memory bandwidth and power problems. A systolic array gives you the parallelism you want without constantly re-reading the same data. It is a much more practical way to do matrix multiplication at scale.

That is why AI accelerators use systolic arrays. They turn one of the hardest parts of AI compute into a flowing, local, reusable data pipeline instead of a memory-starved pile of random arithmetic.