Tech

Computer architecture and GPUs

Explain where the nanoseconds go: CPU (central processing unit) time and pipelines, caches and memory layout, Amdahl's law and the traps of threads, power limits, and why GPUs (graphics processing units) win some workloads and starve on others.

6 concepts7 min read25-question test, from a bank of 33Updated

Concept 1 of 61 min

CPU time, pipelines, branches and out-of-order execution

The rule
CPU time = instructions × cycles per instruction ÷ frequency, and mispredictions add branch share × mispredict rate × penalty to the CPI.
Worked examples

CPU time

Given
Instructions6 billion
CPI0.8
Clock4 billion cycles/s
  1. Cycles = 6 × 0.8 = 4.8 billion
  2. Time = 4.8 ÷ 4 = 1.2 seconds

Fewer cycles per instruction or a faster clock both cut the time.

Branch mispredictions

Given
Branches25%
Mispredicted4%
Penalty20 cycles
Base CPI1.2
  1. Added = 0.25 × 0.04 × 20 = 0.2
  2. CPI = 1.2 + 0.2 = 1.4

Rare mispredictions still add 0.2 cycles to every instruction.

A pipeline

Given
Stages10
Instructions1,000
  1. Pipelined = 1,000 + 10 − 1 = 1,009 cycles
  2. Without overlap = 1,000 × 10 = 10,000

Overlapping the stages makes the pipeline almost 10 times faster.

The picture
Formula
Time=instructions × CPI ÷ frequency
effective CPI=base CPI+branch share × mispredict rate × penalty
k-stage pipeline: about n + k − 1 cycles for n instructions
Why it holds

A pipeline overlaps stages of different instructions, so throughput nears 1 per cycle while each still takes every stage. Out-of-order execution fills stalls with independent work.

A pipeline raises throughput, not a single instruction's latency. Forgetting that only a share of instructions are branches overstates the penalty.

A test such as 'value > threshold' on sorted data is false then true in long runs that the predictor learns; on shuffled data it mispredicts about half the time.

While one instruction waits on memory, later independent ones run, and in-order retirement makes the program see the same outcome.

Watch out
  • Thinking a pipeline shortens each instruction.
  • Applying the branch penalty to every instruction.
  • Believing out-of-order execution changes a program's results.
In the interview
  1. A pricing loop runs twice as fast on sorted trades as on shuffled ones, with the same code. Explain it to a non-specialist.

    A complete answer covers

    • The processor guesses which way each 'if' will go to keep its pipeline full.
    • On sorted data the answer repeats for long stretches, so it guesses right.
    • On shuffled data it guesses wrong about half the time and throws work away.
    • Branch-free code, or sorting first, removes the cost.

Also in our technical interviews, on camera, with a score.

Concept 2 of 61 min

Power, clock speed and transistor counts

The rule
Dynamic power scales with capacitance × voltage² × frequency, so lowering voltage and frequency together saves power steeply.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 3 of 61 min

Caches, memory layout and the memory hierarchy

The rule
Average access time = hit time + miss rate × miss penalty, and a working set that fits in cache runs far faster.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 4 of 61 min

Parallel speedup, false sharing, NUMA and SIMD

The rule
Amdahl's law gives a speedup of 1/(s + p ÷ n) for serial share s, parallel share p and n cores, never above 1/s.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 5 of 61 min

GPUs: arithmetic intensity, divergence and training memory

The rule
A GPU runs thousands of simple cores in parallel, and a kernel with little arithmetic per byte is memory-bound at bandwidth × intensity.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 6 of 61 min

AdvancedThe roofline: memory-bound or compute-bound

The rule
A kernel's attainable speed is the lower of the chip's peak arithmetic rate and its memory bandwidth times the kernel's arithmetic intensity.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Recap

  • CPU time = instructions × cycles per instruction ÷ frequency; mispredicted branches add penalty × their share.
  • Average access time = hit time + miss rate × miss penalty; walk memory in order and keep the working set in cache.
  • Amdahl: speedup = 1/(s + p ÷ n), capped at 1/s; watch false sharing and remote NUMA (non-uniform memory access) memory.
  • Dynamic power ∝ C × V² × f: clocks stalled on heat, so chips went multi-core, SIMD (single instruction, multiple data) and GPU.
  • A GPU kernel is memory-bound when it does little arithmetic per byte; divergence serialises a warp; batch size grows activation memory.
Now test it

25 questions on Computer architecture and GPUs

Timed, like the real one, drawn from a bank of 33. Every answer explained.

Test yourself →