How CPUs Work: Instruction Cycle and Pipelines
The Central Processing Unit (CPU) is often called the "brain" of a computer.
This article explains how CPUs execute instructions through the instruction cycle and how pipelining boosts performance — knowledge that’s critical for system design interviews and real-world performance engineering.
1. What is a CPU?
A CPU executes program instructions by performing three basic functions:
- Fetching instructions from memory.
- Decoding them into signals the hardware understands.
- Executing them using internal components (like the ALU).
It works in constant coordination with:
- Memory (RAM, cache) → where instructions/data live.
- Registers → fast, small storage inside the CPU.
- I/O devices → external input and output.
2. The Instruction Cycle
The instruction cycle (or fetch–decode–execute cycle) is the repeating process CPUs follow to run programs.
Steps in the Cycle
Fetch
- CPU retrieves the next instruction from memory.
- The Program Counter (PC) holds the memory address.
- Instruction is placed into the Instruction Register (IR).
- PC increments to the next instruction.
Decode
- The Control Unit (CU) interprets the instruction.
- CU translates it into micro-operations (signals to ALU, registers, memory).
Execute
- The Arithmetic Logic Unit (ALU) or other CPU components perform the operation.
- Examples: arithmetic, logic, data transfer, branching.
Store (optional)
- Results may be written back to registers or main memory.
Key Idea: The cycle is simple, but repeated billions of times per second in modern CPUs.
3. CPU Pipelining
Without pipelining, the CPU finishes one instruction completely before starting the next.
Pipelining improves performance by overlapping stages of multiple instructions, like an assembly line.
3.1 How Pipelining Works
- While one instruction executes, the next is decoded, and a third is fetched.
- This parallelism increases instruction throughput (instructions completed per clock cycle).
Example: 3-Stage Pipeline
| Cycle | Stage 1 (Fetch) | Stage 2 (Decode) | Stage 3 (Execute) |
|---|---|---|---|
| 1 | Inst 1 | ||
| 2 | Inst 2 | Inst 1 | |
| 3 | Inst 3 | Inst 2 | Inst 1 |
| 4 | Inst 4 | Inst 3 | Inst 2 |
In 4 cycles, 3 instructions are executed instead of 9 (without pipelining).
4. Benefits of Pipelining
- Higher throughput: More instructions per second.
- Efficient resource use: ALU, fetch unit, and decode unit rarely idle.
- Foundation for modern CPUs: Superscalar, multi-core, and out-of-order execution all build on pipelining.
5. Challenges of Pipelining
Pipelining is powerful but introduces hazards:
Data Hazards
- Instruction depends on result of an earlier one.
- Example:
ADD R1, R2, R3 ; produces result in R1 SUB R4, R1, R5 ; needs R1 immediately - SUB must wait for ADD → pipeline stall.
Control Hazards
- Arise from branches (e.g., if-else, loops).
- CPU doesn’t know next instruction until branch resolves.
Structural Hazards
- Hardware resource conflict.
- Example: two instructions need memory access in the same cycle.
Pipeline Stalls
When hazards occur, the CPU may insert no-op (NOP) instructions, pausing execution.
This reduces efficiency.
6. Solutions to Pipeline Hazards
- Data Forwarding (Bypassing) → feeds ALU output directly into next instruction without waiting for memory.
- Branch Prediction → CPU guesses the outcome of branches to keep pipeline busy.
- Speculative Execution → executes guessed instructions; discards results if guess was wrong.
- Out-of-Order Execution → reorders independent instructions to avoid stalls.
Interview Tip: Be able to explain how branch prediction and out-of-order execution improve throughput but add complexity.
7. Key CPU Components
- Program Counter (PC) → holds address of next instruction.
- Instruction Register (IR) → holds current instruction.
- Control Unit (CU) → directs flow of data.
- Arithmetic Logic Unit (ALU) → performs operations.
- Registers → fastest storage, used for temporary results.
- Cache → small, fast memory to reduce fetch times.
8. Real-World Context
Interviews:
- Expect to be asked about pipeline hazards and branch prediction.
- Be able to explain why pipelining improves throughput but not latency of a single instruction.
Practical Engineering:
- Software optimizations (loop unrolling, instruction scheduling) often aim to reduce stalls.
- Performance tuning requires knowing how CPUs fetch/decode/execute.
Modern CPUs:
- Superscalar architectures: multiple instructions executed in parallel pipelines.
- Speculative execution: executes instructions ahead of time to hide latency.
- Hyper-threading/SMT: allows multiple threads to share pipeline stages.
9. Further Reading
- Computer Organization and Design — Patterson & Hennessy
- Structured Computer Organization — Andrew S. Tanenbaum
- Intel and AMD CPU architecture whitepapers for modern pipeline details.