5. Caching

Caching

Speeding up the CPU is only going to improve performance if the system is CPU-bound. Performance can also be bound by memory.

In von Neumann architectures, the CPU needs to fetch data and instructions from memory. The main memory (MM) may not provide a fast enough access time.

digraph {
	rankdir=LR
	node[shape=box]
	MM[label="MM (10ns)"]
	CPU->MM [dir=both]
}

Cache

We can introduce a cache which is a fast access memory which sits between the CPU and main memory, but it has a smaller capacity. Cache is also **volatile memory**, so in case of an outage, data in the cache is lost.

digraph {
	rankdir=LR
	node[shape=box]
	MM[label="MM (10ns)"]
	CACHE[label="Cache (1ns)"]
	CPU->CACHE [dir=both]
	CACHE->MM [dir=both]
}

Link to original

Pre-fetching

Pre-fetching is where we fetch data before it is needed based on usage patterns but is prone to *cache pollution* if we get it wrong.

Link to original

Memory usage patterns usually follow a usage pattern:

• Temporal locality

  Temporal locality: if a location has been accessed recently it is likely to be accessed again (e.g. top of a stack)

  Link to original

• Spacial locality

  Spatial locality: if a location has been accessed recently, it is likely that nearby locations will be accessed in the near future (e.g. loops or arrays)

  Link to original

• Sequential locality

  Sequential locality: if an address has been accessed recently the next / prev locations are likely to be accessed next (e.g. instructions)

  Link to original

If data in the cache is changed, we need to figure out when data should be written back to main memory, we use write-through and write-back policies.

• Write-through Cache

  Write-through cache: whenever data in the cache is changed, simultaneously write it to main memory. Improves reliability, but reduces performance.

  Link to original

• Write-back Cache

  Write-back cache: wait until an efficient point in time to wrote changed cache data to main memory. When a read operation occurs for main memory, the write operation can be done simultaneously. Improves performance for write operations, reduces performance for read operations and reduces reliability.

  Link to original