1. What is benchmarking

What is benchmarking

Benchmarking

Benchmarking is the science of objectively assessing the performance of a system in comparison to other systems. This means that:

• Users are able to know which system is best suited to your needs.
• Engineers are able to make informed design decisions.

Link to original

There are certain metrics we can attempt to look at when bench-marking but each usually has some flaws:

• Professor frequency: isn’t necessarily a good comparison point, between difference manufacturers different amounts of computation can be done per clock cycle.
• Millions (of) Instructions Per Second (MIPS): but this doesn’t factor in the complexity of instructions being executed, some may do more computation that others and some architectures allow for complex instructions.
• Floating Point Operations Per Seconds (FLOPS) is another measure that attempts to capture the performance of a computer system. However, there is no consensus on what a floating-point operation is.

Challenges with benchmarking

We need to decide what the benchmark will be, whether it is just a script, a video game or a synthetic benchmark. We need to decide what to measure, whether it’s execution time, frames per second, memory use or other factors described above. And we need to look at other considerations such as financial cost, running expenses and size.

Link to original

2. Benchmark examples

Benchmark examples

Simple PC benchmarks:

• Windows / Mac OS: NovaBench
• Linux: sysbench

Notable synthetic benchmarks

• Prime identification is one example of a synthetic benchmark.
• Famous historical examples include:
  • Whetstone: Trigonometric and exponential functions
  • Linpack: Linear equations and double precision arithmetic
  • Dhrystone: String manipulations and integer operations

Simple synthetic benchmarks are easy to optimise so they do not give an honest measure of a system’s performance at general computing.

Benchmark suites

Benchmark suites are preferred to single synthetic benchmarks as:

• They are harder to optimise for
• They are potentially more representative of general computing patterns
• Benchmarks can be easily swapped in and out

Standard Performance Evaluation Corporation (SPEC) has one of the most widely used benchmark suites. It contains several programs for a wide range of integer and floating-point operations, arranged into two sets.

• CINT2017 is a set of programs for integer operations
• CFP2017 is a set of programs for floating point operations

This suite is regularly updated to keep up with general modern computing. https://spec.org/cpu2017/Docs/overview.html#benchmarks

The final result is the geometric mean of the run times for each program in the suite. Times from each program are different scales, so the geometric mean is more appropriate.

Link to original

3. RISC vs CISC

RISC vs CISC

$$\text{execution cycles}=\text{clock cycles per instruction }\times \text{number of instructions}$$

There are two main kinds of instruction sets in von Neumann systems:

• Reduced Instruction Set Computer

  Reduced Instruction Set Computers (RISC) shortens execution time by reducing the clock cycles per instruction.

  • For RISC, we have a few, simple instructions, fixed in length. This means: ? $+$ Control units can be hardwired for maximum speed. $+$ Instructions take $1$ cycle to process meaning easier pipelining. $-$ Programming requires more instructions for complex operations.

  • For RISC, we only have Load / Store instructions for memory access, with few addressing modes. This means that: ? $+$ Parameter passing through registers means fast sequential execution. $-$ Requires more (expensive) registers.

  Link to original

• Complex Instruction Set Computers

  Complex Instruction Set Computers (CISC) improve performance by reducing the number of instructions per program.

  • For CISC, many complex instructions with variable length means: ? $-$ Control units need to use special circuits for interpreting instructions as they are fetched from memory, this can take additional time. $-$ Instructions may take multiple / variable numbers of cycles to process hence harder to manage pipelines. $+$ Easier to program, fewer instructions required for complex operations.

  • For CISC, many instructions can access memory using many addressing modes which means: ? $-$ Parameter passing through memory means additional strain on von Neumann bottleneck. $+$ Fewer registers needed, fewer operands per instruction.

  Link to original

Optimal Solution

The first digital computers were RISC due to their simple programs. As programs became more complex, CISC became dominant. RISC processors are generally used in embedded devices.

Modern CISC processors breakdown complex instructions into a series of simpler ones to be executed. Sometimes this allows them to be pipelined more easily but it requires more space on the chip, power and produces additional heat.

Link to original

4. Branch optimisations and code optimisations

Branch Optimisations

Pipelines offer significant speed up if the pipeline can be kept full.

instr A
instr B
if COND
	instr A
	instr B
else
	instr A
	instr B
instr G
instr H

• Delayed branching

  Delayed branching: fetch instructions after the conditional. When executing instr B, we could start fetching instr G since it doesn’t depend on the outcome of the conditional statement.

  Link to original

• Branch prediction

  Branch prediction: ==speculatively fetch the next instruction from one of the branches. We could fetch either instr C or instr E and hope we fetch the correct one. If we guess wrong, data has to be removed from registers==.

  To pick the correct branch (in branch prediction), we can choose a few options: ?

  • We could pick randomly, yielding 50% success rate.
  • Could use static prediction, hence analyse the code at compile-time, yielding 80% success rate on average. (from SPEC analysis)
  • Use dynamic prediction: during run-time keep track of how often each branch is taken and when the conditional is encountered in the future, fetch the most common branch. This does require some training ahead of time.

  Link to original

Code Optimisations

Sometimes performing code optimisations at the software level can yield better results. Some common tips include:

• Use appropriate data types: don’t use say double when int is fine
• Eliminate unnecessary branches: minimise the no. of branches by combining conditions and removing those that will never execute
• Use multiplication instead of division
• Profile programs: identify parts of the program using most CPU time

Compiler Optimisations

In modern high-level languages, the compiler does most of the optimisations for us. Sometimes attempted optimisations may be detrimental, often it’s better to just give the compiler as much information as possible rather than trying to optimising it by hand.

Link to original

5. Caching

Caching

Speeding up the CPU is only going to improve performance if the system is CPU-bound. Performance can also be bound by memory.

In von Neumann architectures, the CPU needs to fetch data and instructions from memory. The main memory (MM) may not provide a fast enough access time.

digraph {
	rankdir=LR
	node[shape=box]
	MM[label="MM (10ns)"]
	CPU->MM [dir=both]
}

Cache

We can introduce a cache which is a fast access memory which sits between the CPU and main memory, but it has a smaller capacity. Cache is also **volatile memory**, so in case of an outage, data in the cache is lost.

digraph {
	rankdir=LR
	node[shape=box]
	MM[label="MM (10ns)"]
	CACHE[label="Cache (1ns)"]
	CPU->CACHE [dir=both]
	CACHE->MM [dir=both]
}

Link to original

Pre-fetching

Pre-fetching is where we fetch data before it is needed based on usage patterns but is prone to *cache pollution* if we get it wrong.

Link to original

Memory usage patterns usually follow a usage pattern:

• Temporal locality

  Temporal locality: if a location has been accessed recently it is likely to be accessed again (e.g. top of a stack)

  Link to original

• Spacial locality

  Spatial locality: if a location has been accessed recently, it is likely that nearby locations will be accessed in the near future (e.g. loops or arrays)

  Link to original

• Sequential locality

  Sequential locality: if an address has been accessed recently the next / prev locations are likely to be accessed next (e.g. instructions)

  Link to original

If data in the cache is changed, we need to figure out when data should be written back to main memory, we use write-through and write-back policies.

• Write-through Cache

  Write-through cache: whenever data in the cache is changed, simultaneously write it to main memory. Improves reliability, but reduces performance.

  Link to original

• Write-back Cache

  Write-back cache: wait until an efficient point in time to wrote changed cache data to main memory. When a read operation occurs for main memory, the write operation can be done simultaneously. Improves performance for write operations, reduces performance for read operations and reduces reliability.

  Link to original

Link to original