What is benchmarking
Benchmarking
Benchmarking is the science of objectively assessing the performance of a system in comparison to other systems. This means that:
Link to original
- Users are able to know which system is best suited to your needs.
- Engineers are able to make informed design decisions.
There are certain metrics we can attempt to look at when bench-marking but each usually has some flaws:
- Professor frequency: isn’t necessarily a good comparison point, between difference manufacturers different amounts of computation can be done per clock cycle.
- Millions (of) Instructions Per Second (MIPS): but this doesn’t factor in the complexity of instructions being executed, some may do more computation that others and some architectures allow for complex instructions.
- Floating Point Operations Per Seconds (FLOPS) is another measure that attempts to capture the performance of a computer system. However, there is no consensus on what a floating-point operation is.
Challenges with benchmarking
We need to decide what the benchmark will be, whether it is just a script, a video game or a synthetic benchmark. We need to decide what to measure, whether it’s execution time, frames per second, memory use or other factors described above. And we need to look at other considerations such as financial cost, running expenses and size.