next »

[Photon]

Having not done a whole lot of this, wanted to ask how do y'all go about writing benchmark tests. Any particular techniques you employ? What should be avoided to prevent significant result skew?

[lithander]

Writing benchmark tests? The way I usually benchmark is by using a profiler - either the one in Unity or the one shipping with Visual Studio. Even if you work in an environment with no profiler support there are a lot of stand alone tools. Even GPU specific ones to profile the rendering or GPGPU stuff. https://en.wikipedia.org/wiki/List_of_performance_analysis_tools

These tools help you identify your bottlenecks and then you just figure out how to optimize the code causing those bottle necks.

[Cheesegrater]

If your profiler functions by inserting breakpoints (most do) it won't be a true benchmark because the breakpoints/profiling code itself will affect the runtime.

My best tip for benchmark tests is to make sure your program is in a steady state before you start the timers. Even after your code is done loading things off disk, the OS and/or the graphics driver may still be moving things in and out of cache or graphics memory, so you want to work around that.

Except for when you're trying to benchmark startup times, of course - but my experience is when you do that you're mostly just benchmarking the hard disk.

If you have a large budget, it might be worthwhile to set aside special hardware running a stripped down OS for benchmarking. Bigger companies sometimes have a couple of hardware configurations set aside to run nightly benchmarks to spot slow-creeping performance losses. Windows embedded and/or Linux from scratch are good for that.

[BorisTheBrave]

You want to write a standalone program that does the same thing every time and prints the benchmark results, similar to writing unit tests.

Here are some classic pitfalls, and how to avoid them

• using a poor resolution timer: google for the OS specific timers to use for benchmarking, they are often different from the general purpose timers
• micro benchmarking: ensure you are testing a workload that resembles real data, or else you will optimize the wrong thing
• too little work: the larger a piece of work is, the more (relative) accuracy you get on your timers. If your test is too quick, insert a loop to do it lots of time (but avoid micro benchmarks, above)
• sampling: each time you run a benchmark, the results will randomly vary. You should run the same benchmark several times and look at the median to get more consistent results. Looking at the 95th percentile can also be useful, depending on what your performance goals are.
• averaging: if you have several benchmark tests covering different aspects of your program, use the geometric mean to combine then into an overall number. This avoids problems where the slowest test becomes the dominant factor in the overall measure.
• hot vs cold: there are many, many caches in a computer, and benchmarks will run faster when they are hot. You should always run a benchmark a couple of times, and discard the results before recording anything, to ensure a consistent hot result. I have yet to figure out how to consistently benchmark cold results, even though it's often the more interesting number.