Linked by Thom Holwerda on Thu 3rd Oct 2013 16:07 UTC
Benchmarks

With the exception of Apple and Motorola, literally every single OEM we've worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It's possible that older Motorola devices might've done the same thing, but none of the newer devices we have on hand exhibited the behavior. It's a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.

Pathetic, but this has been going on in the wider industry for as long as I can remember - graphics chip makers come to mind, for instance. Still, this is clearly scumbag behaviour designed to mislead consumers.

On the other hand, if you buy a phone based on silly artificial benchmark scores, you deserve to be cheated.

Permalink for comment 573823
To read all comments associated with this story, please click here.
RE[3]: not a cheat
by Alfman on Thu 3rd Oct 2013 20:49 UTC in reply to "RE[2]: not a cheat"
Alfman
Member since:
2011-01-28

viton,

"Typical mobile workloads (non-games) occurs in bursts after some sleep. So likely it can run on max frequency. Throttled-down cpu in benchmarks will show worse performance than the device is actually capable of."

I understand that, however it misses the point that we're no longer measuring the performance as a real application would witness it. It was always possible for benchmark authors to ignore samples from the ramp-up period if that's what they want to do, I've done it myself.

A benchmark can find genuine performance problems caused by OS/CPU throttling, but only if we don't start to make special rules/exceptions for benchmarking code. The whole point is to take the same paths that regular programs take, otherwise we are compromising the integrity of the benchmarks.

I do get the point that there may be times that you want to measure the maximum performance of the hardware without regards to the OS/scheduler/etc, but obviously that's not the case here.

Reply Parent Score: 3