Linked by Thom Holwerda on Thu 3rd Oct 2013 16:07 UTC
Benchmarks

With the exception of Apple and Motorola, literally every single OEM we've worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It's possible that older Motorola devices might've done the same thing, but none of the newer devices we have on hand exhibited the behavior. It's a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.

Pathetic, but this has been going on in the wider industry for as long as I can remember - graphics chip makers come to mind, for instance. Still, this is clearly scumbag behaviour designed to mislead consumers.

On the other hand, if you buy a phone based on silly artificial benchmark scores, you deserve to be cheated.

Order by: Score:
Deserve to be cheated?
by RshPL on Thu 3rd Oct 2013 16:20 UTC
RshPL
Member since:
2009-03-13

Why is that? If one wants to have the smoothest web experience it would be perfectly reasonable to look at these particular benchmarks. I am rarely looking at benchmark results myself but isn't it a bit unfair to judge people that do?

Edited 2013-10-03 16:24 UTC

Reply Score: 6

RE: Deserve to be cheated?
by Bill Shooter of Bul on Thu 3rd Oct 2013 16:32 UTC in reply to "Deserve to be cheated?"
Bill Shooter of Bul Member since:
2006-07-14

Well, first of all, no one "deserves to be cheated". That's just a lie con men tell themselves to help them sleep at night.

But seriously, benchmarks aren't very accurate to real life scenarios on phones. I don't understand why anyone would take the results very seriously. I think this is what Thom is trying to say. Anyone who took them seriously has already been conned into thinking they were useful.

Reply Score: 10

RE: Deserve to be cheated?
by Alfman on Thu 3rd Oct 2013 17:31 UTC in reply to "Deserve to be cheated?"
Alfman Member since:
2011-01-28

RshPL,

Agreed. I don't really do much with phones, however I do consider benchmarks when it comes to buying personal computer components and I have to wonder the extent of cheating that goes on there.

"On the other hand, if you buy a phone based on silly artificial benchmark scores, you deserve to be cheated."

-1 for this Thom.

Benchmarks can certainly can be valuable for people who understand what they mean and how it relates to their needs. However if the results are being manipulated into producing unfair results then it completely defeats the point in having them for comparison. The shame should be squarely on those artificially manipulating the results, not those who were mislead into believing those results were genuine.


As an aside: if it's easy (and safe) to control the performance characteristics of the chips per application, then perhaps this functionality could be exposed as a bona fide feature for boosting performance in general (presumably at the expense of battery life). This way it would not be misleading, and the users could turn on and off the performance boost as they deem fit.

Edited 2013-10-03 17:46 UTC

Reply Score: 3

RE: Deserve to be cheated?
by lucas_maximus on Thu 3rd Oct 2013 21:15 UTC in reply to "Deserve to be cheated?"
lucas_maximus Member since:
2009-08-18

I think the issue is that they disabled the thermal limits to bench better, which isn't going to reflecting real world usage.

Reply Score: 4

themwagency
Member since:
2013-03-06

This really makes the consistent wins for the iPhone 5S all the more impressive. It was these wins that many on these forums called into question because they beat Android phones so significantly.

Apple seems to be quite a bit ahead of the other ARM vendors, which is pretty impressive. More impressive still is that Apple bested these phones with their artificially inflated benchmarks.

Edited 2013-10-03 16:33 UTC

Reply Score: 1

Fergy Member since:
2006-04-10

This really makes the consistent wins for the iPhone 5S all the more impressive. It was these wins that many on these forums called into question because they beat Android phones so significantly.

Apple seems to be quite a bit ahead of the other ARM vendors, which is pretty impressive. More impressive still is that Apple bested these phones with their artificially inflated benchmarks.

What makes you think Apple didn't cheat?

Reply Score: 1

themwagency Member since:
2013-03-06

What makes you think Apple didn't cheat?


Did you not read the article or even the summary???

Reply Score: 1

Benchmarks
by Jbso on Thu 3rd Oct 2013 16:35 UTC
Jbso
Member since:
2013-01-05

It seems a bit silly to for journalist to mislead readers into believing irrelevant benchmarks are meaningful, then complain about manufacturers trying mislead them about benchmark results. Why not run tests on apps people really use? Problem solved.

Reply Score: 1

RE: Benchmarks
by RshPL on Fri 4th Oct 2013 11:25 UTC in reply to "Benchmarks"
RshPL Member since:
2009-03-13

1) Test apps that people use. 2) Measure time and performance. 3) Compare figures with other products. 4) Eureka! You have just invented a benchmark.

Reply Score: 1

themwagency
Member since:
2013-03-06

All of this just shows Apple's A7 to be that much more amazing performance wise. If Android OEM's spent as much effort optimizing silicon as they did trying to optimize benchmarks, they would probably be neck and neck in performance.

Reply Score: 1

phoenix Member since:
2005-07-11

Or, spent a bit of time optimising drivers, Dalvik, apps, etc for the existing silicon.

You can get impressive performance gains by compiling AOSP yourself using newer versions of GCC, the Linaro stack, updated drivers, etc.

Problem with Android is it's very much a "lowest-common denominator" situation, where the phone makers don't take a lot of time to optimise for their own hardware.

Reply Score: 6

themwagency Member since:
2013-03-06

Or, spent a bit of time optimising drivers, Dalvik, apps, etc for the existing silicon.


Why not both? The types of benchmarks referenced in this article have more to do with showing the capability of the hardware. The drivers would should benefits for the apps however they wouldn't benefit these types of benchmarks. It doesn't mean performance gains wouldn't be achieved however if they're looking for a benchmark speed crown then silicon optimization is where they'll have to invest resources.

Edited 2013-10-03 16:53 UTC

Reply Score: 1

darknexus Member since:
2008-07-15

Or, spent a bit of time optimising drivers, Dalvik, apps, etc for the existing silicon.

Couldn't agree more. The A7 is the smaller part of why the iPhone 5S is so smooth. The greater part is the optimized operating system. This is one thing Apple has always understood: you don't need nearly as much raw power if you optimize for the hardware you're going to be using. Android is anything but optimized, and OEMs have no interest in improving the state of things for it. Why put the work into code optimization when you can just pack more raw power in and convince people to buy a brand new phone in a year? I'd love to see what Android could be if someone took AOSP and optimized the hell out of it for a specific device as Apple do with iOS. Unfortunately no one making Android phones has any reason to try, not even Google.

Reply Score: 5

leos Member since:
2005-09-21

Or, spent a bit of time optimising drivers, Dalvik, apps, etc for the existing silicon.

You can get impressive performance gains by compiling AOSP yourself using newer versions of GCC, the Linaro stack, updated drivers, etc.

Problem with Android is it's very much a "lowest-common denominator" situation, where the phone makers don't take a lot of time to optimise for their own hardware.


They should take all the time they spend adding crapware and stupid "features" to their phones and spend it on optimization instead.

Reply Score: 3

tkeith Member since:
2010-09-01

To be fair, they should be ahead. Apple designs the processor itself, meaning they can design the phone around it from the beginning. Other manufacturers(besides Samsung) have to wait for Qualcom or Nvidia, ect. to design their processor, then start a phone design. In addition Apple is generally behind the curve on adding new features,(LTE) so they can devote more resources to speed and efficiency.

Reply Score: 3

themwagency Member since:
2013-03-06

To be fair, they should be ahead. Apple designs the processor itself, meaning they can design the phone around it from the beginning. Other manufacturers(besides Samsung) have to wait for Qualcom or Nvidia, ect. to design their processor, then start a phone design.


Agreed. The integrated approach tends to create more efficient hardware and software


In addition Apple is generally behind the curve on adding new features,(LTE) so they can devote more resources to speed and efficiency.


I disagree here. Apple is just as capable to add features as any other solutions provider and it shows in what they deliver. They are often times first in new features or offer best of breed alternatives. Unlike their Android counterparts, Apple is often very selective about what they add and don't add. The goal is not to pack the most features into a device. That's something I believe many technology companies haven't yet grasped yet Apple has learned very well.

It's not as if Apple is taking engineers off of silicon development to work on other projects. They can and do both simultaneously. LTE integration had less to do with lack of resources than it did maximizing returns on their pre-manufactured products.

Edited 2013-10-03 20:05 UTC

Reply Score: 2

BushLin Member since:
2011-01-26

Agreed. The integrated approach tends to create more efficient hardware and software


Yep. There is no doubt that developing software for fixed hardware can lead to a better experience compared to running on higher specification kit with software which caters for more devices.

That's something I believe many technology companies haven't yet grasped yet Apple has learned very well.


Games consoles and network appliances are obvious examples of where this also happens. I remember this thing called the Amiga too.

LTE integration had less to do with lack of resources than it did maximizing returns on their pre-manufactured products.


I was going to make an argument that Apple are not doing all these things solely to maximise the experience of their product (that's obviously very important to them too) but also not to offer so much that they can't offer a compelling upgrade in a year or so.

Yet, when I read your text, despite the general gist of 'Apple do everything perfect for our benefit', your quote is spot on.

Everything they do is to maximise returns.

I don't like most of it (not that I buy their stuff), I can almost admire the boldness of what they do especially as their fan base continues to increase and pay their homage, willingly in monetary form on a regular basis.

But let's not kid ourselves, Apple are not increasing the efficiency in the software of anything other than their current device. The widely accepted status quo is that Apple will cripple your previously buttery experience with a new OS and pretty much force you to upgrade because that's what brings them profit.

Reply Score: 3

Tony Swash Member since:
2009-08-22


But let's not kid ourselves, Apple are not increasing the efficiency in the software of anything other than their current device. The widely accepted status quo is that Apple will cripple your previously buttery experience with a new OS and pretty much force you to upgrade because that's what brings them profit.


You are kidding yourself. I am running iOS7 on an iPhone 4s and iPad 2 with no problems and the last few OS upgrades have delivered plenty of new features. The simple and obvious truth is that Apple supports old iOS devices with OS updates and offers and delivers OS updates to it's installed user base in a vastly better and more comprehensive way than any other mobile OS or device vendor. Less than a month after release iOS is on two thirds of all iOS devices. There is nothing in the Android world like Apple's support of it's installed base, nothing.

Reply Score: 4

BushLin Member since:
2011-01-26

Oh god, I forgot what happens what you dare speak against the almighty fruit god. Please don't mistake me for an Android apologist (or an anything apologist).

Just imagine for a moment that you're objective and don't ignore problems:

Are your older Apple devices less responsive or more responsive following their new iOS versions?

Reply Score: 2

TroubleFunk Member since:
2013-10-04

It seems Tony gave you a fairly calm and rational answer.

You respond with some silly "fruit company" comment.

You should probably answer like an adult next time.

Reply Score: 3

BushLin Member since:
2011-01-26

"Tony" tried to misframe the argument.

I was talking about software optimisation in context and unfortunately "Tony" has a well documented history of fitting the stereotype I spoke of.

Are you "Tony" in disguise?

Reply Score: 2

BushLin Member since:
2011-01-26

Are your older Apple devices less responsive or more responsive following their new iOS versions?


Not going to answer that one then eh?

Reply Score: 2

themwagency
Member since:
2013-03-06

I have both android and iOS devices but I'm a big fan of the engineering skill of Apple when it comes to leer frequency chips outperforming much higher freq chips.

I'd love to see the 5S in a chRt with the top phones with all the cheats turned off as an example of what can be done with honest to goodness engineering know how and hard work without cheating.

Reply Score: 1

Dollars to donuts
by themwagency on Thu 3rd Oct 2013 18:28 UTC
themwagency
Member since:
2013-03-06

Dollars to donuts that someone here is writing an editorial to either mitigate these android findings or try and demonstrate that Apple does it too.

Reply Score: 1

RE: Dollars to donuts
by darknexus on Thu 3rd Oct 2013 18:57 UTC in reply to "Dollars to donuts"
darknexus Member since:
2008-07-15

Dollars to donuts that someone here is writing an editorial to either mitigate these android findings or try and demonstrate that Apple does it too.

Well, if Apple are cheating and it is exposed, that's all for the better. I'd prefer that all cheating be exposed. However, it doesn't appear that Apple are cheating and, judging by the real-world performance of the new A7 coupled with software that is optimized for it, they have no need to even consider fudging the numbers here.

Reply Score: 2

RE[2]: Dollars to donuts
by themwagency on Thu 3rd Oct 2013 19:09 UTC in reply to "RE: Dollars to donuts"
themwagency Member since:
2013-03-06

"Dollars to donuts that someone here is writing an editorial to either mitigate these android findings or try and demonstrate that Apple does it too."

"Well, if Apple are cheating and it is exposed, that's all for the better. I'd prefer that all cheating be exposed. However, it doesn't appear that Apple are cheating"



I totally agree... however they key words in my statement was "try and demonstrate" e.g. introduce unsubstantiated doubt in an effort to mitigate the difference in performance.

Edited 2013-10-03 19:11 UTC

Reply Score: 1

RE[2]: Dollars to donuts
by Delgarde on Thu 3rd Oct 2013 23:28 UTC in reply to "RE: Dollars to donuts"
Delgarde Member since:
2008-08-19

However, it doesn't appear that Apple are cheating


Or at least, aren't getting caught at it. If you're going to cheat, you should at least cheat well...

Reply Score: 2

not a cheat
by viton on Thu 3rd Oct 2013 18:34 UTC
viton
Member since:
2005-08-09

For accurate results benchmark needs stable frequency across the particular interval of measurement.

Reply Score: 3

RE: not a cheat
by Alfman on Thu 3rd Oct 2013 19:06 UTC in reply to "not a cheat"
Alfman Member since:
2011-01-28

viton,

"For accurate results benchmark needs stable frequency across the particular interval of measurement."

It's wrong to conduct the benchmark against a cpu/gpu super-configuration if the intention is to get an idea of the performance for a normal configuration (which it generally is).


Non deterministic behavior like this should be mitigated by conducting the benchmark over a longer period and/or repeating it a few times. At this point you get a better idea of min/max/avg/median/etc.

Changing the configuration for the benchmark makes the benchmark less accurate in terms of what it's trying to measure (even if the measurements are more consistent as you suggest).


O/T news: usa.gov (every gov website for that matter) "Due to a lapse in funding, the US government has shut down.".

Reply Score: 5

RE: not a cheat
by phoenix on Thu 3rd Oct 2013 19:14 UTC in reply to "not a cheat"
phoenix Member since:
2005-07-11

Depends.

Are you trying to measure what the absolute best performance is for taskX or hardware-bitY?

Or, are you trying to measure the typical experience of a user doing taskX?

For the former, sure, enable the "performance" CPU governor, the "performance" GPU governor, switch to the "noop" disk scheduler, and run things at 100% utilisation. This is really only good for marketing purposes, since the battery life is atrocious if you actually run the phone/tablet like this.

For the latter, you want to measure how things work when they "race-to-sleep", how/if they bring extra cores online, etc. This is what people really want to see, but that no one wants to publish. The closest you get is in battery life tests.

Reply Score: 4

RE[2]: not a cheat
by viton on Thu 3rd Oct 2013 19:53 UTC in reply to "RE: not a cheat"
viton Member since:
2005-08-09

Typical mobile workloads (non-games) occurs in bursts after some sleep. So likely it can run on max frequency. Throttled-down cpu in benchmarks will show worse performance than the device is actually capable of.

Reply Score: 3

RE[3]: not a cheat
by Alfman on Thu 3rd Oct 2013 20:49 UTC in reply to "RE[2]: not a cheat"
Alfman Member since:
2011-01-28

viton,

"Typical mobile workloads (non-games) occurs in bursts after some sleep. So likely it can run on max frequency. Throttled-down cpu in benchmarks will show worse performance than the device is actually capable of."

I understand that, however it misses the point that we're no longer measuring the performance as a real application would witness it. It was always possible for benchmark authors to ignore samples from the ramp-up period if that's what they want to do, I've done it myself.

A benchmark can find genuine performance problems caused by OS/CPU throttling, but only if we don't start to make special rules/exceptions for benchmarking code. The whole point is to take the same paths that regular programs take, otherwise we are compromising the integrity of the benchmarks.

I do get the point that there may be times that you want to measure the maximum performance of the hardware without regards to the OS/scheduler/etc, but obviously that's not the case here.

Reply Score: 3

In other news:
by lucas_maximus on Thu 3rd Oct 2013 20:20 UTC
lucas_maximus
Member since:
2009-08-18

Fisherman describes his yesterdays smelly catch as:

The finest freshly fished Mediterranean tuna caught by line and hook


Edited 2013-10-03 20:21 UTC

Reply Score: 3