Linked by Thom Holwerda on Mon 19th Sep 2016 14:58 UTC

Oh, benchmarks.

Benchmarks of computer hardware have their uses. Especially if you have a relatively narrow and well-defined set of calculations that you need to perform, benchmarks are great tools to figure out which processor or graphics chip or whatever will deliver the best performance - scientific calculations, graphics processing (e.g. video games), these are all use cases where comparisons between benchmarks of different hardware components can yield useful information.

A different way to put it: benchmarks make sense in a situation where "more power" equals "better results" - better results that are noticable and make a difference. A GTX 1080 will result in better framerates than a GTX 1070 in a modern game like The Witcher 3, because we've not yet hit any (theoretical) framerate limit for that game. A possible future GTX 1090 will most likely yield even better framerates still.

Where benchmarks start to fall apart, however, is in use cases where "more power" does not equal "better results". Modern smartphones are a perfect example of this. Our current crop of smartphones is so powerful, that adding faster processors does not produce any better results for the kinds of ways in which we use these devices. Twitter isn't going to open or load any faster when you add a few hundred megahertz.

In other words, modern smartphones have bottlenecks, but the processor or RAM certainly isn't one of them. Before you can even reach the full potential of your quad-core 2.4Ghz 6GB RAM phone, your battery will run out (or explode), or your network connection will be slow or non-existent.

As a result, I never cared much for benchmarking smartphones. In 2013, in the wake of Samsung cheating in benchmarks, I wrote that "if you buy a phone based on silly artificial benchmark scores, you deserve to be cheated", and today, now that Apple is leading (in one subset of processor) benchmarks with its latest crop of mobile processors, the same still applies.

So when John Gruber posted about Apple A10 Fusion benchmarks...

Looking at Geekbench's results browser for Android devices, there are a handful of phones in shouting distance of the iPhone 7 for multi-core performance, but Apple's A10 Fusion scores double on single-core.

...I snarked:

Funny how just like in the PPC days, benchmarks only start mattering when they favour [insert platform of choice].

Setting aside the validity of Geekbench (Linus Torvalds has an opinion!), this seems to be the usual pointless outcome of these penis-measuring contests: when the benchmarks favour you, benchmarks are important and crucial and the ultimate quanitification of greatness. When the benchmarks don't favour you, they are meaningless and pointless and the world's worst yardsticks of greatness. Anywhere in between, and you selectively pick and choose the benchmarks that make you look best.

I didn't refer to Apple's PowerPC days for nothing. Back then, Apple knew it was using processors with terrible performance and energy requirements, but still had to somehow convince the masses that PowerPC was better faster stronger than x86; claims which Apple itself exposed - overnight - as flat-out lies when the company switched to Intel.

When I use my Nexus 6P and iPhone 6S side-by-side, my Nexus 6P feels a lot faster, even though benchmarks supposedly say it has a crappier processor and a slower operating system. Applications and operations seem equally fast to me, but Android makes everything feel faster because it has far superior ways of dealing with and switching between multiple applications, thanks to the pervasiveness of activities and intents or the ability to set your own default applications.

Trying to quantify something as elusive and personal as user experience by crowing about the single-thread performance of the processor it runs on is like trying to buy a family car based on its top speed. My 2009 Volvo S80's 2.5L straight-5 may propel the car to a maximum speed of 230km/h, but I'm much more interested in how comfortable the seats are, all the comfort options it has, if it looks good (it does), and so on. Those are the actual things that matter, because the likelihood of ever even approaching that 230km/h is very slim, at best.

I bought an iPhone 6S and Apple Watch late last year and used them for six months because I feel that as someone who writes about every platform under the sun, I should be using them as much as (financially and practically) possible. I used the iPhone 6S as my only smartphone for six months, but after six months of fighting iOS and Apple every step of the way, every single day, I got fed up and bought the Nexus 6P on impulse.

Not once during those six months did I think to myself "if only this processor was 500Mhz faster" or "if only this thing had 4GB of RAM". No; I was thinking "why can't I set my own default applications, because Apple's are garbage" or "why is deep linking/inter-application communication non-existent, unreliable, broken, and restricted to first-party applications?" or "why is every application a visual and behavioural island with zero attention to consistency?".

iOS could be running on a quantum computer from Urbana, Illinois, and it wouldn't solve any of those problems.

The funny thing is - Gruber actually agrees with me:

I like reading/following Holwerda, because he's someone who I feel keeps me on my toes. But he's off-base here. I'm certainly not saying that CPU or GPU performance is a primary reason why anyone should buy an iPhone instead of an Android phone. In fact, I'll emphasize that if the tables were turned and it were Android phones that were registering Geekbench scores double those of the iPhone, I would still be using an iPhone. In the same way that I've been using Macs, non-stop, since I first purchased a computer in 1991. Most of the years from 1991 until the switch to Intel CPUs in 2007, the Mac was behind PCs in performance. I never argued then that performance didn't matter - only that for me, personally, the other benefits of using a Mac (the UI design of the system, the quality of the third-party apps, the build quality of the hardware, etc.) outweighed the performance penalty Macs suffered. The same would be true today if Apple's A-series chips were slower than Qualcomm's CPUs for Android.

So, he'd be buying iPhone even if the benchmark tables were turned, thereby agreeing with me that when it comes to phones, benchmarks are entirely meaningless. Nobody buys a smartphone based on processor benchmark scores; at this point in time, people mostly buy smartphones based on the smartphone they currently have (i.e., what platform they are currently using) and price.

That being said, there is one reason why benchmarks of Apple's latest mobile processors are quite interesting: Apple's inevitable upcoming laptop and desktop switchover to its own processors. OS X (or macOS or whatever) has been in maintenance mode ever since the release and success of the iPhone, and by now it's clear that Apple is going to retire OS X in favour of a souped-up iOS over the coming five years.

I know a lot of people still aren't seeing the forest through the trees on this one, but you can expect the first "iOS" MacBook within 1-2 years. I put iOS between quotation marks because that brand of iOS won't be the iOS you have on your phone today, but a more capable, expanded version of it.

Vlad Savov:

It sounds wild, but the A10 looks to have the power and efficiency to handle the workload of a full PC. This coalescence of mobile and desktop PCs is driven by forces on both sides: mobile chips are getting more potent at the same time as our power needs are shrinking and our tasks become more mobile. If you think your workplace isn't changing much because there are a bunch of weathered Dell workstations sitting next to frumpy HP printers, consider just how much more work every one of your officemates is doing outside the office, on their phone. And all those grand and power-hungry x86 applications that might have kept people running macOS - Adobe's Photoshop and Lightroom being two key examples - well, they're being ported to iOS in almost their full functionality, having been incentivized by the existence of Apple's iPad Pro line, last year's harbinger for this year's performance jump.

Unlike Windows, whose x86 reliance is tied to its dominance of the lucrative PC gaming market, Apple really has very few anchors locking it down to macOS. The Cupertino company has been investing the vast majority of its development time into the mobile iOS for years now, and that shows in the different rates of progress between its two pieces of software. macOS is, in many ways, legacy software just waiting for the right moment to be deprecated. It’s getting a fresh lick of paint now and then, but most of its novelties now relate to how it links back to Apple's core iOS and iPhone business.

This is where benchmarking and the performance of Apple's A10 Fusion processor do come into play, because even in the constrained environment of a smartphone, it seems to be reaching performance levels of laptop and desktop processors.

That "iOS" MacBook is closer than you think.

Thread beginning with comment 634566
To read all comments associated with this story, please click here.
CPU power and OSes
by dpJudas on Mon 19th Sep 2016 15:59 UTC
Member since:

That "iOS" MacBook is closer than you think.

No, it really isn't. Let me try to explain an alternative viewpoint:

Phones have been fast enough to drive a desktop for many years now. The phone in your pocket is far faster than the G4 PowerBook and the phone GPU is faster than a 2006 high end GPU. The memory is has is also more than plenty. And it has multiple cores. This means a modern phone could *easily* drive a Windows XP/Vista age desktop PC.

So why doesn't it? The answer to this question is complicated. And ultimately also why an "iOS" MacBook will fail if it is executed the way you're imagining it will. There's basically several factors at play here:

1) The user interface between a phone and a desktop is different. At a distance WIMP and touch look kinda alike, they both have views in a tree, they react to events, and draw to the screen. But the devil is in the detail. When you have a much larger screen, a physical keyboard, and a mouse, your way of interacting with the application changes fundamentally in ways the require the entire user interface to be designed differently.

Microsoft had to learn this lesson TWICE. First when they tried to minimize WIMP with Windows Phone 5 and older, and second time when they tried to upscale Metro to the desktop with Windows 8.

If Apple is foolish enough to try this they will fail in the same way. The only way to do this properly is to include BOTH Cocoa and Cocoa Touch into such a solution, and I don't mean where a phone app can kinda pretend to be a WIMP app like Microsoft thinks it can.

2) The simplicity of having one device do one thing and do it well. For the phone to replace your desktop PC, it has to integrate seamlessly with your monitor, keyboard and mouse. This seems simple at a distance, but software and hardware tend to not play well together unless they come from the same vendor.

For a classic example of how difficult this is to do, see cars and phone integration. Naturally, Apple has the arrogance to think they can do a fully integrated solution here, but I think they will fail to understand that not everyone is 110% Apple-only hardware. The same lesson they might be about to learn about headphones.

3) The battery. It may never become an acceptable price to pay that your phone battery is running low just because you're at home using your desktop PC. I don't think this is about to be solved anytime soon.

The actions of Apple lately seem to indicate they just might be foolish enough to attempt what you're suggesting though. If they do, it will be their waterloo.

Reply Score: 17

RE: CPU power and OSes
by ilovebeer on Mon 19th Sep 2016 16:38 in reply to "CPU power and OSes"
ilovebeer Member since:

I agree with you, and it's beating a dead horse. Yes, cellphones and desktops have overlapping uses but that doesn't mean it's smart to ignore their differences. We've seen this movie before. Whether it's trying to shoehorn one into the other or create some hybrid of the two, it always ends in failure.

The reason we have different tools for different jobs is because it works better that way. Software isn't an exception. You wouldn't think that's such a hard concept to grasp.

Reply Parent Score: 4

RE: CPU power and OSes
by Bill Shooter of Bul on Mon 19th Sep 2016 17:39 in reply to "CPU power and OSes"
Bill Shooter of Bul Member since:

See Plasma active.

Now its buggy as heck and doesn't have a great reference implementation (IMHO). But look beyond that to the design ( how it works, not the skins or icon sets). One application, different representations of input. It was/is beautiful.

#2 That is simple. Hooking up peripherals to hardware is a solved problem.

#3 Don't run on battery when using as a desktop! Plug it in/wireless charge. Solved.

In my next post, we'll discover world piece can be achieved in 4 simple steps.

Reply Parent Score: 3

RE[2]: CPU power and OSes
by fretinator on Mon 19th Sep 2016 21:30 in reply to "RE: CPU power and OSes"
fretinator Member since:

In my next post, we'll discover world piece can be achieved in 4 simple steps.

As we all know, there is no I in World Peace.

Sorry, couldn't resist.

Reply Parent Score: 9

RE: CPU power and OSes
by sukru on Mon 19th Sep 2016 20:03 in reply to "CPU power and OSes"
sukru Member since:

There are certain assumptions in mobile applications that do not scale to larger screens, and similarly desktop applications make assumptions that does not work well on mobile hardware.

The keyword is the "user experience". It's not about how you render a button or a list, but more about how it behaves in general.

A mobile application (like a browser, or email reader) will use a "back" button at the top (in IOS), and if you're lucky a navigation menu on the left in landscape / tablet mode. This is done, because app assumes limited screen real estate, and control by a human finger.

A corresponding desktop application can have a menu bar at the top of the screen, have multiple tabs in the app for navigation, and can expect mouse and keyboard to be ready.

The games will depend on tilt and motion on mobile devices, but the desktop one can have a controller with several axes, and 10+ buttons connected. (Yes there are bluetooth controllers, but the market penetration is very small).

Overall it *is* possible to design to different UIs for the same application with separate mobile and desktop experiences. (Actually you can now include tablets, and TVs as separate modes as well). But then you're left with two completely different applications (like Photoshop on the desktop and Photoshop mobile)

Edited 2016-09-19 20:03 UTC

Reply Parent Score: 3

RE: CPU power and OSes
by nicubunu on Tue 20th Sep 2016 06:13 in reply to "CPU power and OSes"
nicubunu Member since:

For me, a phone/phone UI could probably never replace a desktop, but for a large category of people it will do it very well: they simply uses their devices to consume content created by others.

Reply Parent Score: 3

RE: CPU power and OSes
by jbauer on Tue 20th Sep 2016 11:52 in reply to "CPU power and OSes"
jbauer Member since:

Microsoft had to learn this lesson TWICE. First when they tried to minimize WIMP with Windows Phone 5 and older, and second time when they tried to upscale Metro to the desktop with Windows 8.

They still haven't learned that lesson though. Every new UI they build on Windows 10 is a UI meant for touch that you're supposed to use in the exact same way with a keyboard and mouse.

I cringe every time I have to go into the new Settings.

Reply Parent Score: 4

RE: CPU power and OSes
by unclefester on Wed 21st Sep 2016 11:13 in reply to "CPU power and OSes"
unclefester Member since:

Phones have been fast enough to drive a desktop for many years now. The phone in your pocket is far faster than the G4 PowerBook and the phone GPU is faster than a 2006 high end GPU. The memory is has is also more than plenty. And it has multiple cores. This means a modern phone could *easily* drive a Windows XP/Vista age desktop PC.

I'm using a Windows 10 laptop with a "phone" hardware (Intel Atom 7535F, 2GB RAM, 32GB SSD) and it is barely usable. Even scrolling a webpage smoothly is impossible. I also use a Windows 10 Core 2 from 2008 with similar benchmark performance that is vastly faster (except booting) and more responsive in the real world despite having a slower hard disk.

Reply Parent Score: 2