Linked by Thom Holwerda on Mon 19th Sep 2016 14:58 UTC
Apple

Oh, benchmarks.

Benchmarks of computer hardware have their uses. Especially if you have a relatively narrow and well-defined set of calculations that you need to perform, benchmarks are great tools to figure out which processor or graphics chip or whatever will deliver the best performance - scientific calculations, graphics processing (e.g. video games), these are all use cases where comparisons between benchmarks of different hardware components can yield useful information.

A different way to put it: benchmarks make sense in a situation where "more power" equals "better results" - better results that are noticable and make a difference. A GTX 1080 will result in better framerates than a GTX 1070 in a modern game like The Witcher 3, because we've not yet hit any (theoretical) framerate limit for that game. A possible future GTX 1090 will most likely yield even better framerates still.

Where benchmarks start to fall apart, however, is in use cases where "more power" does not equal "better results". Modern smartphones are a perfect example of this. Our current crop of smartphones is so powerful, that adding faster processors does not produce any better results for the kinds of ways in which we use these devices. Twitter isn't going to open or load any faster when you add a few hundred megahertz.

In other words, modern smartphones have bottlenecks, but the processor or RAM certainly isn't one of them. Before you can even reach the full potential of your quad-core 2.4Ghz 6GB RAM phone, your battery will run out (or explode), or your network connection will be slow or non-existent.

As a result, I never cared much for benchmarking smartphones. In 2013, in the wake of Samsung cheating in benchmarks, I wrote that "if you buy a phone based on silly artificial benchmark scores, you deserve to be cheated", and today, now that Apple is leading (in one subset of processor) benchmarks with its latest crop of mobile processors, the same still applies.

So when John Gruber posted about Apple A10 Fusion benchmarks...

Looking at Geekbench's results browser for Android devices, there are a handful of phones in shouting distance of the iPhone 7 for multi-core performance, but Apple's A10 Fusion scores double on single-core.

...I snarked:

Funny how just like in the PPC days, benchmarks only start mattering when they favour [insert platform of choice].

Setting aside the validity of Geekbench (Linus Torvalds has an opinion!), this seems to be the usual pointless outcome of these penis-measuring contests: when the benchmarks favour you, benchmarks are important and crucial and the ultimate quanitification of greatness. When the benchmarks don't favour you, they are meaningless and pointless and the world's worst yardsticks of greatness. Anywhere in between, and you selectively pick and choose the benchmarks that make you look best.

I didn't refer to Apple's PowerPC days for nothing. Back then, Apple knew it was using processors with terrible performance and energy requirements, but still had to somehow convince the masses that PowerPC was better faster stronger than x86; claims which Apple itself exposed - overnight - as flat-out lies when the company switched to Intel.

When I use my Nexus 6P and iPhone 6S side-by-side, my Nexus 6P feels a lot faster, even though benchmarks supposedly say it has a crappier processor and a slower operating system. Applications and operations seem equally fast to me, but Android makes everything feel faster because it has far superior ways of dealing with and switching between multiple applications, thanks to the pervasiveness of activities and intents or the ability to set your own default applications.

Trying to quantify something as elusive and personal as user experience by crowing about the single-thread performance of the processor it runs on is like trying to buy a family car based on its top speed. My 2009 Volvo S80's 2.5L straight-5 may propel the car to a maximum speed of 230km/h, but I'm much more interested in how comfortable the seats are, all the comfort options it has, if it looks good (it does), and so on. Those are the actual things that matter, because the likelihood of ever even approaching that 230km/h is very slim, at best.

I bought an iPhone 6S and Apple Watch late last year and used them for six months because I feel that as someone who writes about every platform under the sun, I should be using them as much as (financially and practically) possible. I used the iPhone 6S as my only smartphone for six months, but after six months of fighting iOS and Apple every step of the way, every single day, I got fed up and bought the Nexus 6P on impulse.

Not once during those six months did I think to myself "if only this processor was 500Mhz faster" or "if only this thing had 4GB of RAM". No; I was thinking "why can't I set my own default applications, because Apple's are garbage" or "why is deep linking/inter-application communication non-existent, unreliable, broken, and restricted to first-party applications?" or "why is every application a visual and behavioural island with zero attention to consistency?".

iOS could be running on a quantum computer from Urbana, Illinois, and it wouldn't solve any of those problems.

The funny thing is - Gruber actually agrees with me:

I like reading/following Holwerda, because he's someone who I feel keeps me on my toes. But he's off-base here. I'm certainly not saying that CPU or GPU performance is a primary reason why anyone should buy an iPhone instead of an Android phone. In fact, I'll emphasize that if the tables were turned and it were Android phones that were registering Geekbench scores double those of the iPhone, I would still be using an iPhone. In the same way that I've been using Macs, non-stop, since I first purchased a computer in 1991. Most of the years from 1991 until the switch to Intel CPUs in 2007, the Mac was behind PCs in performance. I never argued then that performance didn't matter - only that for me, personally, the other benefits of using a Mac (the UI design of the system, the quality of the third-party apps, the build quality of the hardware, etc.) outweighed the performance penalty Macs suffered. The same would be true today if Apple's A-series chips were slower than Qualcomm's CPUs for Android.

So, he'd be buying iPhone even if the benchmark tables were turned, thereby agreeing with me that when it comes to phones, benchmarks are entirely meaningless. Nobody buys a smartphone based on processor benchmark scores; at this point in time, people mostly buy smartphones based on the smartphone they currently have (i.e., what platform they are currently using) and price.




That being said, there is one reason why benchmarks of Apple's latest mobile processors are quite interesting: Apple's inevitable upcoming laptop and desktop switchover to its own processors. OS X (or macOS or whatever) has been in maintenance mode ever since the release and success of the iPhone, and by now it's clear that Apple is going to retire OS X in favour of a souped-up iOS over the coming five years.

I know a lot of people still aren't seeing the forest through the trees on this one, but you can expect the first "iOS" MacBook within 1-2 years. I put iOS between quotation marks because that brand of iOS won't be the iOS you have on your phone today, but a more capable, expanded version of it.

Vlad Savov:

It sounds wild, but the A10 looks to have the power and efficiency to handle the workload of a full PC. This coalescence of mobile and desktop PCs is driven by forces on both sides: mobile chips are getting more potent at the same time as our power needs are shrinking and our tasks become more mobile. If you think your workplace isn't changing much because there are a bunch of weathered Dell workstations sitting next to frumpy HP printers, consider just how much more work every one of your officemates is doing outside the office, on their phone. And all those grand and power-hungry x86 applications that might have kept people running macOS - Adobe's Photoshop and Lightroom being two key examples - well, they're being ported to iOS in almost their full functionality, having been incentivized by the existence of Apple's iPad Pro line, last year's harbinger for this year's performance jump.

Unlike Windows, whose x86 reliance is tied to its dominance of the lucrative PC gaming market, Apple really has very few anchors locking it down to macOS. The Cupertino company has been investing the vast majority of its development time into the mobile iOS for years now, and that shows in the different rates of progress between its two pieces of software. macOS is, in many ways, legacy software just waiting for the right moment to be deprecated. It’s getting a fresh lick of paint now and then, but most of its novelties now relate to how it links back to Apple's core iOS and iPhone business.

This is where benchmarking and the performance of Apple's A10 Fusion processor do come into play, because even in the constrained environment of a smartphone, it seems to be reaching performance levels of laptop and desktop processors.

That "iOS" MacBook is closer than you think.

Thread beginning with comment 634594
To read all comments associated with this story, please click here.
I Disagree
by galvanash on Mon 19th Sep 2016 21:40 UTC
galvanash
Member since:
2006-01-25

That "iOS" MacBook is closer than you think.


I just don't think this is really where we are going...

We already have an iOS Macbook. Its call an iPad Pro, and it isn't exactly setting the world on fire. It is selling ok, it isn't a failure or anything, but imo it has a rather limited audience when it comes to "professional" computing.

Its not because of the CPU, and its not because of the form factor, its because iOS simply cannot host most "professional" applications as well as OSX can. And I have seen nothing that would make me think that is going to change in the future.

When will iOS get a real bash shell? When will it get real mouse support? How do I access shares on my corporate network? How do I spin up a web server or a node server? How do I run a python or ruby app locally? Where is xCode? Where is bootcamp? How do I run a VM? Where's docker support?

None of those things are contingent on there being an x86 chip under the hood, they could easily be done on an A10 or future ARM CPUs. The problem isn't ARM, its iOS...

I personally do think we will see an ARM Macbook sooner rather than later, it just won't be running iOS. It might have the ability to run iOS software, but without all the stuff I mentioned above it won't be a macbook, it will just be a glorified iPad.

The problem is an iPad simply isn't (and never will) be good enough for most "professionals". Stuffing an iPad into a Macbook case won't make it a Macbook - it will just make it a tablet with a better keyboard. It will still have all the same problems, all of them rooted in the fact that it runs iOS instead of OSX.

There is nothing wrong imo with a Macbook with an ARM CPU, as long as it is running OSX. Maybe we lose the ability to run bootcamp, but that is a small lose in functionality compared to what we lose switching to iOS.

Microsoft figured that out and adapted to it. They simply include both their mobile and desktop environments together on their systems. Apple will have to do the same or they will never be able to convert the majority of professional users.

Reply Score: 6

RE: I Disagree
by darknexus on Tue 20th Sep 2016 11:45 in reply to "I Disagree"
darknexus Member since:
2008-07-15

I do have to point out one thing: professional is not necessarily "software developer." Don't make the same mistake I did. A few of your points such as network shares are relevant to all professionals (for those there actually are ways to do it by the way), but all your questions about bootcamp/vm/docker/server, to 95% of the professional crowd, is absolutely irrelevant.

Reply Parent Score: 2

RE[2]: I Disagree
by Alfman on Tue 20th Sep 2016 16:04 in reply to "RE: I Disagree"
Alfman Member since:
2011-01-28

darknexus,

A few of your points such as network shares are relevant to all professionals (for those there actually are ways to do it by the way), but all your questions about bootcamp/vm/docker/server, to 95% of the professional crowd, is absolutely irrelevant.


I agree with galvanash, the CPU architecture is mostly irrelevant to me and the mobile CPUs are technically capable of handling most workloads, but unless/until the mobile platforms make a serious effort to support productivity use cases, they're not going to be able to replace desktops for my work.

Obviously needs will vary from one professional to the next. I know some developers like to organize their work into different VMs for each unique client and test configuration, I'm sure you can understand why that is useful. Frankly, many of the daemons/tools I need are lacking or poorly implemented on mobile platforms. I expect more from my development desktop. Part of that is just the form factor, and adding a keyboard can help, but even so the software just is not there for me, at least not yet.

Reply Parent Score: 2