The latest browser benchmarks are in… again – seems like there’s a new one every week. This is one of the best “browser battle” articles though. Chrome 13, Firefox 6, IE9, Opera 11.50, and Safari 5.1 are put through 40-something tests on both Windows 7 and Mac OS X Lion. As a PC guy I was pretty impressed with the performance of Safari on OS X, and the reader feature looks awesome too. The author also uncovered a nasty Catalyst bug that makes IE9 render pages improperly and freeze up under heavy loads of tabs. The tables at the end pinpoint the strengths and weaknesses of each browser, which is nicer than a 1-10 or star rating. Good article, and thorough.
“Also, remember these tests were not conducted on an actual Apple-branded Mac system. Based on what we saw in the results of WBGP2: Linux, we really didn’t expect the OS X scores to get so close to the best from Windows. With such a slim margin of victory favoring Chrome in Windows (versus Safari on OS X), it is entirely within the realm of possibility that running these tests on a genuine Apple rig could tip the scales in favor of Safari (or back the other way). A more in-depth Hackintosh versus Macintosh comparison would be needed to confirm one way or the other.”
A major tech review site can’t shell out a few bucks to buy a real Mac? Or is there some philosophical opposition? Way to go, Tom’s.
Whoa, I literally just clicked on submit and the comment score was 2. Is this a new policy…?
Edit: Looks like it is.
Edited 2011-08-29 15:38 UTC
My comments are starting with a score of 2 as well. It’s been that way for a few months.
Maybe it does that after you’ve added a certain number of comments, or gotten a certain number of up votes etc.
That happens to all “Trusted users”. I’m not really sure how you receive such title, but trusted users always start with a score of 2.
Actually, it would be nice if new comments started off with a few points. That way, crappy comments could get voted down below newer (and presumably less crappy) comments could get priority.
You can achieve this in practice by going to My Account -> Preferences and setting the “Score Treshold” in “Site Preferences”.
I have set this to -5 myself.
Edited 2011-08-29 18:35 UTC
Well, if they used a Mac, you’d perhaps have the reverse problem. For some reason, Windows over Bootcamp is, in my (admittedly limited) experience, quite sluggish compared to its performance on a similar-specced PC machine.
Edited 2011-08-29 17:20 UTC
Apple’s drivers for the windows side are a bit slow, not to say bloated. The package is 800~900MB.
After installing Windows on the Mac mini it behaved reasonable but it slowed down after the driver install
Perhaps it’s to make OS X look better…
I figured it was so they could run the tests using the exact same hardware, instead of having to use two different computers that would be slightly different hardware-wise.
Sure, but that makes it an inaccurate real world metric.
It’s not entirely fair, either. I’m sure that OS X is tuned to Apple’s specific hardware. Since Apple alone manages the hardware platform, develops OS X, and also develops Safari, and since they have a limited number of engineers to assign to various tasks, at some point they may very well choose to improve overall system performance or individual application performance through tuning. Actually that would make a ton of sense, since no one else can do that to the same degree. Should Safari then get penalized for a different development strategy, then?
And before someone accuses me of being a fanboy, I run Slackware and ratpoison exclusively on my hand-me-down MacBook. Would run OBSD if the inteldrm driver worked properly.
Eh, I understand both sides to it. I think the best would be to have run them on the same hardware as they did in the article and find a Mac that was closest in specs to the machine and run the same tests on that ( including the windows tests with bootcamp. You’d see if it made a difference. I’m guessing it wouldn’t. Maybe the benchmark creator already knew that from experience. If so, he should have said that.
Looks to me like Firefox is over-optimizing for benchmarks. It takes first or second place on every platform for things like Kraken, Dromaeo, and SunSpider, but takes last or second-last place for more mundane tests like “How long does it take to load page X?” or “How long does it take to open up the browser?”
Of course, given how much emphasis they’ve placed on benchmarks, I’m surprised they came in last in the maze solver race by such an embarassing margin.
At least they added a ‘reliabilty benchmark/proper page loads’ section which shows that Chrome and IE don’t always actually load the page properly. They may be fast, but don’t load the whole page.
Knowing a bit how Firefox handles these things, I would say it very much depend on how you test.
Take the ‘eight tabs’-test, Firefox when you start it up with ‘tabs open’ will first load the page from cache and than refresh to see if something changed. Many of the other browsers don’t. I couldn’t find how they actually test that though.
I also wonder about the page-load tests if they load the same page from a proxy or something like that, what about ads ? Does it load the same ad ? What about DNS ? Or do they just run the test 25 times, clearing the cache each time and hope for the best ?
I’m just wondering because I noticed the same things and benchmarks usually don’t depend on the above mentioned things so that might explain it.
Also with the memory management tests, I think Firefox keeps closed tabs in memory so when you choose to ‘open a recently closed tab’ it will just load the rendered page from memory I think. Thus you can’t just look at the memory usage of the browser and say it doesn’t clean up the memory.
So memory management is different in that case for Firefox. You may agree or disagree with the choice the Firefox developers made, but they are testing different things (there is a setting to disable this behaviour if I remember correctly).
Interresting enough the Acid3 test is included, the parts Firefox does not implement actually didn’t make it into the final specifications. They specifically choose not to implement it because of that.
For example the SVG font test that is done is testing something that is actually broken as I understand it.
Also why not have a performance benchmark for the Acid3-test ? I wonder what the results would be. I know it isn’t a real performance test, I’m just wondering.
I hope all browser makers look at the tests and see if they can learn something from it, because all perform bad in atleast some of the tests.
You’re correct, and actually there’s also something about the maze CSS test that explains why Firefox is slow in that test. MS purposefully force a complete refresh, which no one else does of course, to slow down other browsers while they optimized IE to bypass the refresh.
Other browsers have since made that optimization too, but not Firefox, yet.
The thing is that such optimizations are nearly done ONLY for the sake of completing a single, non-real life benchmark so that the browser don’t look bad in benches.
Kinda sad if you ask me
FTA: “With only one weakness and the highest number of non-winning strong scores, Mozilla Firefox is once again our runner-up. A bit of trouble in CSS performance, as well as a high number of merely acceptable scores, are all that hold back Firefox from taking the gold. Although the new rapid development cycle may be hurting Firefox in our reliability benchmarks, it also allows Mozilla to keep pace with Google Chrome on the performance front. Firefox 7 is supposed to bring improvements to both JavaScript and memory usage/management, which are two key areas that might allow Firefox to pull into a first-place finish.”
I am expecting that the Javascript and memory management improvements that are in Firefox 7 will cure the issue with the maze solver race.
Actually, as well JavaScript and memory usage/management, Firefox 7 will introduce Mozilla’s Azure graphics to replace Cairo, and thereby considerably improve Firefox’s Canvas 2D perfomance.
http://www.phoronix.com/scan.php?page=news_item&px=OTcxNQ
http://blog.mozilla.com/joe/2011/04/26/introducing-the-azure-projec…
http://www.basschouten.com/blog1.php/comparing-performance-azure-vs…
With the three areas of considerable improvement already built in to Firefox 7 (which is now in beta), I have high expectation that Firefox will pull ahead of Chrome in a couple of months, and so claim outright first place in the next round of such testing.
Personally, I am already running (testing) Firefox 7 beta under the alpha 2 testing pre-release of Kubuntu Oneiric.
Edited 2011-08-30 01:07 UTC
I seem to remember a bug report where they tracked down the issue to the fact that Firefox was constantly reflowing the layout of the page during the test, when it didn’t really need to. (I’m not 100% sure it was that test, but i think it was) I don’t think it’s been fixed yet, although Azure might help it somewhat as well.
Edited 2011-08-30 01:59 UTC
The 2 bug reports are:
https://bugzilla.mozilla.org/show_bug.cgi?id=641340
https://bugzilla.mozilla.org/show_bug.cgi?id=641341
Actually it wont help with the maze, but it helps with everything else.
They have a bug for the maze and as i explained in the previous comment its not actually as big deal as it seems. CSS is actually fast, not just in that very rare case.
Hopefully they’ll fix it tho, so that the benches look pretty.
it has nothing to do with anything you say.
if you optimize js it doesnt change how quickly the browser starts, its COMPLETELY unrelated.
There’s been benchmarks focused on the complete time to open pages and in perceived time Firefox is actually faster than anything else.
Also 300ms slower startup time big deal much?
I stumbled upon a irc channel with web developers one day. I asked them about whats the best browser out there,, theese were pros at webpage coding, so I thought these would be able to give some good advice on the matter.
The response was: any.., any is good, just don’t use IE.
They all seemed to agree about that.
IE is becoming better I think, thanks to mozilla and the recent excellent competition. But lets continue not using IE. I don’t want to see the web crippled by MS again as it was during IE6. That were sad sad times indeed, and the IE6 monopoly of the past still affects the web today in many ways.
Well, that is easily answered, it is already happening.
1. IE9 implements the least amount of standards of all browsers: http://caniuse.com/
I know IE9 supports a lot of things people never expected a few years ago and it is a happy surprise.
But let’s say they are at least 3 years behind when just before IE9 was released. And just after they where 1 or 2 years behind. And IE release process is slow in comparison. And deployment of IE-versions is even slower. It takes at least one year for a new version of IE to gain any significant percentage on the market. IE9 in July 2011 has 7 % and IE8 still has 26 %: http://gs.statcounter.com/#browser_version-ww-monthly-201009-201107
What is really sad IE7 until June 2011 had a bigger market share than IE9.
Also see the issue below.
2. Because IE on Windows XP is just IE8 and it will stay IE8 and remain IE8 for many, many, many years to come.
Just as long as it takes for Windows XP to die.
Many people are upset about it: http://html5forxp.com/
They should install a more modern browser in their boxes instead of getting upset!
Well, it is mostly the people that have to create the websites that are upset.
Although there are obviously people that like IE and still want to use it. But also want to use the new features-of-the-web.
It is not really that much of a big deal IE8/9 not being that up2date.
Most new semantic elements can be rendered in IE6/7/8 correctly with the shiv technique.
Video and Audio can be replaced with flash … and a bit of progressive enhancement on a well designed site should mean that it looks pretty decent in most browsers.
Apparently the current trend is chrome trying to cripple the web.
there’s more and more “chrome only” websites (not even “webkit only”!) and chrome implements may “so called open standard techs” that they know others will not implement, then implement it server side since they have so many services and attempt to achieve complete web lock-in while covered by the “but we’re open and doing this for the right reasons!”
While in reality they’re no better than MS.
See SPDY, Native Client, etc.