Linked by Brooss on Tue 15th Mar 2011 23:32 UTC
Benchmarks A comment on the recent article about the Bali release of Googles WebM tools (libvpx) claimed that one of the biggest problems facing the adoption of WebM video was the slow speed of the encoder as compared to x264. This article sets out to benchmark the encoder against x264 to see if this is indeed true and if so, how significant the speed difference really is.
Thread beginning with comment 466321
To read all comments associated with this story, please click here.
Things we already knew about
by Eugenia on Wed 16th Mar 2011 01:59 UTC
Eugenia
Member since:
2005-06-28

From his conclusion:

>comparisons in quality are always somewhat subjective

No, they're not. Just use a pixel difference utility and do it objectively. For my tests I use this: http://pdiff.sourceforge.net/

>encoding speed to quality ratio with x264 ‘baseline’ at the higher quality settings

So basically he's telling us that you have to push the encoder (and your CPU) extra, to just get "h.264 baseline" quality out of it. This is not competitive.

>the gap never exceeds around 50% encoding time

Which is huge for a service like Vimeo or Youtube, that get thousands of encoding requests per minute. Time is money, and this would be a huge waste in resources.

>expect vpxenc to take around twice as long to encode a video of comparable quality to a x264 ‘high’

Yup. Not good good enough. Not for video services, not for mobile applications that might or might not have a DSP for webm, and definitely not good for professionals. Remember, it's the professionals who upload video that the majority of people watch. Webm is only good for people who don't mind the extra time, as long as they can do it for free. For example, a screencast of someone in his bedroom showing the new Gnome3, and uploading on his own blog bypassing youtube. These are people who do niche video that don't matter in the grand scheme of things.

WebM won't make it, in my personal, and professional opinion as a videographer. I wish it did, because I hate MPEG-LA and all they stand for, but there is no h.264 killer out there. Except h.265 that is, that comes out in 2013.

Of course, just like in the past, for my Ogv vs h.264 article, people will say that quality/speed don't matter as long as it's Free, but that's not true at all. Professionals in my field don't like h.264 for a variety of reasons, but they like webm even less.

Edited 2011-03-16 02:02 UTC

Reply Score: 5

woegjiub Member since:
2008-11-25

"Not good good enough."... yet ;)
With a group like x264 working on it, it should improve significantly, given the massive gap between the h.264 and x.264 encoders.

New replaces old, as services evolve.
Hopefully with time, it will replace x.264 as it becomes better optimised by the FOSS community.

That, or software and medicine copyright is over-ruled :p

Reply Parent Score: 2

RE: Things we already knew about
by smitty on Wed 16th Mar 2011 03:37 in reply to "Things we already knew about"
smitty Member since:
2005-10-13

From his conclusion:

>comparisons in quality are always somewhat subjective

No, they're not. Just use a pixel difference utility and do it objectively. For my tests I use this: http://pdiff.sourceforge.net/

Except that certain pixels are often more important to scene quality than others. It is subjective, even if you can quantify parts of it.

Yup. Not good good enough. Not for video services,

Note that Youtube is already encoding everything in VP8, so apparently it is good enough for them.

Clearly, it has a long way to go before it catches up with x264. I don't think anyone has ever claimed otherwise. Luckily, they are continuing to release updates every 3 months and the gap will continue to close. The improvements from the original release are already significant. Also remember that all these tests tend to be against x264, which is the best in class h.264 encoder. It actually blows away a lot of the commercial encoders as well, and plenty of companies are making lots of money off them without being told their codecs are useless.

Edited 2011-03-16 03:40 UTC

Reply Parent Score: 5

Eugenia Member since:
2005-06-28

>Note that Youtube is already encoding everything in VP8, so apparently it is good enough for them.

Youtube's quality sucks. It's visibly soft on my HDTV (watching its HD videos via my Roku). Vimeo is way better, about 25% better. And youtube doesn't use webm for normal usage, it's only used as a last resort, if h.264/flash is not found.

Reply Parent Score: 2

TechGeek Member since:
2006-01-14

How much difference is there between a PSNR of 39 and 45? Numerically that's only a 12% difference, at most.

There is also the issue that there is only a 4% difference between the best x264 settings and the baseline. Saying that WebM can only keep up with the x264 baseline really isn't saying anything bad.

The only thing this data proves is that WebM NEEDS a multi threaded encoding machine. Now do you really think anyone doing anything serious with video is going to try to encode it on a single core computer? If we're really gonna talk about Google and other video sites, understand that they will have a cluster on the back end. The cost of the license to use H.264 probably far outweighs the cost of the extra hardware it *might* take to produce video of the same quality in the same time frame. Hardware is cheap these days.

There is also NO data here on decoding, so you really cant speculate from this article about how they will perform on a mobile platform.

I am sick and tired of hearing people declaring that professionals are never going to support WebM. Google owns the largest video site and they choose WebM. They obviously can make it work.

Reply Parent Score: 5

galvanash Member since:
2006-01-25

How much difference is there between a PSNR of 39 and 45? Numerically that's only a 12% difference, at most.


It doesn't matter to me, I am firmly in the "PSNR is meaningless" camp and just ignore PSNR scores. It is somewhat useful for selecting a optimal bitrate constraint for particular source material, but for comparing quality across codecs it just doesn't mean anything.

There is also the issue that there is only a 4% difference between the best x264 settings and the baseline. Saying that WebM can only keep up with the x264 baseline really isn't saying anything bad.


I am a webm supporter, but a spade is a spade and that statement isn't fair to x264. A 4% difference when you are using a low resolution source with a highly constrained bitrate is meaningless... Try the same video at 720p with a reasonable bitrate (say 1500kbps) and that 4% will turn into upwards of 10-20% real quick. x264 main and high profile produce much better quality than baseline at high resolutions with comparable bitrates. That is frankly what those profiles are for (i.e. trade higher decode complexity for increased quality at high resolutions).

You can't take the 4% swing on a 320x240 source file at such a low bitrate and form a conclusion like that from it.

The only thing this data proves is that WebM NEEDS a multi threaded encoding machine.


You are reading too much into that. It needs multi-threading to get within the "twice as slow" range compared to x264, but x264 is multi-threaded by design and by default - webm's multi-threading support is rather poor and hurts quality rather significantly (unfortunately). x264 quality is virtually unaffected by multi-threading. In reality webm needs "better" multi-threading support, using threads to make it faster as things are now is something of a crutch that doesn't work all that great (but granted it is better than nothing).

I am sick and tired of hearing people declaring that professionals are never going to support WebM. Google owns the largest video site and they choose WebM. They obviously can make it work.


Google is a content distributer. They are not video professionals by any stretch of the imagination. When people like me say "professionals will never support webm" we are talking about content producers, not distributors. And I still don't think professional content producers will ever embrace it, but I also don't think it actually matters much either. It is a fine distribution format, at least as good as x264 baseline and getting progressively better with each release. Who cares if content producers use it? It doesn't much matter - any video is just a re-encode away from being free from the grips of MPEG-LA...

Granted, Eugenia lumps content producers in with distributors (even online distributors), but that is her bias since she works in the industry. I frankly don't see it that way at all.

Edited 2011-03-17 00:17 UTC

Reply Parent Score: 3

RE: Things we already knew about
by lemur2 on Wed 16th Mar 2011 04:23 in reply to "Things we already knew about"
lemur2 Member since:
2007-02-17

From his conclusion:

>comparisons in quality are always somewhat subjective

No, they're not. Just use a pixel difference utility and do it objectively. For my tests I use this: http://pdiff.sourceforge.net/

>encoding speed to quality ratio with x264 ‘baseline’ at the higher quality settings

So basically he's telling us that you have to push the encoder (and your CPU) extra, to just get "h.264 baseline" quality out of it. This is not competitive.

>the gap never exceeds around 50% encoding time

Which is huge for a service like Vimeo or Youtube, that get thousands of encoding requests per minute. Time is money, and this would be a huge waste in resources.

>expect vpxenc to take around twice as long to encode a video of comparable quality to a x264 ‘high’

Yup. Not good good enough. Not for video services, not for mobile applications that might or might not have a DSP for webm, and definitely not good for professionals. Remember, it's the professionals who upload video that the majority of people watch. Webm is only good for people who don't mind the extra time, as long as they can do it for free. For example, a screencast of someone in his bedroom showing the new Gnome3, and uploading on his own blog bypassing youtube. These are people who do niche video that don't matter in the grand scheme of things.

WebM won't make it, in my personal, and professional opinion as a videographer. I wish it did, because I hate MPEG-LA and all they stand for, but there is no h.264 killer out there. Except h.265 that is, that comes out in 2013.

Of course, just like in the past, for my Ogv vs h.264 article, people will say that quality/speed don't matter as long as it's Free, but that's not true at all. Professionals in my field don't like h.264 for a variety of reasons, but they like webm even less.


Most of the optimal methods for video are patented by other parties, so WebM must use sub-optimal methods. There has to be a compromise made somewhere, and WebM has made this compromise in the encoding speed.

WebM has probably surpassed H264 now in quality/bit, because it was only just behind h264 by that metric at launch, and WebM has improved quality/bit by 12.8% over its two releases since then.

However, only in the last release was any attempt made at improving the quality/encoding time.

http://blog.webmproject.org/2011/03/vp8-codec-sdk-bali-released.htm...

The recent Bali release improved encoder speed significantly, but it is still not close to x264 on this metric (and this metric alone).

Perhaps the next (Cayuga) release will make up more ground by Q2, 2011:
http://blog.webmproject.org/2011/03/next-up-libvpx-cayuga.html

We will continue to focus on encoder speed in Cayuga. Though our Bali encoder is up to 4.5x faster than the initial VP8 release (at "Best" quality mode), there are more speed improvements to be had. As always, we'll continue to improve video quality in the encoder.


However, it must be said that encoder speed is the area where webM does make a (necessary) compromise, and it may never catch up with x264 (for this metric alone, since webM has already caught up in terms of quality/bit).

It is perfectly reasonable for this to be the area in which a compromise is made, because for every time a digital video is encoded it is played on average many thousands of times, and likewise for every person who encodes digital videos there are many thousands who merely view them.

For professional and high-volume use, perhaps the solution to the encoding speed problem will be found in a hardware encoder:

http://blog.webmproject.org/2011/03/introducing-anthill-first-vp8-h...

AFAIK there is also a project underway to implement WebM encoding via contemporary GPUs (shaders) using the OpenCL language, which if completed would also become a means (usable by anyone with a 3D-capable GPU) to considerably speed up the WebM encoding time.

Edited 2011-03-16 04:38 UTC

Reply Parent Score: 6

Timmmm Member since:
2006-07-25

> WebM has probably surpassed H264 now in quality/bit

Rubbish. Show me an example.

Reply Parent Score: 1

RE: Things we already knew about
by Alfman on Wed 16th Mar 2011 04:33 in reply to "Things we already knew about"
Alfman Member since:
2011-01-28

Eugenia,
">comparisons in quality are always somewhat subjective"

"No, they're not. Just use a pixel difference utility and do it objectively."

This test is highly appealing do to it's simplicity, however it is a little deceptive if we dig deeper.

Lossy compression algorithms deliberately throw away information we cannot perceive. This is by design, and should not be considered a fault. Creating an objective quality tests turns out to be rather difficult.

Using audio as an example, consider two cases:
Audio codec A throws away very low and high frequencies which we cannot hear.
Audio codec B replicates the waveform best on paper, but introduces other audible artifacts.

The diff test would prefer codec B, but the human ear would prefer codec A.

The nuances are even more complicated with visual media.

Consider a black screen on a white surface. A motion estimation algorithm might place the "lines" a couple pixels out of place, but never the less produce a great visual image overall. The trivial diff test would fail to recognize the quality a human observer would perceive.


">the gap never exceeds around 50% encoding time"

"Which is huge for a service like Vimeo or Youtube, that get thousands of encoding requests per minute. Time is money, and this would be a huge waste in resources."

I'm not particularly impressed by those numbers either, but we need to recognize that encoding performance may be less crucial than decoding performance and size ratios. The importance of either depend on the use cases.

You mentioned mobile applications, Today's devices have more computing power than bandwidth, so it could make sense to sacrifice one to help the other.

The other example you mentioned was youtube. If one is distributing millions of copies, then encoding speed isn't nearly as important as the decoding speed and compression ratio.

Just to emphasis the point... What if we had an algorithm capable of compressing a typical DVD to 50MB with no perceivable quality loss but it took 48hours to encode, but also played back on modest hardware? This would be of great interest to youtube, netflix, etc. The savings on bandwidth * millons would offset any inconvenience to the service providers.

I'm not familiar enough with these codecs to make any judgements, but I do believe it's likely the battle will be fought on political rather than technical grounds, assuming both codecs are "good enough".

Reply Parent Score: 11

Neolander Member since:
2010-03-08

Confusing encoding and decoding requirements ? Pretty bad for a professional videographer.

People don't encode videos on mobile devices, and decoding WebM has less overhead than decoding H.264. Try again.

Also, I think you badly overrate the impact of professional video and quality on the web. Video on the web is mostly Youtube and Dailymotion, not carefully polished professional video.

Edited 2011-03-16 06:23 UTC

Reply Parent Score: 4

konfoo Member since:
2006-01-02

People don't encode videos on mobile devices, and decoding WebM has less overhead than decoding H.264. Try again.


But yes, they do. 99% of SoC chipsets on mobile devices these days are capable of encode and decode. Any mobile device recording video is doing encoding. The abundance of 1st and 3rd party on-device (iMovie for iPhone, etc.) software speaks for itself as to consumer adoption.

As for WebM, repeat after me: WebM SoC/DSP acceleration is not yet widely available. Right now it is completely irrelevant to even attempt to assert that CPU usage is less than H264 when H264 devices are using SoCs/DSPs with no battery drain.

Last, we can fight about this all day but the only real thing that counts is that consumers don't care as long as their content can be recorded and viewed without problems. WebM has a long way to go.

Reply Parent Score: 0

Not2Sure Member since:
2009-12-07

You don't know what you're talking about in the mobile space. As usual.

You think the cameras present on nearly EVERY smartphone are storing raw video?

The Nokia N8 for example which is probably still the best quality "cameraphone" encodes HD via H.264 with AAC for the stereo audio. Video encodes at around 9mbits/sec with stereo audio encoded at 128 kbits/sec with 48kHz sampling.

Reply Parent Score: 0

WereCatf Member since:
2006-02-15

WebM won't make it, in my personal, and professional opinion as a videographer. I wish it did, because I hate MPEG-LA and all they stand for, but there is no h.264 killer out there. Except h.265 that is, that comes out in 2013.


I don't really understand you. You're saying that because the currently existing WebM encoders are slow that they'll never be able to be fast?

Gee, remember when H.264 came out and it was still new? All the tools were abysmal and it took years to encode something. Well, lookie here, we have the exact same situation: a young, new codec and abysmal tools. So why is WebM treated differently?

I'm just saying that YES, the encoders are still slow, but for f*ck's sake look at their age! Drawing the conclusion that because it's slow NOW it can't ever be fast is just plain short-sighted ignorance, nothing else.

Reply Parent Score: 6

lemur2 Member since:
2007-02-17

"WebM won't make it, in my personal, and professional opinion as a videographer. I wish it did, because I hate MPEG-LA and all they stand for, but there is no h.264 killer out there. Except h.265 that is, that comes out in 2013.


I don't really understand you. You're saying that because the currently existing WebM encoders are slow that they'll never be able to be fast?

Gee, remember when H.264 came out and it was still new? All the tools were abysmal and it took years to encode something. Well, lookie here, we have the exact same situation: a young, new codec and abysmal tools. So why is WebM treated differently?

I'm just saying that YES, the encoders are still slow, but for f*ck's sake look at their age! Drawing the conclusion that because it's slow NOW it can't ever be fast is just plain short-sighted ignorance, nothing else.
"

Especially considering the very first software upgrade which was aimed at improving the decoder speed achieved a factor of 4.5x improvement.

http://blog.webmproject.org/2011/03/vp8-codec-sdk-bali-released.htm...

Also consider that the next release, which aims for another similar improvement factor again, is due in about three months time.

Edited 2011-03-16 11:19 UTC

Reply Parent Score: 4

RE: Things we already knew about
by gerg on Wed 16th Mar 2011 17:51 in reply to "Things we already knew about"
gerg Member since:
2011-03-16

I know this article is about encoding, but ultimately, it doesn't matter.

The professionals here are getting far too caught up in what's important to them, as a professional, as compared to what's really truly important, which is decoding.

You see, encoding only matters to a small number of people who create video and really, an even smaller number of people who do the transcoding (vimo, youtube, etc). For encoding, its the later that even matters. And really, once encoding performance becomes good enough, anything else is gravy. So really the question is, has WebM encoding performance become good enough. Everyone that matters seems to be saying absolutely.

Beyond encoding, decoding is ultimately what matters. Streams can be decoded millions of times. Here, it appears WebM has a lead in decoding performance. Furthermore, hardware is now becoming available with hardware assist. For one of largest emerging markets, this means WebM is most definitely a market winner.

Given that WebM has achieve visual parity with X.264 and is beating X.264 on decoding (which translates into improved battery life), WebM most definitely is providing serious competition.

As a professional, it may hurt your feelings that your take on it doesn't really matter, but reality is, your technical analysis of why X.264 is king is nothing but noise in the market. The reality is, where it matters, WebM is already adjusting the entire market to accommodate it; and its just barely out of the gate.

Reply Parent Score: 5

galvanash Member since:
2006-01-25

No, they're not. Just use a pixel difference utility and do it objectively. For my tests I use this: http://pdiff.sourceforge.net/


That is a great utility, but it has exactly the same problem as using PSNR for this kind of stuff... The fact that it uses perceptual biases instead of strict signal to noise doesn't matter much. The problem is both a utility like the one you use and PSNR is meant for comparing an input to an output. One measures SNR, the other perceptual differences in pixels. In either case you get a relative difference to the input, i.e. a "how much has changed" value.

Both are a fair facsimile of human vision, and arguably pdiff is a better one. Either metric tends to work decently at low compression ratios, but neither do a good job for comparing "subjective" video quality when you start dealing with high compression ratios where the output is often significantly different from the input.

The point is that being as close as possible to the input is not the goal at high compression ratios, it is more important to have the output look clean and natural, contain minimal artifacts, not lose too much "subjective" detail, etc.

Higher compression ratios (ratios that you as a video professional would outright reject because they adversely affect quality too much) are the norm for web video... PSNR and pdiff metrics for such video are mostly useless as you can easily find examples where the relative scores do not at all match most peoples subjective opinions.

Edited 2011-03-16 20:09 UTC

Reply Parent Score: 3