Linked by Brooss on Tue 15th Mar 2011 23:32 UTC
Benchmarks A comment on the recent article about the Bali release of Googles WebM tools (libvpx) claimed that one of the biggest problems facing the adoption of WebM video was the slow speed of the encoder as compared to x264. This article sets out to benchmark the encoder against x264 to see if this is indeed true and if so, how significant the speed difference really is.
Thread beginning with comment 466306
To read all comments associated with this story, please click here.
Nice
by galvanash on Wed 16th Mar 2011 00:02 UTC
galvanash
Member since:
2006-01-25

Since I'm the one that made the comment prompting this comparison... I'm very glad to see Bali Shows some significant improvement on a real world encode. It's still slower than x264, but it's no longer "embarrassingly" slow it seems. I've also never seen those settings that were used for 2 pass, that is good info. Kudos to the author!

However, I'm still going to do my own tests with more realistic source material when I have the time. Using a 320x240 resolution for the source is not representative of what most users would actually encode (not even close really).

Edited 2011-03-16 00:07 UTC

Reply Score: 4

RE: Nice
by Brooss on Wed 16th Mar 2011 01:26 in reply to "Nice"
Brooss Member since:
2010-11-13

Yeah, getting good documentation on the encoder settings for vpxenc really is a bit of a problem.

240p video is very low resolution but its not unusual to see it on the web, youtube still offers it. It seemed like a reasonable starting point. I'd be interested to see how your higher resolution tests compare.

Reply Parent Score: 1

RE: Nice
by molnarcs on Wed 16th Mar 2011 06:45 in reply to "Nice"
molnarcs Member since:
2005-09-10

Remember that most users are not really home users in this case. WebM does well at low resolutions - the kind of resolutions you need for your mobile devices or youtube. Of course 320x240 is too small for that, 360p or 480p would be the most common use case for WebM.

Reply Parent Score: 2

RE[2]: Nice
by MissTJones on Thu 17th Mar 2011 15:02 in reply to "RE: Nice"
MissTJones Member since:
2010-03-25

The MSU test on the original release of VP8 found that it did much better on HD content than it did on SD/DVD content. I think they'd just spent more time tuning for that size.

I don't know if that has changed since, you'd think Google would have some interest in the lower resolutions used on Youtube, but on the other hand maybe every extra bit of quality on HD saves them more bandwidth than an extra bit of quality for a tiny video.

Reply Parent Score: 2

RE: Nice
by galvanash on Wed 16th Mar 2011 22:28 in reply to "Nice"
galvanash Member since:
2006-01-25

I still need to do some more tests to be completely fair, but just to give a taste of what I finding so far:

I used this video as source: http://media.xiph.org/video/derf/y4m/sintel_trailer_2k_480p24.y4m

This is the Sintel trailer at 854x480. Its about 52 seconds long. I used the exact same encoder binaries as the linked article, although my machine is different (Phenom II X3 720 with 4GB memory, Windows 7 64-bit).

I compared x264 "baseline/slow" to webm "good/cpu-used=0" using a variable bitrate of 1000Kbs. Since multi-threading affects webm quality adversely, and x264 is mulththreaded by default, I am choosing to force x264 to run single threaded by using --threads 1 for this comparison. This is as apples-to-apples as I think you can get for a single threaded comparison (Ill do a multithreaded comparison later). I'm reporting PSNR, but frankly I don't thing it is at all relevant at this bitrate.

x264

Commandline: x264.exe -o sintel_trailer_2k_480p24.mp4 sintel_trailer_2k_480p24.y4m --profile baseline -B 1000 --preset slow --threads 1 --psnr

Avg PSNR: 52.375
FPS: 15.04
Filesize: 5436Kb
Actual Bitrate: 851.94Kbps

webm

Commandline: vpxenc.exe -o sintel_trailer_2k_480p24.webm sintel_trailer_2k_480p24.y4m --good --cpu-used=0 --target-bitrate=1000 --end-usage=0 -p 1 --psnr

Avg PSNR: 50.862
FPS: 6.43
Filesize: 5017Kbs
Actual Bitrate: 766.710Kbps

Conclusion

Quality is very comparable subjectively, x264 is a tad better imo (but not much). When I get all of this together I will post the actual output videos so people can look at them.

Filesize is close enough since both stayed well under the target. Webm is a bit smaller, but in this case that isn't actually a good thing. Both had plenty of bitrate to work with, and both ended up not needing anywhere near all of it (I used a high target because frankly that is what people normally do at these low resolutions).

note: I have noticed that webm when run with the "--good --cpu-used=0" with a high bitrate cap tends to undershoot much more than when run using "--best" (but it is also twice as slow when run that way). I prefer using best because it tends to get much better quality (i.e. higher bitrate) when given alot of headroom, but I didn't use it here because I did not want to be accused of an unfair comparison.

The big difference, as I have stated before, is in speed. x264 is over twice as fast (and frankly it is being hobbled here since running it designed to be run multi-threaded)... I'm still in the same ballpark speedwise as the original article, but the gap has widened a wee bit (i.e. roughly 60% slower). Well see how that pans out at HD resolutions.

Reply Parent Score: 3

RE[2]: Nice
by galvanash on Wed 16th Mar 2011 23:32 in reply to "RE: Nice"
galvanash Member since:
2006-01-25

Same setup, only change is the source is 1280x720 and the bitrate is set to 2000.

x264

Commandline: x264.exe -o sintel_trailer_2k_720p24.mp4 sintel_trailer_2k_720p24.y4m --profile baseline -B 2000 --preset slow --threads 1 --psnr

Avg PSNR: 53.647
FPS: 7.23
Filesize: 10702KB
Actual Bitrate: 1678.26Kbps

webm

Commandline: vpxenc.exe -o sintel_trailer_2k_720p24.webm sintel_trailer_2k_720p24.y4m --good --cpu-used=0 --target-bitrate=2000 --end-usage=0 -p 1 --psnr

Avg PSNR: 52.067
FPS: 3.35
Filesize: 10118KB
Actual Bitrate: 1548.36Kbps

Conclusion

Encoding speed relative to x264 stays about the same (this is much better than previous versions of webm at this resolution). Quality is a mixed bag. Webm looks much better during low motion scenes (detail is higher), but suffers during high motion (smearing). Overall quality is about the same imo.

Ill do a full battery of tests offline (multi-threaded too) and put a comparison together with graphs and links to the output files. Will probably take me a few days.

Reply Parent Score: 2