Linked by Brooss on Tue 15th Mar 2011 23:32 UTC
Benchmarks A comment on the recent article about the Bali release of Googles WebM tools (libvpx) claimed that one of the biggest problems facing the adoption of WebM video was the slow speed of the encoder as compared to x264. This article sets out to benchmark the encoder against x264 to see if this is indeed true and if so, how significant the speed difference really is.
Thread beginning with comment 466387
To read all comments associated with this story, please click here.
WebM vs H.264 benchmark
by WereCatf on Wed 16th Mar 2011 11:25 UTC
WereCatf
Member since:
2006-02-15

I personally am much more interested in the quality aspect than in encoder speed. Unfortunately, there seems to be no quality benchmark anywhere that is actually fair to both parties; either it tries to pose H.264 as superior and thus uses inferior settings for WebM, or vice versa.

How I would like a quality benchmark to be done:
1) Have a 30min video at really high resolution and no lossy compression as the source, with a few steady scenes where it's important to preserve the colours and clarity as well as possible, a few scenes with lots of movement where it's important not to produce many compression artifacts, and then lastly low-light versions of all the afore-mentioned scenes so we can quantify how well the encoders manage low-contrast situations.
2) Find a H.264/x264 enthusiast who knows his stuff and tools to do the H.264 encoding, and a WebM/VP8 enthusiast to do the WebM encoding. It's not quite possible to avoid bias anyways, so why not just choose biased parties from the start but only let them work on what they prefer? This way both of them will do their best to get the best output they can.
3) Establish the conditions for the encoding tests, like for example bitrates as 150kb/s, 600kb/s and 1800kb/s and resolutions as 320x240, 640x480 and 1024x768, and then encode all the combinations above.
4) Upload all the resulting output files somewhere so people can make their own comparisons, but also clearly state file sizes, encoding times and any settings used and provide a few sample images from all the different conditions.

Then we could actually have some meaningful discussion without having to resort to "he is biased, he doesn't use the best possible settings!" arguments.

Reply Score: 3

RE: WebM vs H.264 benchmark
by kahen on Wed 16th Mar 2011 11:33 in reply to "WebM vs H.264 benchmark"
kahen Member since:
2009-09-07

5) Send the original file through a blur filter (no compression) to show how effing stupid tuning for PSNR is

Reply Parent Score: 2

RE: WebM vs H.264 benchmark
by lemur2 on Wed 16th Mar 2011 12:27 in reply to "WebM vs H.264 benchmark"
lemur2 Member since:
2007-02-17

I personally am much more interested in the quality aspect than in encoder speed. Unfortunately, there seems to be no quality benchmark anywhere that is actually fair to both parties; either it tries to pose H.264 as superior and thus uses inferior settings for WebM, or vice versa.


Here is a decent attempt at quality comparison done in May 2010:

http://compression.ru/video/codec_comparison/h264_2010/index.html

In June 2010 they produced this update which included VP8:

http://compression.ru/video/codec_comparison/h264_2010/vp8_vs_h264....

Considering only quality per bit, and ignoring encoder speed, this SSIM metric comparison puts WebM quality behind that of x264 but ahead of XviD h264. The VP8 "best" preset just overtakes the x264 high-speed preset, all other presets of x264 are progressively slightly better.

In June 2010 the only version of WebM was the launch version of libvpx, the reference codec.

Since that time there have been two subsequent releases of libvpx.

http://www.osnews.com/story/24502/WebM_Project_Releases_New_Version...

These WebM releases were announced here (Aylesbury):
http://blog.webmproject.org/2010/10/vp8-codec-sdk-aylesbury-release...
and here (Bali):
http://blog.webmproject.org/2011/03/vp8-codec-sdk-bali-released.htm...

http://www.osnews.com/permalink?465497

Bali: "Best" mode average quality improved 6.1% over Aylesbury using the SSIM metric.
Aylesbury: "Best" mode average quality improved 6.3% over launch release using the SSIM metric.

Total improvement from launch release to Bali release = 1.061 * 1.063 = 1.127843. I rounded this out to 12.8%.


The most recent release of the libvpx reference code for WebM is 12.8% better quality than the version shown in the graphs plotted in June 2010 at compression.ru.

Hope this helps.

Edited 2011-03-16 12:34 UTC

Reply Parent Score: 2

RE[2]: WebM vs H.264 benchmark
by WereCatf on Wed 16th Mar 2011 12:42 in reply to "RE: WebM vs H.264 benchmark"
WereCatf Member since:
2006-02-15

The most recent release of the libvpx reference code for WebM is 12.8% better quality than the version shown in the graphs plotted in June 2010 at compression.ru.

Hope this helps.


No, it doesn't. Percentages do not say anything about how it actually looks like to the eye and thus referring to them as some sort of a holy bible doesn't really tell me much.

Reply Parent Score: 2