Linked by Brooss on Wed 23rd Mar 2011 23:14 UTC
Benchmarks A new set of x264 and vpxenc encoder benchmarks have been published. The new benchmarks address many of the concerns raised in the comments about the methodology used in the previous article, such as using SSIM for quality measurement. Theora is also included in these tests.
Thread beginning with comment 467716
To read all comments associated with this story, please click here.
Would love to see a few more graphs
by gerg on Thu 24th Mar 2011 15:49 UTC
gerg
Member since:
2011-03-16

SSIM vs bitrate overage

Also, since some seem to have trouble staying at the designated bitrate, shouldn't encoding bitrates be adjusted to stay at the requested criteria. Doesn't this mean x264 should be encoding at a lower bit rate? Likewise, doesn't this mean Theora should be encoding to a much higher bitrate, especially given your statement, "The Theora encoder considers container file overhead when encoding, possibly explaining its lower bitrate." Which appears to mean Theora is artificially punished on the SSIM evaluation because its pro-active efforts to maintain the requested bitrate.

It appears x264-high consistently exceeds the designated bitrate which would have the affect of artificially driving up its SSIM rating. It really seems its still not an apples to apples comparison until all are actually encoding at the designated bitrate; else those that do honor the requested bitrate are effectively punished. Seems like a cleaver way for x264 to consistently obtain an undue advantage in comparisons such as these.

Reply Score: 1

Brooss Member since:
2010-11-13

I should point out that the container format overhead is included in the bitrate charts so a small margin over the target is expected for vpxenc and x264 (and Theora should be expected to hit it exactly). Container overhead should be in the rage of around 0.5%


Also, since some seem to have trouble staying at the designated bitrate, shouldn't encoding bitrates be adjusted to stay at the requested criteria.

All the encoders do a good job of matching the requested bitrate in 2 pass mode, all bitrates are within around 1% of each other. IMO not significant enough to worry about.

The differences are a little more significant in 1 pass mode indicating that the problem isn't just Theora considering container overhead but rather Theora is just doing a worse job of matching the requested bitrate. I consider this an encoder decision and passing Theora a larger bitrate to compensate would be be unfair.

But again the differences here are still very small, getting a comparison with all encoders hitting the exact bitrate in 1 pass mode would be quite difficult.

Reply Parent Score: 2