First things first: the disclaimers.
1. This is not a scientific test. Both sides will be arguing about this arcane switch, and the other arcane switch, and "oh, you should have used another encoder", this or that. I don't wanna hear that. I use the most popular and free front-ends for these encoders: ffmpeg with x264/faac and ffmpeg2theora. FFmpeg is a command line front-end to many encoders and itself is used by many GUI front-ends too, so it's one of the best utilities for the job. Ffmpeg2theora on the other hand, is the most popular Theora front-end too. I didn't use Ffmpeg for Theora because that implementation is still buggy. FFmpeg2theora does a better job.
2. Tests were done on Windows XP 32bit because the Ubuntu version of ffmpeg does not bundle AAC for legal reasons. I used a May 2007 version of Ffmpeg (the latest version crashes repeatedly on Windows when libx264 is used on HT Intel machines), and for ffmpeg2theora I used the latest version, 0.20.
3. I used an HDV .m2t file as source file. My first idea was to use the lossless intermediate Huffyuv format, but it would be problematic trying to upload this to the osnews server, because the file size would be huge. So, I went with .m2t, I put two clips side by side on Sony Vegas Pro 8, and then I exported again as .m2t 1080/60i (interlaced). Vegas does not re-encode clips that were not edited, so this is as close to "straight out of the camera" as possible (only a few frames at the beginning and the end of each clip were re-encoded by Vegas). Download the source clip here (unzip it).
I did two tests: one for a widescreen QVGA format, and one for a 720p HD one. I chose these two sizes on purpose. One format will show us how the codecs perform in low bitrate, which is super-important for web usage, and the other one will show us how well the codecs perform with HD.
FFmpeg command line:
ffmpeg -deinterlace -y -i "orignal.m2t" -threads 2 -f mp4 -title "h.264 QVGA test" -vcodec h264 -level 41 -refs 2 -loop 1 -deblockalpha 0 -deblockbeta 0 -parti4x4 1 -partp8x8 1 -partb8x8 1 -me full -coder ac -subq 6 -brdo 1 -me_range 21 -s 320x180 -r 30000/1001 -b 320k -maxrate 450k -bufsize 5000k -g 300 -acodec aac -ac 2 -ab 128k "qvga.mp4"
Ffmpeg2theora command line:
ffmpeg2theora --optimize --deinterlace --title "Theora QVGA test" -f mpg -v 4.2 -F 30 -x 320 -y 176 -A 128 -c 2 -H 44100 -o "qvga.ogg" "orignal.m2t"
Two things of note here: Ffmpeg2theora would not let me use the resolution that keeps the correct aspect ratio when downscaled from 1920x1080 (anamorphic 1.3333). I had to use 320x176 (direct multiple of 8), instead of the correct 320x180. FFmpeg/x264 had no problem with that.
The second thing is something that really made me angry: Ffmpeg2theora will upsample the NTSC frame rate of 29.97 to 30fps. Well, I don't want 30fps, I want 29.97. I am accustomed to view lots of my videos on my TV (usually by using a PS3), so should a Theora2TV device comes out, I want to be able to have smooth frame rate, without adding pulldown and all the problems that brings. No matter what trick I tried to do, Theora would keep adding pulldown to 30fps. Update: You can get to 29.97 only by either not specifying it at all, or via 30000:1001 (instead of 30000/1001 or 29.97).
Now, you probably noticed that I gave a "videoquality" of "4.2" to Theora, while I gave FFmpeg 320kbps video with a 450kbps peak rate (VBR). I did this because Theora would simply NOT do what I asked it to. It would not stay close to the requested 320kbps, and so it would create a 550kb file instead of a 640kb one. So I changed the Theora command line, by trying different VBR quality numbers, until I got an output file size that matched that of h.264's. Remember, the important thing here to remember is THE WEB. Bandwidth matters. So it's important to know what kind of quality you get from each codec, for the same file size (not necessarily the same bitrate).
As you can see from the tests, h.264 whooped Theora pretty bad on this test. Even when I gave the benefit of the doubt to Theora and re-encoded it in 400kbps, h.264 was still visibly much better.
FFmpeg command line:
ffmpeg -deinterlace -y -i "orignal.m2t" -threads 2 -f mp4 -title "h.264 720p test" -vcodec h264 -level 41 -refs 2 -loop 1 -deblockalpha 0 -deblockbeta 0 -parti4x4 1 -partp8x8 1 -partb8x8 1 -me full -coder ac -subq 6 -brdo 1 -me_range 21 -s 1280x720 -r 30000/1001 -b 5000k -maxrate 6000k -bufsize 15000k -g 300 -acodec aac -ac 2 -ab 128k "720p.mp4"
Ffmpeg2theora command line:
ffmpeg2theora --optimize --deinterlace --title "Theora 720p test" -f mpg -v 4.3 -F 30 -x 1280 -y 720 -A 128 -c 2 -H 44100 -o "720p-theora.ogg" "orignal.m2t"
Again, a quality number was used for Theora to match h.264's file size at the requested 5 Mbps (with 6mbps peak rate). I did two ffmpeg passes, one with the CABAC entropy coding, and one with the CAVLC one (just change the "-coder ac" to "-coder vlc" in the ffmpeg code above). CABAC creates better looking video than CAVLC, but it's harder to decode (Quicktime chokes good on it, for example, on my P4 3Ghz). I usually encode my stuff in CAVLC simply because I might find myself sharing my own videos to people with AppleTVs or slower PCs, so I am trying to play nice.
HD is HD, so even if Theora again performs worse than h.264, there is enough resolution that the differences might not as obvious as in the QVGA example. For example, I have an old CRT HDTV, which barely manages well 1080i, so the difference would not be much visible (heck, on my TV I barely see the difference between Comcast's HD channels and progressive DVDs). However, people with big LCDs or Plasmas will be able to see more artifacts -- should a device that playbacks Theora on TVs comes to market.
Another interesting tidbit about the 720p test is the encoding times! H.264 did 1'.15" for CAVLC and 1'.20" for CABAC on my PC. Theora took its sweet time and finished at 3'.14". Granted, I used two threads on the FFmpeg command line, while it was obvious to me by monitoring my Hyperthreaded CPU that FFmpeg2theora was single-threaded. But even if you divide the 3'.14" by 2, you get 1'.37", which is still slower than h.264's encoding times.
Now, a last point, CPU utilization during decoding. Of course, this is implementation-specific, but you know, you gotta test with whatever is popular, no matter if their implementation sucks or not. So I used VLC and a new version of Media Player Classic. VLC used on average 65% of my hyperthraded CPU on CAVLC, 75% on CABAC and 75% on Theora. MPC used 34% on CAVLC, 43% on CABAC and 33% on Theora. These were rough averages during multiple times of running each video.
I prefer h.264. Goodnight everyone.
Ok, ok, not yet goodnight. Truth is, I prefer open standards and open source and Free Software and what not, over proprietary royalty/patent-ridden solutions. But pictures (and videos) don't lie. Theora is simply not at the same league as h.264. Vorbis fairs well compared to the big 3 audio formats, but Theora is visibly worse in comparison. By their own admission in the past, the Theora developers had to use less efficient algorithms to go around patents. This sucks of course, but it all comes down to this: do you want an open FOSS solution, or you want the best solution? Many readers on OSNews will choose the first option, but their teenager brother would probably choose the second.
Some say that Dirac is better, and it's free/open. But until their developers actually do take the time and commit their implementation to ffmpeg (instead of only offering patchsets), it just doesn't exist for me. FFmpeg -- with all its problems... Come on, no automatic letterboxing??-- is used by many GUI front-ends, so if they want market penetration, they gotta add their encoder to ffmpeg. Then, I will give them a whirl too.