I won't claim here that I know anything about encoding video, and what options yield the best possible results. All I can do is rely on the expertise of others, but in doing so, it becomes quite hard to figure out, as a layman, how all these various codecs compare to one another. Let's look at three (relatively) recent encoding benchmarks.
Jan Ozer from the Streaming Learning Center. Ozer compared the results from Theora (1.1) to H264 at various bitrates, and they show that H264 simply performs better. "These tests are very aggressive, but purposefully so - at very high data rates, all codecs look good," Ozer argues, "In particular, YouTube encodes their H.264 video at 2mbps, about 2.5X higher than my tests. So my conclusion isn't that Ogg is a bad codec; it's that producers seeking the optimal balance between data rate and quality will find H.264 superior."
Ozer readily admits that this is his first foray into Theora encoding, so he asked for input from the community as to how to yield the best results. Xiph's Greg Maxwell took up on this, and noted that several of the setting employed by Ozer weren't optimal. "The bigest reason why the 1.1 clips have surprisingly poor quality is because these are stricly buffer constrained encodes," Maxwell notes, "The buffer constraint means that user with a very bandwidth limited connection can playback without stalls and excessive buffering. So it's a good thing to have if you're streaming. But - its absolutely brutal on the quality and it's not a restriction that your h264 encode has been subjected to."
Maxwell performed his own benchmarks last year, and he drew two conclusions. First, that Theora is better than H263. No surprise there. Second, that Theora is competitive compared to H264. "In the case of the 499kbit/sec H.264 I believe that under careful comparison many people would prefer the H.264 video," Maxwell concluded, "However, the difference is not especially great. I expect that most casual users would be unlikely to express a preference or complain about quality if one was substituted for another and I've had several people perform a casual comparison of the files and express indifference."
So, we have two benchmarks going either way. Luckily, we also have a third benchmark, by Martin Fiedler, which places H264 firmly in first place. "The H.264 format proves that it’s the most powerful video compression scheme in existence," Fiedler writes, "The main competition in the web video field, Ogg Theora, is a big disappointment: I never expected it to play in the same league as x264, but even I didn't think that it would be worse than even Baseline Profile and that it's in the same league as the venerable old XviD which doesn't even have in-loop deblocking."
To further prove just how incredibly difficult this whole video encoding business is: Maxwell commented on Fiedler's benchmark, and together they found out that Fiedler was using the wrong Theora branch, which negatively affected Theora's performance. It turns out that the branching scheme for Theora isn't particularly intuitive.
All this illustrates two things: first, it takes experts to do this stuff. Video encoding is hard, with all those options and tweaks you can apply - made worse by the different encoders you can choose from. Second, that the people behind Theora need to make it clearer which version or branch produces the optimal results. Of course, this is also an inherent problem with Theora: always the moving target.
In any case, this all overlooks the big, massive advantage that Theora has over H64: it's unencumbered by patent nonsense. That alone is a massive win for Theora, and something H264 doesn't have an answer for.