
Being a BeOS user (a purely desktop system) and because I code under Linux, I see
XFree86 (v4.1 on my machine) as a user and as a developper. And this is where the problem lies. My Gnome or KDE desktops are slow in comparison with other operating systems, but XFree86, the 'engine' behind these desktops, proves me that it's not. Let's look at what I have in front of me: a dual Pentium III at 933Mhz with 512MB of memory, a Radeon 32 AIW, a modified Mandrake 8.0 powered by kernel 2.4.18.
Using the Free generic drivers that came with Mandrake 9.0 at first, I noticed a definite speed improvement when I switched to using NVidia's drivers.
Obviously this isn't a solution for this fellow, but this is to say that X performance is most definitely affected by the quality of the driver you have.
I stopped buying ATI cards a while ago becasue of shoddy drivers. I think you might be surprised with the quality difference a simple GeForce 2 (64megs RAM) and good NVidida drivers make.