Linux on the AMD Opteron: Are We Ready?

Early last week I received an AMD Opteron 240 and an Asus SK8N motherboard. I was so anxious to get Linux on it I could hardly sit still… A week later, Linux is on it, in 32 bit mode only, and my hard drive has informed me that if I reinstall again it is going to go on strike.

I will first describe the hardware in general terms, since it has been reviewed in detail at various locations on the net, and then I will go into
the good, the bad and the ugly of running Linux for the AMD64, aka Hammer, aka Opteron.


The AMD Opteron chip is a marvel of silicon technology no doubt. The first
thing we note is that unlike the AMD XP series, the die is concealed in
a metal casing which covers almost the entire package. Unlike the previous
CPU’s, there is no “notch in the pin layout” on the bottom, but a tiny
notch in the package itself with an even smaller arrow pointing to the
pin that corresponds to the corner of the ZIP (Zero Insertion Force) socket
for proper alignment. There are a few pins missing on the bottom in a non
ambiguous way, but obviously forcing it would be hazardous to its health.
The heatsink is (relatively speaking) gigantic and heavy. The model I received
was an Opteron 240, non-oem (e.g. the “Processor in a box” according to AMD)
and therefore the heatsink and fan were included, along with a nice full
color instruction set on how to install the mounting bracket (it was already
installed on the Asus), CPU and heatsink. The heatsink has a fairly standard
mounting clip, with the addition of a locking cam so that it can not
come off, and the installer is pretty much certain it is in once the cam
is locked. The heatsink-cpu combo is very heavy. so heavy in fact that the
mounting bracket (in my opinion) really serves to keep the motherboard
from flexing or breaking under the weight of the heatsink. The heatsink
itself seems of good construction, with Al fins and a Cu bottom plate.
The fan is not as loud as I expected also.


The motherboard has been reviewed on the net, so only brief points are noted here:
Seems like any other motherboard (generally), of interesting note however is the
very small heatsink on the Nforce3 chipset. It does not get very hot in
operation also. Layout is nice, with the power supply connectors out
of the way. Electrolytic capacitors and small heavy gauge wirewound iron
core inductors abound on this motherboard, indicative of the power
requirements and low noise in the voltage rails required by the Opteron.
The only other point of interest is a funny locking clip on the AGP slot,
which was a real pain to close as my video card extended well past the
end of the AGP slot. I ignored this as long as all the pins on the edge
of the card were in the APG slot.


As stated in the into, I will recount my good, bad and ugly experiences with
this setup, first however, a little background on what else I am using
and my previous bleeding edge experience. Also, I will go in reverse order,
presenting the ugly, bad and then good, and final comments.


My AMD Opteron/Asus rig: This motherboard-cpu combo replaced my
old ABIT-AMD XP (1.6Ghz) setup in my main machine. The case is a
Lian-Li PC60 case, Enermax 360watt power supply, Nvidia GeForce4-TI
(by MSI), dual 3Com 3c509 boomerang nics, old Adaptec SCSI-UW card,
Yamaha SCSI CD-RW drive, and two Maxtor 20gig 7200 rpm drives. (I am
a little behind the curve in storage, having replaced my 4 2.1 gig
SCSI UW drives a while back). There is also a Lite-On IDE CD-ROM drive
for installing software at high speed. All fans in the case are active,
and the system runs cool and was rock solid (with the ABIT setup).


I am not a newcomer to the 64 bit world, and I have in fact been
running old Sun Ultra machines with Solaris, and also an IBM RS/6000
machine. I have also installed Linux on the Sun machine(s) in the past.
As far as bleeding edge goes, I have not been on it in some time,
and the last I can recall was getting an AMD K6 right when they were
released, having been stuck on a 486/25 for some time. I am not new
to kernel hacking, nor am I new to having to compile and modify drivers
and software to get it to work under Linux. That experience still
did not prepare me for the little things that could not easily be
circumvented in this situation.


The ugly:
Having built my own systems since as long as I can recall (getting
a kit in for my first computer and breaking out the soldering iron),
I am not new to system building, nor to receiving broken parts. That said,
the ASUS arrived DOA. Ok, I can deal with that, I have received a bad
motherboard from almost every manufacturer in the past, no biggie. What
was a biggie was that the vendor was back-ordered for weeks, as I apparently
received the second to last one in stock. I ordered another one from
a different vendor, and it arrived two days later. It failed to work
also, but this time the motherboard was not DOA, but giving me a RAM
error beep code. Strange, I purchased 1 GB of Crucial memory explicitly
for this motherboard. (PC2100, DDR, ECC, Registered) I put the ram in
the Abit and it worked perfectly. I then noted that I had PC2100, ECC,
Registered ram in another machine (two 256 MB sticks from Crucial also)
and they worked in the ASUS SK8N. Strange, a call to Crucial and the tech
said the same thing “Strange, that should work Sir, especially if the
others (same model it turns out) work”. A friend of mine claims he had
a similar experience with ASUS in that they did not play well with 512 MB
sticks. I also upgraded the BIOS with the 256 MB sticks in it, but that
proved useless also, with the exception that I am running the latest
BIOS now. Ah well, I want to play with it, so I move on.


The BIOS is interesting, as it has some nice built in over clocking
features (which I was not about to play with on my new $240 CPU), and
another interesting note is the lack of boot device order options.
I merely state this as it has less than my other machines (e.g.
boot order is: CD, Floppy, HD0, HD1, etc). Other than that, it seems
pretty standard.


First choice of the matter, which Linux distribution do I put on my
shiny new system? I have heard that Suse, Mandrake, Redhat (beta),
and Gentoo have ports. I was running RedHat and also have run Gentoo
on my previous ABIT so I grab the Gentoo CD while I have RedHat
beta (Taroon-AMD64) downloading on another machine.


The bad:
The Gentoo install was fairly painless at first, and there was no
appreciable difference. The initial dmesg output showed an Opteron
240 and 1MB cache (what a good feeling). I will note, however, that
something seemed sluggish. It seemed like transferring files or writing
to the drive was slow. After the stage3 tar-ball was on the drive (see
http://www.gentoo.org under installation for more details on stages),
I tried to download a kernel to build, but this resulted in it telling
me the image was masked. Apparently the KEYWORD option in the make.conf
file was set to the default and it did not match certain kernels.
In fact, a lot of packages in the portage tree were not ported yet,
and therefore did not (by default) want to emerge with the amd64 keyword.
I quickly downloaded the vanilla-source tree and was on my way.
The rest of the install went fairly smooth (after I found out about
the emerge issue) and I was then rebooting, anxiously awaiting my 64 bit
bliss. Upon rebooting it crashed and burned. It would not boot at all
and was kernel panic’ing on me. Grabbing the Gentoo CD and going
through the install again, I hopped on the #gentoo channel on IRC from
the second terminal after emerging an IRC client (now that is nice,
getting support _while_ you are installing on the machine you are
installing on). It just so happens that a Gentoo developer was online
who was playing with the 64 bit system also. They informed me of a few
important notes which I share here:


1) only the 2.4.21 kernels have success, not the newer ones under gentoo.
2) I can forget the Nvidia Geforce4, or any other Nvidia chipset video card.
The Nvidia driver from the Nvidia site fails miserably, as I found out personally later on.
3) drive access is poor, since it defaults to PIO. The Gentoo developer did
provide me with a patch for the amd7xx source code which fixed this issue.

Ok, now a recompile of the kernel, install it and reboot. Gentoo comes up and
64 bit is go! This great, with the exception of two small notes: a lot of
packages that I wish to install do not work, or crash and burn during compilation
. This could be a lack of knowledge on my part, but it was a little disturbing
all the same. The other real point of note is this: I need to work in X due to
the project I have to complete in graduate school and X will not work on this
box. I did not have any other video card to try other than a TNT, which will put me in the same boat. Even pulling down the AMD64 drivers from Nvidia did not work (as the developer said they would not). One person claimed to have it working, in 2d with no GL, etc, under the Geforce, but they also stipulated that no window manager would work other than the built in one for XFree. This was going to be a major problem for me. I look at my other machine and note that I am finished downloading RedHat Taroon-AMD64 beta. Here we go…


After a moment of thought I reboot the machine and stick the RedHat Taroon disc one in the drive. It boots, and launches the kernel, but then when it gets to the install, it “looses the interrupt” for the IDE CDrom drive. Since my SCSI controller is so old, booting from it is not an option. I found out from trial and error that if I take the CD off of the secondary IDE channel it works fine.
Ok, booting into RedHat install again and it all seems fine (except for the scary blood red BETA across the install splash screen in huge spray paint like fashion). After a hitch free install of RedHat beta I reboot. Going through the booting process I see that it can not get DHCP from nic2. Hrm. I am greeted with the “final steps in X windows (works in RedHat) and complete the install and login as root. BlueCurve greets me, but I notice something else… it is excruciatingly slow to read and write from the drive. Looking over the dmesg output I notice two things, I am in PIO mode on the IDE drives and the 3Com driver is not loading due to a PCI bus conflict. Hrm. Thinking back to the
night before, I decide to rebuild the kernel and patch the IDE driver as I did
in Gentoo. The source code itself for RedHat is different, but not hard to
figure out (they just go through more checking to see what chipset is being
used). I insert the appropriate modifications for the Nforce3, and also hack the
pci_ids.h file. I then note that the Nforce3 is listed already in that file.
Strange. I save all of those, double check them and do my normal recompile.
This crashes horribly. The kernel compile can not even get past the first few
items. It seems to be crashing on everything that gcc has to touch. I look
through the rpms and see what is installed. Everything looks good, but it seems
as though what they sent in the beta release does not correspond to the version
of headers and gcc that is distributed. I give it up to it being beta (as they
warn) and sit there looking at my new hardware. I have a choice to make.


I can either run this machine in server mode with no X windows under Gentoo and
have the high speed I/O, or I can have X windows under RedHat beta and deal
with it being slow. First, let us define slow: I copied the second RedHat Taroon disc from one drive to another (approximately 650megs) and it took over 6 mins, or roughly 1Mb/s. This is unacceptable. This can be fixed under Gentoo but not
under RedHat. Under Gentoo we have no X windows.


Idea: I will go and look for other distributions, recalling that I noted earlier
that Mandrake and Suse were supposed to have AMD 64/Opteron versions out. Well, the Mandrake version is 9.0 and is now unavailable. They have a corporate server
edition out, (2.1 for AMD Opteron) and it is reasonably priced at $749. Ouch.
Well, I can not afford that (much less $100) so I head on over to the Suse web
site. They also do not have a personal edition that supports Opteron, but they
do have an “Enterprise” server edition that they would like to sell me for $767.


I give it all up and proceed with the following plan: install RedHat 9.0
standard and rebuild the kernel with the pci and IDE patches in place.
I put my RedHat 9.0 CD in and away I go. After install X windows works fine,
IDE access is slower than a 486 with a 5400 rpm drive and I am in 32
bit mode. I let out a large sigh and go back to working on my homework, oddly
enough, for my VLSI class.


The good:
Performance notations: I wish to state the following, when I started all
of this I was going to put forth some performance metrics of my own
choosing, namely the standard povray benchmark and others. On the ABIT
motherboard, with the same ram and hard drive running the AMD XP 1.6Ghz
cpu, under RedHat 9.0, the standard povray benchmark takes right at 49 mins. Under RedHat
9.0 on the Opteron/ASUS setup (same ram/hard drives, etc) the difference is
mere seconds, at just under 49 mins. Let me point out that in reality,
this is impressive. I have read all of the benchmarks on the review sites
where they go on about it being faster even though they are running a 32
bit OS, but let me remind the reader that in order for the Opteron to run
a 32 bit app, it is doing its own internal translation which takes time.
The end user really is not gaining any 64 bit performance increase, and
typically we would expect a performance penalty and not a gain. This is
a good thing. A CPU which is performing translation, taking a penalty
for it, and running a 200Mhz slower clock speed is still faster
than a CPU running natively on the same hardware (less the motherboard).
This gives us a good feeling for the performance of the opeteron, in
that it should be an excellent performer with 64 bit apps running a
64 bit operating system.


Note that I did not run this test under gentoo as povray compiles crashed
and burned and running the 32 bit app on the 64 bit OS may be interesting,
but it was not to me. I wish to see native performance of 64 bit apps on
a 64 bit OS. In particular, on Linux.


Conclusions:
Linux is close to being on the AMD 64/Opteron systems, and
in fact may be there with Suse and Mandrake (RedHat claims “sometime”),
but at that price range for the OS itself, its use will (for now) be
limited to commercial application. I wonder about personal adoption of
the platform. AMD claims that it is to be the next generation of
personal computing, and I tend to agree (in part). The normal user
may not have as much to gain as say myself, or a CAD/CAM user or someone
doing imaging, etc (there are many applications which stand to gain).
But this adoption will be slow if all of the OS vendors think that
this platform will be limited to commercial/industry use only. Funny,
this reminds me of Sun Microsystems applications. I once wanted a copy
of mathematica for my Sun Ultra workstation for my Thesis, and they could
not believe that as an individual, I owned a Sun workstation. I received
the same response from Suse, who I called and inquired about getting a
copy of the server for Opteron to use at a reduced rate. They could
not fathom that an individual would actually own an Opteron system,
much less have a need for it. Gentoo looks very promising, and I will
probably devote my spare time (when it comes along) to porting window
managers and other things to the Opteron. For those who were paying
attention, X should actually work fine with say an ATI card, as it is
the Nvidia drivers themselves which are problematic on the Nforce3 and
not the XFree server itself. A working window manager is another thing.
Then again, Nvidia may rebuild their drivers to actually work (according
to the Gentoo developers, no one has had luck getting their drivers
to work on the Nvidia).


It would be advantageous for Linux users and others if a personal
desktop edition would come from a large vendor for the AMD Opteron.
A fully working Gentoo release would be good also, but the mainstream
workstation users may not want to go that route (as eye candy may be
important to them or a less demanding install).


For the time being, its 32 bit land for my Opteron.


About the Author:
Robert Minvielle is a Ph.D. student at the Center for Advanced Computer Studies at UL at Lafayette. He has a Masters in Physics and a B.S. in Computer Engineering, and enjoys building hardware, embedded systems, and programming. He can be reached at [email protected].

55 Comments

  1. 2003-09-22 7:36 pm
  2. 2003-09-22 7:46 pm
  3. 2003-09-22 7:51 pm
  4. 2003-09-22 7:52 pm
  5. 2003-09-22 7:57 pm
  6. 2003-09-22 8:00 pm
  7. 2003-09-22 8:08 pm
  8. 2003-09-22 8:20 pm
  9. 2003-09-22 8:22 pm
  10. 2003-09-22 8:23 pm
  11. 2003-09-22 8:24 pm
  12. 2003-09-22 8:34 pm
  13. 2003-09-22 8:38 pm
  14. 2003-09-22 8:39 pm
  15. 2003-09-22 9:19 pm
  16. 2003-09-22 9:20 pm
  17. 2003-09-22 9:41 pm
  18. 2003-09-22 9:55 pm
  19. 2003-09-22 10:23 pm
  20. 2003-09-22 10:28 pm
  21. 2003-09-22 10:47 pm
  22. 2003-09-22 11:03 pm
  23. 2003-09-22 11:04 pm
  24. 2003-09-22 11:04 pm
  25. 2003-09-22 11:40 pm
  26. 2003-09-23 12:50 am
  27. 2003-09-23 1:38 am
  28. 2003-09-23 2:11 am
  29. 2003-09-23 4:33 am
  30. 2003-09-23 4:35 am
  31. 2003-09-23 4:54 am
  32. 2003-09-23 6:31 am
  33. 2003-09-23 6:51 am
  34. 2003-09-23 8:21 am
  35. 2003-09-23 9:25 am
  36. 2003-09-23 11:29 am
  37. 2003-09-23 11:39 am
  38. 2003-09-23 1:57 pm
  39. 2003-09-23 7:27 pm
  40. 2003-09-23 8:08 pm
  41. 2003-09-23 9:40 pm
  42. 2003-09-24 1:49 am
  43. 2003-09-24 5:20 am
  44. 2003-09-24 6:27 am
  45. 2003-09-24 11:24 am
  46. 2003-09-24 11:28 am
  47. 2003-09-24 11:42 am
  48. 2003-09-24 1:51 pm
  49. 2003-09-24 3:33 pm
  50. 2003-09-24 4:35 pm
  51. 2003-09-24 4:56 pm
  52. 2003-09-24 6:58 pm
  53. 2003-09-29 1:50 am