Linux on the Opteron, Part II

This is the second installment of the “Linux on the Opteron, are we ready?” article. Basically, it is a “where are we now?” article, noting that what once did work now does not, and others that did not work now do. The first article was published on OSNews almost three months ago. Since that time not too much has happened publicly in regards to the amd64 Linux situation, but a lot of people mailed to tell me that I should have checked out SuSE or the new Mandrake which was “about to be released” at that time. Also since that time I have upgraded the RAM and acquired a larger hard disk for the machine. I will give a brief rundown of the system as it stands now, what I tried to install on it, and what works.

The system now: Basically it is the same as before with some minor changes and a BIOS update:


Motherboard: Asus SK8N, BIOS upgraded to 1004
CPU: AMD Retail Opteron 240 (1.4GHz)
RAM: two 512MB sticks of Kingston (ASUS recommended 333MHz dual channel DDR, ECC registered)
DISKS: One Maxtor 7200rpm 80GB UltraATA drive, One Liteon CDR/RW drive.
Network: One $5 Linksys PCI 10/100 NIC
Video: ATI FireGL, AGP, 128MB ram.
Case: Lian-Li PC-7, with optional Coolermaster white cold cathode fan in the back (makes it easier to see in the case when working on it)
Power Supply: Enermax 460Watt, fan monitor controlled.


Click for a larger image Recall from the previous article, that this is the same setup with a new case, new ram, bigger drive, different video and bios update.
Also the 3COM NIC is now in another machine.


So, first, a super quick rundown of my previous experience with this setup (mostly from the first article, and more observations from time spent with the machine about a week after the article came out):


Mandrake Beta2 for AMD64 worked, sluggish drive performance, KDE worked like a charm, intermittent lockups, but mostly only when writing large files to the drive (e.g. a Linux iso).


Red Hat Taroon beta for AMD64 worked also, X worked slowly and again, had intermittent lockups similar to the Mandrake install.
Gentoo for AMD64 worked great and fast, but no X windows. (you could run X in twm mode but no other window managers)
SuSe 8.2 for AMD64 worked, but again sluggish IDE performance and compilation of 32bit software would fail badly. KDE worked.
Red Hat 9.0 for i386 worked fully, but X acceleration was not supported.


Note that in all of the above setups with the exception of Mandrake
that compilation of 32bit applications failed. Precompiled binaries worked
as far as I could tell (with the limited number of binaries I loaded).
Keep in mind that all of the above was with the Nvidia Geforce4, which
has been replaced by the ATI FireGL. After the article was
said and done, I installed Red Hat 9.0 for i386 and used it until
I switched back to my old AthlonXP so I could have accelerated X windows.
Note that I did try the FireGL in the original Opteron setup also.


Having a 64 bit machine sitting around collecting dust is not
my idea of a good time, so I decided (since the semester is over 🙂
to have another go at installing again, and also to see what improvements
were made in distributions since some new releases have come out.
Specifically, I have installed (or tried to) the following (all of
which are the 64bit version, also known as amd64 or x86_64):


Gentoo LiveCD beta (10-24-2003)
Gentoo LiveCD beta (11-09-2003)
Mandrake RC1
Suse 8.2 beta2
Suse 9.0 (ftp install, no iso files)
Red Hat Taroon (again, to see on the FireGL)

I also installed Fedora core 1 to check it out just to compare
since I had installed Red Hat 9 on the original Opteron setup.


I shall present my problems and observations with the above
distributions in order, but first I would like to say two things:


1) YMMV. I have read through some of the problems others have had
and how they overcame them on various mailing lists. It seems
that some of the problems I had were specific to my setup and
others did not have them at all on other motherboards or with
other video cards.
2) I did not spend all year testing each distribution. I was
on a quest to get something on my Opteron that I can run my
analysis software on, and compile my OSS on. I wish to use
the Opteron due to its large cache and 64bit architecture.
Having said that, I did not just install and say “eh, it does
not work” and move on either. I at least tried to get each
one working and looked for ways to overcome problems, being
as I had already spent time putting the software on the machine.


Ok, here we go. Gentoo LiveCD 10-24-2003: this is the one
I actually used the least, as I could not get X working at all,
and wanted to move on. I will note however, that with the
first Gentoo amd64 release that I picked up (not the liveCD)
X worked, but no window managers would compile (less the built
in one). It is worth mentioning that everything else worked,
and worked well and fast. I would recommend this (actually
the 11-09-2003 CD) for servers.


The second Gentoo LiveCD (11-09-2003) was no different from
the first (IMHO). It seemed like everything works with the exception
of the X, _however_ X would not even compile with this distribution.
A little poking around the IRC channel and the bugzilla list
did not seem to point to anything in particular, and people on
the list said they had no idea. Others seemed to have it working
but could not get the Radeon driver to work, while even more
users said it worked fully. I received compile time errors
on Xfree all over the place, too many for me to give a good diagnosis
of. I gave up on this one and moved on. Again, other than that,
it is using the 2.6.x line of kernels, all of my hardware was
found (with the exception of the on board NIC, which no distribution
of Linux can “see”), and it is quick. I recommend this for a server
box.


Before I stuck in the Mandrake RC1 CD, I did have that funny smile
on my face, knowing that since Mandrake beta1 worked so nicely with
the exception of the “writing a large file to the drive lockup” that
this was going to be the distribution. This was going to work, and
work great. (sometimes I have a soft spot for Mandrake) This was not
the case. The install would go smoothly for the first two steps and
then tell me that there is a problem with my hard disk and that it
needs to be wiped, and would I like to proceed? Odd. Telling it to
move on and wipe the disk results in an error: “Can not format
disk, drive error”. I put in the Gentoo disk and fdisk the drive,
no problems. I then partition the drive into three partitions (/boot,
swap and /) so that Mandrake can have an easier time installing.
Again, when I go to install Mandrake it tells me that the drive
is corrupt/damaged/no space, and would I like to wipe/format it.
I tell it yes, and then it tells me that there is a drive error.
To make a long story short, I checked the drive, I even installed
Fedora core 1 on it, and Mandrake still would not install. I even
went so far as to try all possible hard disk access modes in the
BIOS with no luck. Mandrake was no go. I decided to fall back
on Mandrake beta1 and then see if I could upgrade. Mandrake
beta1 had the exact same error. My conclusion is that the updated
BIOS has some issue that Mandrake hates. Oddly, no other distribution
exhibited this problem.


Suse 8.2 beta2 went well during install, and seemed to not
have any problems. It did identify that there is an on board
network controller on the ASUS, but could not do anything with it.
After install I started to have the same problems that I had with
Mandrake etc on the old setup, random lockups when writing huge
files to/from the drive. Other than that problem, everything seemed
to work ok, even X and KDE. Having seen that 9.0 was up on the
ftp site, and having some time to install it, I decided to once
again scrap the drive and try out Suse 9.0…


Suse 9.0 (what I am running now, and typing this on) was a fairly long
install (due to my slow net access). The install took about two and a half
hours to fully complete from start to finish.


I only encountered two problems with Suse, one of which is also present
in some fashion in all other distributions with the exception of Gentoo:
ACPI. In Suse (again, on this ASUS motherboard) unless you turn ACPI
off (by going into failsafe on the install, or passing the acpi=off
kernel argument) the kernel will fail to load completely. Slightly
annoying. Secondly, during the install phase, it would not let me change
the resolution on the video card (it was defaulted to 1024×768) and
I wanted to change it before I “started the system”. It is not a
big deal, but made me wonder if anything else that I left default
before would also have this problem. I am not sure, and after
2 hours of installing via ftp, I did not want to go back and find out.
In Suse 9.0, KDE works, X works, and amazingly, OpenGL works.
I have not noted this before, but no other distribution (which
has had X working at least) has been able to do OpenGL on my
ATI card. Xfree under Suse is using the “radeon” driver, but so was
Mandrake and Red Hat 9.0 (original setup, with the FireGL installed).
I make a second notation here in that the Nvidia also would not
do OpenGL on any distribution (original setup). I was slightly
pleased to see this. The only questionable aspect of the Suse
distribution is the choice of kernel, which is 2.4.21. I know
that 2.6.x is beta for now, but it does seem (from the Gentoo
installs) that it is faster and able to play nice with the ACPI,
unlike 2.4.x on this motherboard.


Some notes that may be of interest, that I have made along the way:
Nvidia cards on these boards are very hard to setup, I suggest
something else, or wait. Nvidia has new 64 bit compiled binaries on
the Nvidia web site, but I have not tried them as my Nvidia card
is now stuck in another machine. When installing on the Opteron
which for now I think all motherboards use the Nforce3 chipset,
pass two kernel parameters just to be safe: acpi=off and idebus=50
I found out by reading the source that the idebus option will
not take a value greater than 50, but setting it to 50 is a
LOT faster than the default 33MHz mode. If you have an all SCSI
system, skip the idebus argument.

Final thoughts:


Benchmarking:
Again, I do not have any benchmarks on CPU performance to show
off. I did run the povray benchmark, but looking over the Povray benchmark
page I see P4 3.0GHz machines getting beat out by Athlon 2800’s and
vice-versa, so I have little faith in the results (or at least in
peoples postings to the site). Other sites on the net have run benchmarks
on the Opteron, but without a 64 bit OS, and without 64 bit compiled
benchmarks, I also have little faith in them. They should be used as
a “touchy-feely-sort-of-may-bee” mark at best, in my opinion.


Cost:


This is the major factor in my mind. The setup I have above is
actually cheap, when you think about it. For those bargain shoppers
out there, this may not seem like a cheap computer at all, but for
those of us who need 64 bit computing, its a deal. For example, the
configuration above costs (price from pricewatch as of
12-16-2003) $1196 with shipping, $300 of which is the video card.
Yes, this may sound like a huge amount for a computer, but consider
the alternatives. A Sun Blade 1500 with comparative features for instance.
The “large” model comes with 1GB ram and an 80GB IDE drive,
it costs $3995, and the Ultra-SPARC CPU is running at 1GHz.
While it is unclear of the performance differences between say
the Opteron and an Ultra-SPARC or a Power CPU, a large corporation
could just purchase one of each and perform their own internal
benchmarking, if they were considering purchasing any more
than a few machines.


“And now for something completely different”:


Again on cost, if you are doing normal day to day activities on the
machine and do not need 64 bit computing, the cost is large. I would
suggest a AMD Athlon XP Barton instead. In general, the performance
of loading and using a web browser, or Open Office, etc,
is about on par with a speedy AMD Athlon XP. I do not have any Intel
based machines to compare it to, so that is why they are left out
here. The cost difference is non trivial in the AMD line too, as a Athlon XP
based machine is going to be much less than an Opteron or FX/64.
I digress once again, having said that, and make the notation that
for a 1.4GHz machine, the Opteron is very speedy.


Rant on Suse 9.0:


In the past I have mostly run Red Hat. I have been using Linux since
1992 (tarballs and TSX-11 archives from MIT), and after slackware came
out I moved onto that, and then Red Hat. Today is a different story,
as we know of the recent changes at Red Hat. I have previously used
(or at least tried out for a week) SuSe 6.2, so I am not a complete
newcomer to SuSe. I find that SuSe 9.0 was very impressive for a
distribution that I think is still classified as beta. Everything
worked out of the box (application wise) and the Xwindows performance
was impressive (compared to the others). My suggestion thus far for
anyone building an Opteron/FX system who wishes to run Linux would
be SuSe 9.0, although the install could take an extreme amount of
time if broadband is not available. One option is to have someone
with broadband mirror the ftp site and then perform an NFS install
if you have a spare machine to mount the CD in. The 9.0 install
does not have ISO’s and can not be installed via CD. This is really
a shame as it is (thus far, IMHO) the best distribution out there
for the Opteron/FX in a desktop/workstation environment.


Conclusion:


The coming months (weeks?) should be interesting in that
Mandrake is set to release the AMD64 version any time now,
as they are taking pre orders for it in the Mandrake store.
Recall, it was one of the best (if not the best) in my first
review, and I blame the drive problems on the Asus BIOS update.
Gentoo is nearing (from what I read) a really stable working
system, and I have read repeatedly that others have it working
fully (as a workstation with X windows) on other motherboards,
so I again blame the Asus for my troubles with Gentoo. Red Hat is
another story, having dropped the desktop edition, the “workstation”
edition is well beyond my financial reach. A corporation may consider
purchasing a copy for evaluation, but I would be tempted to wait
on Mandrake or Suse.


FYI: costs as of 12-16-2003 for AMD64 Linux distributions:


Mandrake pre order $100
Mandrake corporate server $750 (standard support) $1500 (unlimited support)
Red Hat AMD64 workstation $792
Red Hat Advanced Server $1992
Suse Professional 9.0 $120 (distribution on DVDs, no CDs)
Suse Enterprise Server $767 (2 cpu) $1450 (4 cpu)


Looking at the above cost matrix and my experience, it is almost tempting to purchase SuSe just to have the DVDs (no CDs, strange). The enterprise/server editions seem to all be priced about the same, with no definitive mention of CPU capability from RedHat or Mandrake on the server editions. (I assume at least 2 CPU capability built into the kernel)


Final thoughts on hardware:


The Opteron system can be run with a 365 Watt power supply, which is what I was using in the first setup. I switched to a larger one
as more disks will be added to the system. One need not purchase a huge power supply for this system unless you need it, but a warning is in order: looking over the CPU specs, be sure to purchase a good power supply that can sink lots of current on the 3.3V line. Purchase good, high speed RAM. I switched from the 266MHz to the 333MHz _dual channel_ RAM and the difference was non trivial. As always, a good case is in order, not that the Opteron gets very hot in my Lian-Li, in fact after large compilations for hours the CPU heatsink is only warm to the touch, but the RAM can get hot, as well as the video card. Besides, you don’t want $600 worth of RAM/CPU/motherboard to be sweating it out in your case.


About the Author:
Robert Minvielle is a Ph.D. student at the Center for Advanced Computer Studies at UL at Lafayette. He has a Masters in Physics and a B.S. in Computer Engineering, and enjoys building hardware, embedded systems, and programming. He can be reached at [email protected].

37 Comments

  1. 2003-12-16 8:14 pm
  2. 2003-12-16 8:22 pm
  3. 2003-12-16 8:52 pm
  4. 2003-12-16 8:53 pm
  5. 2003-12-16 8:54 pm
  6. 2003-12-16 8:55 pm
  7. 2003-12-16 9:00 pm
  8. 2003-12-16 9:18 pm
  9. 2003-12-16 9:19 pm
  10. 2003-12-16 9:30 pm
  11. 2003-12-16 9:31 pm
  12. 2003-12-16 9:49 pm
  13. 2003-12-16 10:32 pm
  14. 2003-12-16 10:58 pm
  15. 2003-12-16 11:10 pm
  16. 2003-12-16 11:16 pm
  17. 2003-12-17 1:45 am
  18. 2003-12-17 1:55 am
  19. 2003-12-17 2:03 am
  20. 2003-12-17 2:10 am
  21. 2003-12-17 2:12 am
  22. 2003-12-17 2:39 am
  23. 2003-12-17 3:28 am
  24. 2003-12-17 4:13 am
  25. 2003-12-17 4:55 am
  26. 2003-12-17 4:55 am
  27. 2003-12-17 5:20 am
  28. 2003-12-17 6:14 am
  29. 2003-12-17 7:03 am
  30. 2003-12-17 7:25 am
  31. 2003-12-17 8:39 am
  32. 2003-12-17 8:52 am
  33. 2003-12-17 12:50 pm
  34. 2003-12-17 4:06 pm
  35. 2003-12-17 6:58 pm
  36. 2003-12-22 1:44 am
  37. 2003-12-23 4:27 pm