OS Review: NetBSD 1.6.2 on SPARC64

NetBSD is the king of operating system portability, running on 40+ different hardware platforms, including x86, MIPS, and even the Sega Dreamcast. So it comes as no surprise that among the supported platforms, NetBSD runs on my Sun Ultra 5. This will be the first in a series of operating system reviews for the SPARC platform. You can find the specifics of my evaluation machine here.

I had originally done a review of NetBSD 1.6.1, but just as I was finishing up the review NetBSD released 1.6.2. D’oh! So I took a look at NetBSD 1.6.2 as well.

Overview

NetBSD actually has two SPARC ports: SPARC for 32-bit systems (SPARC v7 and v8, which constitutes sun4/4m/4d/4c systems) and SPARC64 for the 64-bit SPARC systems (UltraSPARC processors, v9/sun4u). They are seperate ports and generally are incompatible with each other. The SPARC port will not boot on a SPARC64 system, and vice versa.

The SPARC64 port is the port I’ll be testing since my system is an UltraSPARC box. The SPARC64 port runs on a number of other sun4u/UltraSPARC systems; for more info you can check out the SPARC64 port page.

This NetBSD review is purely for its role as a server as there is no NetBSD support for the PGX frame buffers in the Ultra 5. (In fact, only two of the operating systems reviewed have PGX support, those being Solaris and Linux.)

Installation

You can download the distribution in tarball form or as a bootable ISO from the NetBSD site or one of its mirrors. The entire distribution fits on a single CD-ROM. It’s a pure text-based install and I was able to install from both the keyboard/screen and the serial port.

Installation for NetBSD 1.6.2 and 1.6.1 is kind of a mess. There are a number
of problems with sysinst, the NetBSD installation application. These problems
include segfaults and random — what I can only describe as “freak outs” — where a screen redraws itself 6 or 7 times. I had to go through the installation several times just to get the system to boot. While the 1.6.2 install seemed to be somewhat less troublesome than 1.6.1, it still exhibited the same strange behaviors, although to a lesser degree.

 Thank you for using NetBSD!
+-----------------------------------------------+
| NetBSD-1.6.2 Install System                   |
|                                               |
|>a: Install NetBSD to hard disk                |
| b: Upgrade NetBSD on a hard disk              |
| c: Re-install sets or install additional sets |
| d: Reboot the computer                        |
| e: Utility menu                               |
| x: Exit Install System                        |
 +-----------------------------------------------+

Another problem was that in both the NetBSD 1.6.1 and 1.6.2 installs the network interfaces came up unconfigured despite explicitly setting them during the install. As a result, I had to configure the network interfaces manually after installation.

PCI Devices

I installed an Intel 10/100 NIC, and NetBSD recognized it immediately as interface fxp0. I don’t have have any other PCI devices to install, but it’s good to know they have a good likelihood of working, and that making this system a two NIC system is as cheap as a $10 Ethernet card on eBay. I tested both the hme0 and fxp0 interfaces, and they both worked flawlessly for me.

32-bits Binaries Need Not Apply

NetBSD for SPARC64 is bit different than Solaris or Linux for SPARC, in that NetBSD SPARC64 exists only as a 64-bit operating system. The kernel is 64-bit, and so are the userland libraries and binaries. By default, there is no 32-bit userland and support must be compiled into the kernel (it is not enabled in the default kernel) in order to execute 32-bit binaries. Again, the 32-bit SPARC port will not boot on a 64-bit UltraSPARC system.

This means that even if you could compile a 32-bit binary (there are no 32-bit shared libraries to link to and the built-in compiler lacks required components to build 32-bit binaries), it would not run on the 64-bit NetBSD system. As such, I find the following statement on NetBSD’s SPARC64 port site a bit misleading:

A 64-bit kernel runs 64-bit binaries, and has support for running most 32-bit binaries as well, with the exception of some kernel groveling tools.

While there is a theoretical method for installing a 32-bit userland, it’s quite convoluted, and given the trouble that this guy had, I’m not up for even attempting this.

Another troublesome effect of this 64-bit only port is that while NetBSD distributes a binaries in a nice package system, located at ftp://ftp.netbsd.org/pub/NetBSD/packages/1.6.1, there is no SPARC64 directory. There is a SPARC repository for 32-bit SPARC binaries, but them will not work since there is no 32-bit userland or support in the default kernel and configuration.

Compiling Under NetBSD

Compiling applications under NetBSD can be a bit tricky, and there are a few gotchas that you should keep in mind.

In a previous article on compiler optimizations, I discussed the differences between the V7, V8, and V9 SPARC architectures and their impact on performance. You don’t need to worry too much about it with NetBSD SPARC64, since the included compiler (2.95.3) builds V9-based 64-bit binaries (-m64 -mcpu=v9) by default, although specifying -mcpu=ultrasparc can help tune the code (although it did cause me some signal 11 problems with GCC 2.95.3, but not 3.3.2). Also, specifying -mcpu for v7 or v8 won’t make much difference, since it ends up building a V9 binary anyway (with the performance advantages associated with a V9 build).

Not only does the compiler build for 64-bit by default (-m32 is the default switch for stock GCC, while -m64 is the default for NetBSD SPARC64’s included GCC), the build environment is incomplete to even attempt a 32-bit build.

In general, getting software to compile on NetBSD SPARC64 can be pretty tough going. Even if you get something to compile cleanly, it may not work right. Take OpenSSL 0.9.7c for example. When I ran into an SSH issue covered in the next section, I went to compile a newer version of the OpenSSL libraries. The system comes installed with 0.9.6g, but I wanted to go with the 0.9.7 series to be consistent with the other installs so I downloaded the source from the OpenSSL site.

I used -mcpu=ultrasparc as a compile flag, but the compiler would die with a signal 11:

gcc: Internal compiler error: program cc1 got fatal signal 11
*** Error code 1

I used -mcpu=v9, and for some reason it compiled. But while it compiled cleanly, the binary did not work correctly.

netbsd1# apps/openssl speed rsa dsa
To get the most accurate results, try to run this
program when this computer is idle.
Doing 512 bit private rsa's for 10s: openssl in free(): warning: modified (chunk-) pointer.
openssl in free(): warning: modified (chunk-) pointer.
Bus error (core dumped)
netbsd1# 

Which also shot out this kernel error to syslog:

Feb 19 05:27:00  /netbsd: Alignment error: pid=24954 comm=openssl 
dsfsr=00000000:00800005 dsfar=fffffa48:4959cdd isfsr=00000000:00000000 
pc=40508674

I tried using OpenSSL 0.9.6l (and g) downloaded from the OpenSSL site, and while the binary compiled, it gave another type of error:

netbsd1# apps/openssl speed rsa dsa
To get the most accurate results, try to run this
program when this computer is idle.
Doing 512 bit private rsa's for 10s: 242 512 bit private RSA's in 10.05s
RSA verify failure.  No RSA verify will be done.
24940:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:100:
24940:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:459:

I might be tempted to presume that this issue has something to do with the code not being 64-bit clean, but I know it is, since I compiled the very same source under Solaris 9 with both GCC and Sun’s compiler as a 64-bit working binary.

I poked around and realized when I ran ./config, it configured the system for NetBSD-sparc, and not NetBSD-sparc64:

./config 
Operating system: sparc64-whatever-netbsd
Configuring for NetBSD-sparc

While there is a FreeBSD-sparc64 and an OpenBSD-sparc64 definition, there is no NetBSD-sparc64 definition in either the 0.9.7 or 0.9.6 branch’s Configure script. So while the code is clean, it looks like perhaps the configure script isn’t setting up the right environment. I even tried using both the FreeBSD-sparc64 and OpenBSD-sparc64 build environment to compile, and while they both produced binaries, they still exhibited the same behavior.

I went into the pkgsrc directory, and found the openssl directory. Operating much like the ports envrionment in FreeBSD, it automatically downloaded the OpenSSL 0.9.6l source and compiled cleanly from source, applied patches (which included a NetBSD-sparc64 definition) and produced a working OpenSSL binary.

I used the same NetBSD-sparc64 defination from the OpenSSL 0.9.6g patch and applied it to the OpenSSL 0.9.7c source, and it finally produced a working binary.

The pkgsrc tree

I found the pkgsrc environment, which is downloaded as a tar ball from NetBSD’s site or mirrors, to be very important for getting various applications compiled. While I was able to get tcsh 6.12.00 and OpenSSH 3.7.1p2 to compile on my own, I was not able to get a working copy of OpenSSL compiled without it. The pkgsrc environment also built me a working GCC 3.3.2 specially modified to produce 64-bit binaries by default.

Packages that are compiled from pkgsrc are usually installed under /usr/pkg.

Unfortunately, even pkgsrc couldn’t get mysql 4 to compile, so I don’t have much in the way of benchmarks for this system (see Appendix A for details).

OpenSSL Mystery

Here’s one thing I can’t figure out: How the openssl package distributed with NetBSD ended up being so slow.

I first realized something was wrong when logging into the system via SSH took about 3 seconds to get a prompt back (an issue I wrote about in my GCC optimization article).

When I ran the openssl speed test on the included OpenSSL binary, I got results that were slower than I’ve ever been able to produce in any environment on my Ultra 5, including V7 in 32-bit Solaris.

OpenSSL 0.9.6g 9 Aug 2002
built on: NetBSD 1.6.1
options:bn(32,32) md2(int) rc4(ptr,int) des(ptr,risc1,16,int) blowfish(idx) 
compiler: gcc version 2.95.3 20010315 (release) (NetBSD nb3)
sign verify sign/s verify/s
rsa 512 bits 0.0248s 0.0022s 40.2 449.9
rsa 1024 bits 0.1279s 0.0076s 7.8 131.7
rsa 2048 bits 0.9217s 0.0276s 1.1 36.2
rsa 4096 bits 6.4647s 0.0928s 0.2 10.8
sign verify sign/s verify/s
dsa 512 bits 0.0224s 0.0281s 44.7 35.5
dsa 1024 bits 0.0750s 0.0927s 13.3 10.8

Take RSA verify/second for 512 bits for instance. In the 0.9.6g that came with the system, the results was 449.9. When I compiled 0.9.6l on my own with -mcpu=v9, the result for RSA verify/second 512 bits was around 1,100.

During my Solaris tests, I compiled 0.9.7c with -mcpu=v7, and it produced a 512-bit RSA verify result of 627, which was the slowest result I could produce and it was still faster than NetBSD’s binary. So how NetBSD managed to build a VV9-based binary so slow is a mystery to me.

Bare-Bones

The default installation of NetBSD is pretty baren, and doesn’t even include a Perl binary. Since there are no pre-compiled packages available for SPARC64 on the NetBSD site, you’ll have to compile them on your own. Using pkgsrc is probably your best bet for getting applications to compile.

File System

NetBSD comes with the standard FFS. Enabling soft updates can help file system performance, particularly when untaring a file. To do so, edit /etc/fstab and add “softdep” after the rw option.

/dev/wd0f   /usr  ffs  rw,softdep  1  2  

After a reboot, the system will come up with those file systems set for soft updates.

Conclusion

The NetBSD SPARC64 feels fairly incomplete to me. The lack of a 32-bit userland is a significant hindrance to its usability. I would imagine that the SPARC 32-bit port for sun4/m/d/c systems is much more solid, but the SPARC64 port is difficult to recommend.

Even if you’re an avid user of NetBSD and are very comfortable in that environment, you’ll likely still find this particular port a challenge and would need to put in a lot of work to make it a useful system. Compiling requires a higher level of aptitude on the ins and outs of a build environment.

Still, for all its weaknesses, it’s not a matter of shitty technology or bad craftsmanship, it seems to be more of a lack of attention than anything else. Someone could possibly make a name for themselves by taking on the task of making the SPARC64 port of NetBSD 1.6.2 more complete.

Appendix A: MySQL Odyssey

MySQL was particularly troublesome with NetBSD 1.6.1 and 1.6.2; so much so that I couldn’t get it to compile in either environment. I’ve detailed my journey here, perhaps someone can offer up a suggestion. The experience was identical for both 1.6.1 and 1.6.2.

Remembering the the problems I had with OpenSSL, I started off my MySQL compilation attempt from pkgsrc instead of a source tarball from MySQL.com. It started its compilation run, and after a while it died at this error:

sql_lex.cc: In function 'void lex_init()':
sql_lex.cc:85: 'symbols' undeclared (first use this function)
sql_lex.cc:85: (Each undeclared identifier is reported only once
sql_lex.cc:85: for each function it appears in.)
sql_lex.cc:87: 'sql_functions' undeclared (first use this function)
sql_lex.cc: In function 'int find_keyword(LEX *, unsigned int, bool)':
sql_lex.cc:171: implicit declaration of function 'int get_hash_symbol(...)'
sql_lex.cc:171: initialization to 'SYMBOL *' from 'int' lacks a cast
gmake[4]: *** [sql_lex.o] Error 1
gmake[4]: Leaving directory '/usr/local/src/pkgsrc/databases/mysql4-server/work/mysql-4.0.17/sql'
gmake[3]: *** [all-recursive] Error 1
gmake[3]: Leaving directory '/usr/local/src/pkgsrc/databases/mysql4-server/work/mysql-4.0.17/sql'
gmake[2]: *** [all] Error 2
gmake[2]: Leaving directory '/usr/local/src/pkgsrc/databases/mysql4-server/work/mysql-4.0.17/sql'
gmake[1]: *** [all-recursive] Error 1
gmake[1]: Leaving directory '/usr/local/src/pkgsrc/databases/mysql4-server/work/mysql-4.0.17'
gmake: *** [all] Error 2
*** Error code 2

I had about the same issue with that gen_lex_hash binary when compiling MySQL for 64-bit on Solaris. I found this Google listing. Apparently, the gen_lex_hash binary wasn’t creating a lex_hash.h header file correctly. When I ran gen_lex_hash by itself, I got this error:

netbsd1# ./gen_lex_hash > lex_hash.h
Fatal error when generating hash for symbols
Didn't find suitable values for perfect hashing:
You have to edit gen_lex_hash.cc to generate a new hashing function.
You can try running gen_lex_hash with --search to find a suitable value
Symbol array size = 8000

A Google search on that error showed a dead
end.

But the Google search also showed source code and a diff file for gen_lex_hash.cc from MySQL 4.1 and looked like it included quite a few changes, so I downloaded the gen_lex_hash.cc source from MySQL 4.1.1 and recompiled gen_lex_hash. It compiled fine, the binary work, and I continued the build.

Until… I got this:

/usr/lib/libstdc++.so: warning: reference to compatibility vfork(); include
 for correct reference

Followed by a whole mess of errors:

../innobase/trx/libtrx.a(trx0roll.o): In function 'trx_roll_pop_top_rec_of_trx':
trx0roll.o(.text+0x4d30): undefined reference to 'trx_undo_build_roll_ptr'
trx0roll.o(.text+0x4d78): undefined reference to 'trx_undo_rec_get_undo_no'
../innobase/trx/libtrx.a(trx0rseg.o): In function 'trx_rseg_mem_create':
trx0rseg.o(.text+0xbdc): undefined reference to 'flst_get_last'
trx0rseg.o(.text+0xbe8): undefined reference to 'trx_purge_get_log_from_hist'
../innobase/trx/libtrx.a(trx0sys.o): In function 'trx_sysf_create':
trx0sys.o(.text+0x20e8): undefined reference to 'ut_dulint_create'
(continues for quite a while)

I tried GCC 2.95.3 and 3.3.2’s version of C++, and to no avail.

So I tried addding this to mysqld.cc:

#include  

While this got rid of the libstdc++ error, the other errors still persisted. I used /usr/pkg/gcc3/lib to look at GCC 3’s libstdc++.so, but that didn’t help either.

So I tried downloading the plain source from MySQL’s site. Running the configure script initially complained about -pthreads, so I used the configure script invocation that was used in pkgsrc and the configure script completed.

Then it complained about readline (which allows up-arrow last-command in the MySQL client), so I downloaded readline’s source and adjusted the ldflags to include /usr/local/lib in the client subdirectory.

Then I got this:

my_pthread.c: In function 'my_pthread_setprio':
my_pthread.c:38: error: storage size of 'tmp_sched_param' isn't known
my_pthread.c:41: error: 'SCHED_OTHER' undeclared (first use in this function)
my_pthread.c:41: error: (Each undeclared identifier is reported only once
my_pthread.c:41: error: for each function it appears in.)
my_pthread.c: In function 'my_pthread_getprio':
my_pthread.c:50: error: storage size of 'tmp_sched_param' isn't known
my_pthread.c: In function 'my_pthread_attr_setprio':
my_pthread.c:67: error: storage size of 'tmp_sched_param' isn't known

At that point, I gave up. It’s listed as a bug, and apparently it’s been fixed by a patch already in pkgsrc, but then I’d hit the same walls I hit earlier. I had the same problems with both NetBSD 1.6.1 and 1.6.2.

Appendix B: dmesg

Here is the dmesg from my Ultra 5 running NetBSD 1.6.1. I forgot to grab the dmesg for 1.6.2 before I installed the next operating system.

NetBSD 1.6.1 (GENERIC) #0: Mon Apr  7 10:27:16 UTC 2003
    [email protected]:/autobuilder/build/netbsd-1-6/
sparc64/OBJ/autobuilder/build/netbsd-1-6
/src/sys/arch/sparc64/compile/GENERIC
total memory = 256 MB
avail memory = 228 MB
using 1651 buffers containing 13208 KB of memory
bootpath: /pci@1f,0/pci@1,1/ide@3,0/disk@0,0
mainbus0 (root): SUNW,Ultra-5_10
cpu0 at mainbus0: SUNW,UltraSPARC-IIi @ 333 MHz, version 0 FPU
cpu0: physical 32K instruction (32 b/l), 16K data (32 b/l), 2048K external (64 b/l)
psycho0 at mainbus0 addr 0xfffc4000
SUNW,sabre: impl 0, version 0: ign 7c0 bus range 0 to 2; PCI bus 0
DVMA map: c0000000 to e0000000
IOTSB: 12be000 to 133e000
pci0 at psycho0
pci0: i/o space, memory space enabled
ppb0 at pci0 dev 1 function 1: Sun Microsystems Simba PCI bridge (rev. 0x13)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled
ebus0 at pci1 dev 1 function 0
ebus0: Sun Microsystems PCIO Ebus2, revision 0x01
auxio0 at ebus0 addr 726000-726003, 728000-728003, 72a000-72a003, 72c000-72c003, 72f000-72f003
power at ebus0 addr 724000-724003 ipl 37 not configured
SUNW,pll at ebus0 addr 504000-504002 not configured
se at ebus0 addr 400000-40007f ipl 43 not configured
com0 at ebus0 addr 3083f8-3083ff ipl 41: ns16550a, working fifo
com0: console
kbd0 at com0 (console input)
com1 at ebus0 addr 3062f8-3062ff ipl 42: ns16550a, working fifo
ms0 at com1
lpt0 at ebus0 addr 3043bc-3043cb, 30015c-30015d, 700000-70000f ipl 34
fdthree at ebus0 addr 3023f0-3023f7, 706000-70600f, 720000-720003 ipl 39 not configured
clock0 at ebus0 addr 0-1fff: mk48t59: hostid 80a829a6
flashprom at ebus0 addr 0-fffff not configured
audiocs0 at ebus0 addr 200000-2000ff, 702000-70200f, 704000-70400f, 722000-722003 ipl 35 ipl 36: CS4231A
audio0 at audiocs0: full duplex
hme0 at pci1 dev 1 function 1: Sun Happy Meal Ethernet, rev. 1
hme0: interrupting at ivec 3021
hme0: Ethernet address 08:00:20:a8:29:a6
nsphy0 at hme0 phy 1: DP83840 10/100 media interface, rev. 1
nsphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
ATI Technologies 3D Rage Pro (VGA display, revision 0x5c) at pci1 dev 2 function 0 not configured
pciide0 at pci1 dev 3 function 0: CMD Technology PCI0646 (rev. 0x03)
pciide0: bus-master DMA support present
pciide0: primary channel configured to native-PCI mode
pciide0: using ivec 1820 for native-PCI interrupt
wd0 at pciide0 channel 0 drive 0: 
wd0: drive supports 16-sector PIO transfers, LBA addressing
wd0: 8693 MB, 17662 cyl, 16 head, 63 sec, 512 bytes/sect x 17803440 sectors
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2 (Ultra/33)
wd0(pciide0:0:0): using PIO mode 4, DMA mode 2 (using DMA data transfers)
pciide0: secondary channel configured to native-PCI mode
atapibus0 at pciide0 channel 1: 2 targets
cd0 at atapibus0 drive 0:  type 5 cdrom removable
cd0: drive supports PIO mode 4, DMA mode 2
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2 (using DMA data transfers)
ppb1 at pci0 dev 1 function 0: Sun Microsystems Simba PCI bridge (rev. 0x13)
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled
fxp0 at pci2 dev 3 function 0: i82559 Ethernet, rev 8
fxp0: interrupting at ivec 18
fxp0: Ethernet address 00:90:27:c2:88:5e
inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 4
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
pcons at mainbus0 not configured
No counter-timer -- using %tick at 333MHz as system clock.
Kernelized RAIDframe activated
root on wd0a dumps on wd0b
root file system type: ffs