In my recent article reviewing OpenBSD 3.4, I ran into a few issues. First off, a few days after my OpenBSD 3.4 article went up, OpenBSD (without bothering to consult me) went and released OpenBSD 3.5. I hope no one noticed.
Another issue I ran into was that I unable to get MySQL 4.0.17 or PostgreSQL (7.4.2 or 7.3.2) to run the sql-bench benchmarks successfully. MySQL would die with an apparent file descriptor issue, while PostgreSQL would die from a bus error.
But that was then, and this is now. With OpenBSD 3.5 out, I decided to take a look. I’ve also been able to resolve the issues with MySQL, and I’ve got a batch of benchmarks to share as well as some other goodies.
MySQL Issues
When I attempted to run my compiled version of MySQL 4.0.17, the tests would eventually fail with errors such as this:
Accessing tables Can't find file: './test/bench_60.frm' (errno: 9) at ./test-create line 137
It looked to others as well as myself that this was a file descriptor issue, but kicking up the file discriptors to as high as 32,000 with sysctl as suggested (and documented in my response to the OpenBSD/sparc mailing list) failed to provide a resolution. The sql-bench benchmarks just wouldn’t run without MySQL dying.
Why Didn’t You Use Ports/Packages!?!?!
When I couldn’t get MySQL to run the sql-bench benchmarks without dying, I noticed several comments shaming me for not using packages or ports. Some of the the comments were actually quite obnoxious.
Well, there are a couple of reasons why I didn’t use ports/packages for MySQL. The first being that OpenBSD 3.4 does not have a SPARC64 MySQL 4.0.x package or port. There is a 3.x MySQL package and port for SPARC64, but since I’ve used MySQL 4.0.17 for my other evaluations, comparisons with 3.x wouldn’t provide much insight. Also, 4.0 is the current release of MySQL and has many features that 3.x does not.
OpenBSD 3.5 has added a package (which I tested, read on) for MySQL 4.0.18, but of course that wasn’t around when wrote my OpenBSD 3.4 review.
Packages and Ports versus Compiling From Source
Some are of the the opinion that compiling from source is bad, even making the assertion that the OpenBSD FAQ claims not to compile from source unless absolutely necessary (it says no such thing, only recommending that packages are easier). Regardless, I disagree that you shouldn’t compile from source.
While there are cases where packages and ports are easier, I tend (with certain exceptions of course) to prefer compiling from source code whenever possible.
For one, that’s what they’re there for; to compile applications. Packages and ports are meant as a compliment to source compilation and as a convienence, and not a wholesale replacement.
Also, I deal with a number of operating systems on a regular basis, so compiling from source provides a consistency accross a variety of platforms which simplifies my efforts.
Compiling from source gives me a great deal of control over the application such as where it’s installed, what optimizations I may want to use, what features and modules are compiled in, and so forth. While such control can be available in ports (although arguably more difficult to control than it is in source), it’s not available in packages.
There are of course situations where using ports or packages is perhaps the best solutions. For instance, applications that can be very tedious and take an inordinate amount of time to compile. Take GNOME or KDE, for example. They both take a long time to compile (especially on a 333 MHz UltraSPARC IIi), are comprised of a multitude of smaller libraries and components, and on a non-standard operating system is bound to come up with quite a few errors.
For very simple applications, such as the bash or tcsh shell, I also tend to use packages.
Another way that packages and source may help is with non-x86 platforms. In my NetBSD 1.6.2 SPARC64 evaluation, I elaborated the trouble I had with OpenSSL compilations from source. The OpenSSL source distribution didn’t have a compile environment defined for NetBSD 1.6.2 SPARC64. The NetBSD ports section did, however, so I was able to get OpenSSL up and running. I was also able to use the modifications used in ports to get the OpenSSL source distribution compiled and running.
Also, a package or port may not be available for the version of the software (or available period) I’m interested, especially with the more esoteric applications. NetBSD didn’t have OpenSSL 0.9.7, only 0.9.6. OpenBSD 3.4 didn’t have MySQL 4.0.x. in either packages or ports for SPARC64.
However, applications such as Apache, MySQL, PHP, PostgreSQL, Perl, OpenSSH compile in a reasonable amount of time and are relatively easy to install. They’re also components that are often mission and security-critical, so they may need to be updated frequently, and sometimes more quickly than ports or packages can be released.
Sometimes neither package nor port nor compiling from source will provide a solution, such as with NetBSD 1.6.2. The ports MySQL 4.0 just couldn’t successfully compile MySQL 4.0.18, and the NetBSD group did not maintain SPARC64 packages at the time; only SPARC packages. They’ve since added a SPARC64 packages directory, although still no MySQL 4.0.x package. Compiling from source was also fruitless, and to this day I have no MySQL benchmarks for NetBSD SPARC64, which is a shame.
Where Was I Again? Oh Yeah, MySQL on OpenBSD
So after my article hit Slashdot, Sean Brown (who initially replied to my post to the OpenBSD/sparc mailing list) posted a response on Slashdot about the MySQL Manual having a known issue regarding OpenBSD 2.8.
I checked the MySQL Manual and I found the entry to he was referring to. The entry, listed as an OpenBSD 2.8 issue, seems to cover two distinct problems. The first issue listed is a threading bug where the symptoms manifest as (quoting from the entry) “slow response, high load, high CPU usage, and crashes”. That wasn’t what I was seeing, and it appears to be specific to OpenBSD 2.8.
The second paragraph in that entry seems to describe a problem which matchs exactly the issue I was having, and from the comments and my experiences it seems to affect OpenBSD of a variety of versions for a variety of platforms, not just 2.8
The entry recommend using --open-files-limit=2048 when invoking the MySQL daemon. It turns out it was a file descriptor issue, but I was changing file descriptors at the wrong place (that being sysctl). The user comments in the entry seemed to confirm this, and sure enough, using that flag sql-bench successfully ran.
I submitted a recommendation to the MySQL Manual page to list them as two separate issues.
Finally, Some Benchmarks!
With my compiled version of 4.0.17 running swimmingly, I am now able to share some benchmarks. Just to recap, I’m using the version of MySQL 4.0.17 compiled from source tarball off of MySQL.com‘s site. I used no compiler optimizations (although since this is a 64-bit operating system, the V9 instruction set [-mcpu=v9] is automatically used).
I ran the sql-bench benchmarks, and compared the results to results from other operating systems running on the same hardware (my Sun Ultra 5, as detailed in my first article).
The Solaris, Linux, and FreeBSD results were compiled from earlier tests from the same 4.0.17 source. They all ran as 64-bit binaries on 64-bit kernels with the exception of the Linux binary, which ran 32-bit on a 64-bit kernel (2.4.24).
For alter-table, ATIS, bit-tables, connect, and wisconsin, OpenBSD 3.4 performed favorably compared to Solaris 9, Linux 2.4.24 and FreeBSD 5.2.1.
However with the insert test, OpenBSD did terrible, performing worse than FreeBSD 5.2.1 which itself had a pretty awful showing.
The insert test took about three times longer to complete than on Linux or Solaris, and it was even worse than FreeBSD 5.1 which took 18,730 seconds (roughly 5 hours, 12 minutes) to complete the operation, where OpenBSD 3.4 took 24,226 seconds (about 6 hours, 43 minutes).
And now, OpenBSD 3.5
Since OpenBSD 3.5 is now out, I decided to give it a spin as well. Upgrading was a breeze. I made an OpenBSD 3.5 boot CD and booted up the installer. I chose the upgrade option, used the current networking configuration, and told it where to find a local FTP server with the 3.5 tgz files.
(I)nstall, (U)pgrade or (S)hell? u Welcome to the OpenBSD/sparc64 3.5 upgrade program. This program will help you upgrade OpenBSD in a simple and rational way. At any prompt except password prompts you can run a shell command by typing '!foo', or escape to a shell by typing '!'. Default answers are shown in []'s and are selected by pressing RETURN. At any time you can exit this program by pressing Control-C and then RETURN, but quitting during an upgrade can leave your system in an inconsistent state.
The entire process took about 15 minutes, and I was up and running with an
OpenBSD 3.5 system.
To see how MySQL faired on OpenBSD 3.5, I ran the sql-bench against the newly upgraded system.
The results were roughly the same with OpenBSD 3.5 as it was for OpenBSD 3.4. OpenBSD 3.5 seemed to have a slight performance gain over OpenBSD 3.4 for these tests, although for the insert and select tests, OpenBSD 3.5 still gets blown out of the water by Solaris and Linux.
OpenBSD 3.5 MySQL Package
So what about packages you ask? Well OpenBSD 3.5 for SPARC64 does sport a new package for MySQL 4.0.18, so I set out to run sql-bench against that.
I ran into some problems, however.
Like many package management systems, OpenBSD packages have the concept of dependencies, where certain packages are required before installing other packages.
Installing the mysql-server required mysql-client. No big deal. But it also required a myriad of other semi-related packages, including p5-DBD-mysql-2.90.03, p5-DBI-1.38, p5-PIRPC-0.2017 and probably a few more.
I’m not sure why they were set as dependencies, as DBI and DBD are Perl modules that are only required when running Perl applications that utilize MySQL. The benchmark suite sql-bench requires them, but for the database server itself to run they are not required.
The frustrating part about this is that there doesn’t appear to be a way to successfully force package installation to ignore dependencies. Using various combinations of the -f keys argument failed install the package.
I tried to install the required packages anyway, but I already had Perl DBI installed, so the package installation failed with the following complaint:
Dependencies for p5-DBI-1.38 resolve to: p5-PlRPC-0.2017
Collision: the following files already exists
/usr/local/libdata/perl5/site_perl/sparc64-openbsd/Bundle/DBI.pm
/usr/local/libdata/perl5/site_perl/sparc64-openbsd/DBD/ExampleP.pm
/usr/local/libdata/perl5/site_perl/sparc64-openbsd/DBD/NullP.pm
/usr/local/libdata/perl5/site_perl/sparc64-openbsd/DBD/Proxy.pm
/usr/local/libdata/perl5/site_perl/sparc64-openbsd/DBD/Sponge.pm
...
I gave up on the package system and just untarred the tgz file and installed the binaries manually. It was a little tricky, but I got MySQL up and running, and I ran sql-bench against the 4.0.18 package.
The results were somewhat in line with the 4.0.17 version I compiled, although it performed worse in a few of the individual tests.
It looks like they creators of the package are aware of the --open-files-limit fix, as it was incorporated into the mysqld_safe startup script.
Other Odds And Ends
I also had an issue with my compiled PostgreSQL 7.4.2 from source and PostgreSQL 7.3.2 from ports where sql-bench would die from a bus error. However, I didn’t try the package initially so I gave it a try. It also died with the bus error during sql-bench. So no joy there.
I noticed a couple of comments regarding X11, with several incredulous “of course it wouldn’t work!” comments when I tried starting X11 from the serial console. That starting X11 from the serial console didn’t work wasn’t my point; I even stated it wouldn’t work. My concern was that it actually crashed the system, instead of simply exiting with an error. While probably not a DoS tactic, it’s definately not stable behavior.
Another comment sent my way was that OpenBSD doesn’t actually listen externally on port 25 when sendmail is running; it’s only running as an internal MTA. Yeah, sorry about that, dumb assumption on my part.
Conclusion
While the OpenBSD results didn’t compare to well with the other SPARC64 operating systems performance-wise, it doesn’t detract from its usefulness, especially as a firewall/router system. The test was only MySQL, and with MySQL running, it’s all that much more of a useful system and would make a good development system in addition to a great firewall/router.
If you would like to see your thoughts or experiences with technology published, please consider writing an article for OSNews.
By default FreeBSD is quite slow (focus on stability more then performance out of the box), and needs to be tuned. I’ve seen benchmarks of FreeBSD before and after a good tune and its night and day. I suspect FreeBSD was not properly tuned, since it often at least compares well to Linux or is faster depending on the area.
What about tuning OpenBSD? Sure, even with tuning its not likely to be as fast as Linux but surely it could close the gap. I agree with Bruno, a Solaris 10 beta vs 9 would be very interesting indeed. 🙂
and, for me, none of the graphics loaded so I wasn’t able to view the benchmark results, but the accompaning text still helped me read the results. Whats wrong?
Can you determine why they aren’t loading? DNS issue? Connectivity?
They’re currently being hosted off my personal site, vegan.net.
Has anyone notified Theo of this? The insert performance is atrocious! Maybe some OpenBSD developers can find out what the bottle-neck is?… or is MySQL not something that OpenBSD people consider to be core to the use of OpenBSD? (ie. not a router/firewall/etc)
There are many options that can be tuned on all platforms to make it faster and performance better. I’m sure with appropriate tuning OpenBSD and FreeBSD could come well within the linux benchmarks. And just think on top of that OpenBSDs W^X and GCC+Propolice provide that extra layer of security that your looking for in a secure server
What tuning would you suggest?
Well, I am at work and that might be why, military base n all. I’m behind a proxy. I’ll try again once I get home.
Have you checked out the OpenBSD FAQ?
http://openbsd.org/faq/index.html
Lots of good information in there, also check the man pages. OpenBSD does not yet have a unified buffercache which really hurts performance, change the max nodes value to kick up disk performance –
http://openbsd.org/faq/faq11.html#maxvnodes
and I’m sure theres more I’m missing. There are lots of ways to tweak things, but there are still some fundamental performance areas where OpenBSD lacks like a zerocopy network stack, unified buffer cache, decent threading, etc. Not to say its slow, for my uses its more then fast enough. 🙂 Its merely slower then say Linux.
Solaris is better. It is the most secure operating system!!
The first thing I’d try is enabling soft-updates on the filesystem the database resides on (/var/mysql is the default when using the port).
This isn’t necessarily enabled by default (depending on how you partition the disk during install), you can check with ‘mount’, if it doesn’t say ‘softdep’ in the options for the mountpoint, it’s not enabled. Just edit /etc/fstab and add that option, then remount.
It usually makes a significant difference (sometimes factors 5-10) when creating (or deleting) lots of files (like extracting a tarball, or rm -rf a directory with many entries), but it might affect MySQL’s insertion (favorably) as well.
>Solaris is better. It is the most secure operating system!!
I agree but there are people on this forum that believe Windows colud be as secure as Solaris or/and Linux/BSD and that there are so many
flaws and security problems just because Windows is used more…
Solaris has a bigger userbase then say OpenBSD or FreeBSD so that statement really is untrue
I use OpenBSD for 2 things: firewall/NAT/Router and my old 300mhz laptop, it ROCKS! i did not notice and speed problems but then my laptop has a load average of: 0.0 : 0.1 : 0.2 so its not that heavily
used.
ps. i ALWAYS buy my BSD cds frm the official websites, show some support please.
duh, a big one I left out. It can make a massive difference in filesystem performance, check the faq I posted. 🙂 Re-run the benchmarks, I suspect a large improvement though still slower then Linux and Solaris.
on Net and OpenBSD you can check or enable softupdates by checking the fstab file in /etc, if you see something like
/dev/sd0a / ffs rw,softdep 1 1
then its enabled, if not add “softdep” and reboot. 🙂 Further information can be found at –
http://cvs.openbsd.org/faq/faq14.html#SoftUpdates
I don’t have much time but a little reply:
Funny that you base all your userbase calculations on one saying of Theo De Raadt, and then present it as a fact.
And you also didn’t take embedded NetBSD into account.
And besides that, how can FreeBSD go out of business? It isn’t a company!
Thats an old troll, near word for word, thats been floating around for at least a few years now. Yea, it is funny that its based on the words of Theo, and I’d suspect the user base is quite larger then that. I suspect that, like Apple, the BSDs are actually growing in use but simply not as fast as Linux, so it looks like their shrinking despite growth.
> …a few days after my OpenBSD 3.4 article went up…
> …went and released OpenBSD 3.5
> The OpenBSD team makes a new release every six months,
> with target release dates of May 1 and November 1.
http://www.openbsd.org/faq/faq1.html#Next
Everything alive dies. Its the most logical thing to do. We die, the earth dies,our sun dies, our solarsystem dies, our galaxy dies, our universe dies….
But since you took the time to paste such a nice story i will let you look at mine:
about 100.000 BSD users/systems in the world, for every 10 systems that die every week 1 comes back so thats about a loss of 9 per week, right! no if it continues to die like that it will be around for ore than 200 years. Anyway your dying too.
There are lots of funny stories to find. For example, if you type “OpenBSD” in groups.google.nl, you get this as one of the best results:
http://groups.google.nl/groups?q=openbsd&hl=nl&lr=&ie=UTF-8&selm=4e…
On the other hand, I’d like to buy an Ultra 2 (dual UltraSparc) so I’m very happy he has taken the time to try all the alternative OS-es on another Sparc64 system. Now can anybody point me to his review of Linux on the Ultra 5?
Linux runs on Ultra 5/10. What would you like to test on it?
The fine-tuning comments that a BSD is “as fast” or “faster” than Linux kernel after they’re fine-tuned are bogus since those do not take into account any fine-tuning on a Linux kernel based system. Futhermore, there’s no proof stated for this. Please also note the compare is between Linux kernel 2.4.24 while 2.6.x gives a better performance (multiple articles about this on ie. kerneltrap.org).
However the Ultra5/10’s have IDE which has its advantages and disadvantages. Frankly, i don’t see much use of IDE when the possibility of SCSI (v3) is there on a Ultra2/30/60 — which costs about the same. If you want to run some small database, why not take the Ultra2/30/60 instead?
Have you checked out the OpenBSD FAQ?
No, I’ve never read the FAQ. Don’t let my constant references to it in the past two articles fool you
Softupdates are already enabled for those tests, as they were with FreeBSD. I don’t use a BSD system without soft-updates.
Besides, in my FreeBSD tests soft-updates had little effect on the sql-bench performance metrics. You can see the differences here.
http://www.osnews.com/story.php?news_id=6552&page=2
My point is that it’s frequent that when an operating system perform shows bad performance on a test someone will come up with something to the effect of “oh, tuning will fix it”, which is kind of a cop out.
If is some magic tuning that I missed that OpenBSD didn’t see fit to enable by defaultthat which would have tripled performance on the insert tests to bring OpenBSD in line with Linux and Solaris, I’d be willing to try it. But stabbing in the dark with a nebulous statement like “tuning will fix it” isn’t adding much to the discussion.
Could tuning help? Probably, but I doubt it would bring it to the level of Linux and Solaris. And even so, I could then in turn tune Linux and Solaris and make the disparity in results between OpenBSD and Linux/Solaris even greater.
But this is only OpenBSD. These results don’t preclude this system from being a production database server; the poor I/O of the Ultra 5’s IDE system does. As for a router/firewall or a staging system, the results are either irrelevant or just not as important, and thus don’t detract from OpenBSD being a great choice for the Ultra 5.
But this is only OpenBSD
That line should have read: “But this is only a MySQL test”.
The thing I am most interested in, is how it works for normal use.
If I understand everything right, the Solaris Free Binary License only applies to single-processor systems, so using Solaris un an ultra 2 wouldn’t be really possible for me, thus I would need Linux or *BSD.
Now if I can get such a nice Ultra 2 with a Creator (3D) card, it would be rather sad if I could only use the serial console on it, as with FreeBSD. Now I have looked at the hardware compatibility of UltraLinux, FreeBSD and NetBSD, and came to the conclusion that:
– Solaris (Free) supports only one CPU,
– Linux supports everything,
– NetBSD supports most things, except those Creator cards,
– FreeBSD doesn’t support keyboard, mouse, Creator cards, sound and floppy.
Now the Ultra 2 has SBUS, so adding a cheap PCI video card to get X under Net/FreeBSD would be impossible, thus based on my list the only really useful OS would be Linux. Now there is only one problem: am I right?
The thing why I was interested in if Tony had also done a review of Linux on the Ultra 5, is that I wonder how good Linux would work in general on the UltraSparc.
The reason is the following. On my Apple Performa 6400, I have installed Debian Testing/Unstable. It works really well, all hardware is supported, and I can use KDE 3.2.2, XSane and Gimp 2.0. That is, with kernel 2.4. When I installed kernel 2.6.4, all applications that dared to use a pty (like xterm or ssh) went into an endless loop doing:
modprobe -char-major 2
modprobe -char-major 2 -minor 1
modprobe -char-major 2
modprobe -char-major 2 -minor 2
modprobe -char-major 2
modprobe -char-major 2 -minor …
…
And of course, this makes that these applications don’t work and such intensive use of modprobe also makes the system unusably slow. Now I want to know if any problems like that would await me on the Sparc64 architecture too.
The Linux review is coming up in the next few weeks. I’ll tell you this though; there are a lot of distros out there for SPARC/SPARC64, and they vary widely in usability, much more so than typical x86 distributions.
Sounds like the author might like to read up a bit more on ports and packages, how they work, the differences between them, etc.
I think you’d be hard-pressed to find a valid reason to ever “compile from source” by hand (instead of via ports, which is, in fact, also “compiling from source”).
Cheers.
I enjoyed the article quite well. Thanks for your efforts Mr. Bourke. I have read other OpenBSD articles recently, which used a variety of benchmarks to point out OpenBSDs lack of performance compared to other systems, but this one was more thorough and more interesting to read.
I use OpenBSD a lot and I have to say that it may not be the top performer out there, but it is a solid system that has met every one of my needs. It is easy to maintain, secure, and quite stable.
I mostly use OpenBSD for my file server, firewall and web server, and I have been very pleased with it. I also use OpenBSD for a production PostgreSQL database and have consistantly been able to run complex SQL queries, that take hours to complete, without a problem.
So, while its performance may not be the best, OpenBSD offers other benefits such as ease of use, stability, timely updates, and reliability, to name a few, which outweight its slower performance. At least that is how I look at it.
That was an excellent OpenBSD story on Google groups, thanks for sharing.
freebsd isn’t slow on default. openbsd isn’t necessarily slow but its slower than alot of systems… but offers superior stability and security..
i think you got your bsd’s mixed up
“For one, that’s what they’re there for; to compile applications. Packages and ports are meant as a compliment to source compilation and as a convienence, and not a wholesale replacement.”
I’d like to note that also .. those who create the packages and ports tend to know what they’re _doing_. It’s not some ./configure; make; make install that 99% of the community runs to compile their new software.
An excellent point. I should have added that packages and ports benefit users who aren’t as comfortable around a Makefile, especially when ./configure; make; make install doesn’t work as it should. Packages and ports are a good fall-back then.
However, even for OpenBSD 64-bit, most of the applications I compiled worked fine without any special additions, and the added control over the build environment is worth it to me.
In some cases however, such as PostgreSQL, neither packages nor ports nor source compilation begat a binary that could run sql-bench without dying from a bus error.
A bit about I/O bottlenecks. You had enabled softdeps, ok. But what about IDE modes?
I haven’t notice a link to dmesg of OpenBSD3.5 on your Ultra, but from previous article I’ve got this http://img.osnews.com/img/6892/dmesg.txt. The point was “wd0(pciide0:0:0): using PIO mode 4, DMA mode 2”. This mean that IDE disk uses MultiwordDMA 2 for transfers, which is quite differ
from UtraDMA2 supported by your IDE controller (Built-in UDMA2, 33 MB/s max – taken from http://www.osnews.com/story.php?news_id=5743). I wonder if DMA2 is the top capability of your disk? If not, and if it supports UltraDMA then here is the leak of OpenBSD IDE controller driver.
So is it possible that other OSes from the test works fine, on UDMA33, with your Seagate Medalist? Have you check it?
tony, instead of wasting time trying to benchmark mysql on openbsd why dont you spend time focusing on the new, compelling features of the operating system that your linux reader base might not be aware of (ie, carp, pfsync, etc.) its quite fantastic you were able to read the documentation and get the operating system installed on your ultra5. what would be useful, however, is actual benchmarks on how this machine performs as a firewall. this is a much more likely deployment scenario for openbsd then a database server. also, im curious to know how you handle upgrades, patch management and software removal given your affinity for installing software from source. actually, it may have been good for your previous article if you attempted to rev the existing mysql port with a current version to illustrate to people how the openbsd ports system works.
Dear Tony,
I understand some of your issues about wanting to compile from source. The thing to remember is that, Ports ARE SOURCE. The ports tree merely is a means by which to quickly and easily compile – from source – with patches added by the ports maintainer to account for the non-portability and often the linux-chauvenism of many programmers. In otherwords, ports are a build system. It fetches the original source code from the author’s website/ftpsite, patches it, builds it, packages it, and isntalls it. It’s an important step in software quality assurance. If you want to re-invent the portability wheel and compile from an original tar.gz of source, go right ahead – but at least look at the patches for the relevant port, because someone has probably solved the bus errors and other symptoms you are likely going to run into.
c.