Today we host an interview with Andy Tucker, Distinguished Engineer of the Solaris kernel. We talk about the internals of Solaris, the competition and the future of the Solaris OE. The 9th question was answered by Robert O’Dea, Director of Engineering at Sun Microsystems.
1. Why have the other commercial Unixes all pretty much bitten the dust? Is Solaris that much better, or is it just more important to Sun than HP-UX was to HP, AIX to IBM or IRIX to SGI?
Andy Tucker: I think the most important thing Sun has done to ensure the success of Solaris is simply to remain committed to it. Even in the early days of Solaris, when most Sun customers were still running SunOS 4.x and other companies with UNIX implementations were starting to look at NT, Sun stayed focused on Solaris.
You can also look at some of the “big bets” that were made early in Solaris development. One of the most significant was that of designing in support for multithreading and multiprocessing from the ground up. Doing this work up front allowed Solaris to easily scale on large multiprocessors, and to handle the multithreaded workloads that are increasingly common.
2. Do you think that the proprietary, company-supported development effort that you’re a part in has any specific benefits over the Linux kernel’s Linus-and-his-henchmen method?
Andy Tucker: The main advantage Sun has is that we can make sure our efforts are well integrated and are focused on the needs of Sun’s customers. There’s a lot of great stuff available for Linux, but the decentralized development model means that someone who’s looking for, say, both a fair-share CPU scheduler and network QoS support has to pull the pieces out of different places, build them into a kernel, and hope they work together. Solaris has these as built-in, integrated components that just need to be switched on.
3. Technically-speaking, what do you think of the Linux kernel and the Mach kernel? Also, how FreeBSD 5.x compares to Solaris?
Andy Tucker: I think they’re all fine operating systems, each with their strengths and weaknesses. Mach broke a lot of new ground: it was the first microkernel OS to get widespread use and introduced some basic concepts (such as processor sets) that we’ve since borrowed in Solaris. Linux obviously has a huge developer base, and as a result there’s a tremendous amount of activity and energy around it. FreeBSD (and the other *BSD implementations) are inheritors of the BSD legacy and have been the source of a lot of interesting ideas.
I don’t really like to do head-to-head comparisons, since I like to think of OS development as a collaborative exercise. We’re all working to improve the state of the art and to make life easier for our users. The open source operating systems are often a source of new and interesting ideas; I hope the developers of those operating systems see Solaris similarly.
4. Solaris has some very complex algorithms. STREAMS, page coloring, and multi-level scheduling are all more complex than what is usually implemented in UNIX kernels. In retrospect, which Solaris features have really paid off, despite their complexity, and which ones have not?
Andy Tucker: I’ll note that most modern operating systems incorporate some sort of page coloring and multi-level scheduling algorithms; Solaris is hardly unique in this regard. I think that in most cases the significant work we’ve done has paid off; the complexity (if any) is usually required to meet the customer requirements. We’re also happy to rewrite things if we find a better or simpler way to do something.
On the other hand, there are obviously some features that haven’t really succeeded in the customer base, such as NIS+. And there are also some cases where we took a direction with the underlying technology that turned out to be a mistake. An example is the two-level thread scheduling model, where thread scheduling happens both at user level and in the kernel. Although this approach had some theoretical advantages in terms of thread creation and context switch time, it turned out to be enormously complicated, particularly when dealing with traditional Unix process semantics like signals. In Solaris 8, we made an “alternate” version of the threads library available that relied solely on kernel-based scheduling; it turned out to be not only much simpler and easier to maintain, but also faster in almost every case. It particularly sped up Java code, which is obviously important to us. In Solaris 9 (and later) we switched over to the single-level library as the only one available.
5. What do you think about the Cathedral vs Bazaar idea when applied to OS kernels, where the programming model is rather different than than of regular application programs?
Andy Tucker: In some ways the Cathedral vs. Bazaar distinction seems a bit artificial. I don’t know about other OS companies, but within Sun we have hundreds of engineers from all over the company working on different parts of the operating system. Many of these people aren’t actually part of the Solaris engineering organization; they work on different hardware platforms, or on storage devices, or in the research labs, or on some other product that touches on Solaris in some way. We continuously release the latest code for internal use throughout the development cycle, and do beta tests to get feedback from customers. So in a way we’re doing “Bazaar” style development, even though it’s commercial product and all developers are Sun employees.
The difficulty with this type of development, particularly on a large complex piece of software like an OS kernel, is ensuring that changes are architecturally consistent, well integrated, and of appropriate quality. This doesn’t mean there can’t be a large development community, it just means there needs to be some person or persons that are checking proposed changes to make sure they’re not going to cause a problem. In Linux, this role is filled by Linus and some of the other folks working with him, who review the changes going into the official kernel base. Within Sun, we have groups of senior engineers who similarly review proposed changes for quality, appropriateness, completeness, etc..
6. Is OS research dead? In the past, there was a great deal of research about the basics of OS design, like allocators and scalable scheduling. Today, the focus seems to be on application-level advancements like new virtual machines and user-level development frameworks. Have OS kernels pretty much reached the end of their evolution, or do they see kernels continue to evolve, perhaps incorporating some of the research techniques like orthogonal persistence or exokernels?
Andy Tucker: The nature of OS research has changed over the years. In the 80’s and early 90’s, there was a lot of “big systems” research; universities and industry labs would start by building an operating system, and then use that as a platform for investigating new ideas. So CMU had Mach, Berkeley had Sprite, Stanford had the V System, etc.. This meant that there was a lot of re-examination of basic OS constructs — how to best build an OS from the ground up. As a result we had work on distributed systems, microkernels, etc. — but the systems
were all aimed at supporting the same applications, essentially the ones running on the researchers’ desktops.
Now most of the research I see is based on existing OS platforms, usually Linux or one of the *BSDs. The focus is often on improving support for new types of applications — multimedia, mobility, etc.. So we have fewer people looking at the basic structure of operating systems (with some notable exceptions), but more looking at how to make operating systems perform better from a user’s point of view. The use of existing OS platforms also removes some of the barriers to entry for OS research — universities with small OS groups and budgets can do interesting research without having to build an entirely new operating system.
7. 30 years after UNIX was recoded in C, most people still use C (or in some cases a little bit of C++) for the OS kernel. Is C perfectly adequate, or do they see some of the newer languages (C#, Java, or even modern C++ paradigms) being applied to OS design?
Andy Tucker: There have been various experiments in this area; as an example, Sun has developed operating systems in both C++ (SpringOS) and Java (JavaOS). While object-oriented languages offer a number of advantages in terms of ease of development for higher-level programming abstractions, this doesn’t always benefit OS kernels as much as it would user applications. Since the kernel is the piece of software that most directly interacts with the hardware, the benefits of having a simple mapping between the language and machine instructions is often more compelling than ease-of-development features like garbage collection and templates. There are also issues like runtime support requirements that can be extensive, depending on the language. What we often wind up doing instead is taking some of the concepts from object-oriented languages, such as polymorphism, and finding creative ways to implement them in non-OO languages like C.
8. How do you feel Solaris process management technologies like the Fair Share Scheduler will stack up to the Linux O(1) scheduler. Furthermore, has Sun ever attempted to implement an O(1) scheduler for Solaris and if so, what problems/drawbacks they encountered which kept it out of the released kernels.
Andy Tucker: Solaris has actually had an O(1) scheduler for a number of years. The run queues are also per-CPU to maximize scalability. This isn’t a secret, but we haven’t talked about the technology itself much; we’ve been mostly focused on the results.
The “fair-share scheduler” is one of several scheduling policies in Solaris, which control how priorities are assigned to individual processes. This is separate from the scheduler, which handles dispatching processes onto processors in priority order.
The fair-share scheduler allows the allocation of CPU in the system to be divided among groups of processes according to proportions defined by an administrator. For example, on a system running both a mail server and a web server, the administrator might decide that if the system is busy, 2/3 of the CPU should go to the mail server, and 1/3 should go the web server. Although in the past the fair-share scheduler was available only as a separate product (Solaris Resource Manager), we decided that it was important enough technology for our customers to bundle in the core operating system.
9. Sun killed-off the Sun Linux distribution, but are you going ahead with Project Mad Hatter which includes a Red Hat Linux 9.x distro serving as a cheaper thin client for your existing customers? If yes, when is the project going to be deployed and how important the usability of the user interface and OS is? Is Sun going to pay extra attention in Gnome and its better integration to the underlying OS in the future?
Robert O’Dea: Project Mad Hatter is moving ahead as planned with a release targeted for the second half of this year. Mad Hatter provides an integrated client solution for Linux based on open source technology. The graphical desktop will be LSB compliant, which is more important to our customers than the particular open source components provided.
We are also focusing our efforts on the user experience, interoperability, and enterprise specific functionality. Enhancements include a unified Sun look and feel, enhanced Microsoft interoperability [by which we mean Samba integration, ability to access MS file shares and access to Exchange servers], printer management, file management and sync. Additionally, we will be providing enterprise functionality such as system administration and configuration tools, single-sign on and other applications.
As for GNOME, this is an integral part of the Mad Hatter software stack. Sun has been working with GNOME for many years, has made significant contributions to the GNOME community as well as integrated GNOME with the Solaris Operating Environment. With our experience, you can expect that integration with the Linux OS will be tightly integrated.
10. What is the future holds for Solaris 10? What enhancements are in-store in the OS and kernel level? Are there any plans to integrate the Gridengine into Solaris rather than being a separate application?
Andy Tucker: Solaris 10 will have a number of new features that we think are pretty exciting. One is Solaris Zones — this takes an idea that was initially developed for FreeBSD (jails) and extends it to address the needs of our customers. It allows administrators to divide up a single system into a number of separate application environments, called zones, where processes in one zone are not able to see or interact with those in other zones. This means that multiple applications can run on the same system without conflicting with each other, but the administrator only has to deal with one OS kernel for backups, patches, etc..
We’re also looking at ways to improve system reliability and observability. Solaris 10 will include tools that allow tracing not only what’s going on at user level, but also what’s going on in the kernel. So a developer trying to understand why their application is performing poorly can get information from the whole software stack and get a much better picture of what’s really going on. We’re also using these tools internally to improve the performance and reliability of Solaris and other Sun software.
Pretty Cool Interview.
Good stuff…..
-Nex6
…and wants Linux developers to feel the same about Solaris, how come his company is buying int SCO?
Chuck, I would doubt very seriously that Andy Tucker as any influence on the investement disisions of the Sun Board. I though his interview was well ballenced and showed none of the hostility to linux that you have showed to Sun.
And yes, he does seem real balanced as regards Linux.
So he admits that Sun has borrowed the O(1) scheduler GPL code from Linux and put it into Solaris? Forget SCO. Solaris has lots of goodies that should be GPL’d for Linux’s use.
Is this supposed to be flamebait or are you being serious? He said Solaris has had an O(1) schedular for a long time already. Linux only has one in it’s development kernel. Sun’s schedular is not GPL nor is it from Linux. Why must everything be GPL’d for Linux’s use? Why not BSD license Linux code for BSD use? Please don’t start with silly license wars again.
When Andy Tucker was still at Stanford, he published a number of influential papers on the theme of Scheduling on Shared-Memory Multiprocessors…
http://www-cs-students.stanford.edu/~tucker/home.html
His work was cited in the ALS 2001 Scheduler Paper, “CPU Pooling and Load Balancing for Linux MultiQueue Scheduling” aka “PMQS: Scalable Linux Scheduling for Highend Servers”.
http://lse.sourceforge.net/scheduling/
Andy infuenced the development of Linux SMP Scheduling
In their most recent press releases and in their lawsuit against IBM, The SCO Group is effectively make the claim that:
The SCO Group, and only The SCO Group, has *sole* right to sub-license any and all source code from all of the folowing UNIXs; SCO UnixWare and SCO OpenServer, Sun’s Solaris, IBM’s AIX, SGI’s IRIX, HP’s UX, Fujitsu’s ICL DRS/NX, Siemens’ SINIX, Data General’s DG-UX, and Sequent’s DYNIX/Ptx.( SCO’s CEO expanded that list to include the Free BSDs, Mach and therefore Mac OSX, and even Microsoft’s windows )
The SCO Group is also effectively claiming that The SCO Group, and only The SCO Group, has *sole* right to dictate what can be disclosed about any and all internals of any of the above operating systems, or any derivative operating systems.
According to SCO way of thinking, Andy Tucker is also effectively guilty of revealing such details to people who have not signed an AT&T/USL Unix NDA license aggreement.
I wonder how Andy feels about SCO’s attempts to limit future OS development and about Sun’s recent attempts to capitalize on The SCO Group’s FUD?
BTW, The line of argument that Darl MCBride and The SCO Group is using to claim these rights was quashed by the outcome of the USL vs. BSDI lawsuit
http://cm.bell-labs.com/cm/cs/who/dmr/bsdi/bsdisuit.html
The SCO Group cannot have any effective claim to the UNIX source it did not directly write or directly purchase.
The SCO Group also cannot back out of the GPL license that it has knowingly sold and distributed binaries and source under.
The SCO Group is pulling the same trick as that company that is selling plots of land on the moon, a lunatic scam that it appears both Microsoft and now Sun has fallen for.
> So he admits that Sun has borrowed the O(1) scheduler GPL code from
> Linux and put it into Solaris? Forget SCO. Solaris has lots of goodies >that should be GPL’d for Linux’s use.
Umm no, I think what he said was ‘Solaris has had a O(1) scheduler for a while that we developed, we just haven’t made a big deal about it’. Not ‘we stole linux’s O(1) scheduler’. The low level solaris threading code is quite a bit different from linux’s anyway, it’d be sort of hard to cram one codebase into another.
He said Sun has had an 0(1) scheduler for years – not that they borrowed Linux’s. What he meant was that Sun wrote a scheduler that’s in the same league as Linux’s 0(1) years ago.
why does sun have so many r&d people but no one figuring out what to sell? this is why sun is doomed, they spend billions of dollars on things like slowaris and spark then just dump them for linux and x86. but who will buy linux and x86 from sun when it’s so expensive and you can build a machine for less? or buy one from dell
sun needs to get rid of r&d and work on marketing BADLY. at least dumping solaris and spark is a move in the right direction so I am glad someone there has some sence
WHY SHOULD THEY DUMP SOLARIS FOR LINUX???
Sun isn’t after the commodity market, they’re leaving that to companies like Dell and Red Hat
A refresh of Sparc may be in order if they find themselves unable to keep up on the Hardware scene. However, Solaris is an EXTREMELY competitive operating system, and still holds numerous advantages over Linux. In fact, there was so much outcry over initial decisions to dump the x86 version of Solaris that Sun was forced to bring it back, and even released some x86 hardware of their own for it.
Solaris and Sun are not going away anytime soon.
Quote:
“at least dumping solaris and spark is a move in the right direction so I am glad someone there has some sence”
Are you kidding? I love linux and use it at home, but x86 hardware isn’t even comparable with what you can buy from SUN or HP, SGI & IBM for that matter. Linux is Great, and I love it but please. If you’ve ever worked with Sun (I don’t know if you have) hardware, you would be amazed with it. No it’s not the best desktop, but we use it on MISION CRITICAL enviroments.
SGI has recently released a SGI Altix 3000 series w/ true scaling 64-processor’s for Linux (the first of it’s kind). But SUN is tried and true. I don’t see them going away anytime soon.
rowel is clearly a tool. do not feed the monkey by replying.. he has a strict diet of BS in which he thrives.
That´s what I call an interesting interview. I am quite happy that Sun still has some of the smartest people in the I.T. field working for them.
Also Andy Tucker is pretty much open-minded about Linux, which is great.
I don’t think Sun is borrowing as much from linux as linux is from solaris.
The o(1) scheduler and per cpu run queues have existed in solaris for many years. Andy tucker said it in the interview and it is also public knowledge read Solaris internals by Jim Mauro and Richard McDougall.
The kernel slab allocator in linux is also borrowed from Solaris read the comment in slab.c. It is borrowed from Unix internals by Uresh Vahllia who describes the Solaris allocator in his book and it also credits Jeff Bonwick’s paper presented at USENIX in 1994 Jeff Bonwick is also a Distinguished Engineer at Sun.
The 1:1 thread library is the one Andy is describing as the second thread library in Solaris 8 and the default in Solaris 9. The 1:1 library is in development for linux 2.6 kernel.
The O(1) scheduler and the 1:1 thread library are being touted very much by the linux community but Sun just does things quitely.
I’ve always thought Solaris to be the most roubst UNIX out there.
What a great interview. You can easily tell Andy Tucker is educated and enlightened. It’s one thing to Wizard(highest order of coding or a coding genius), it’s another to be well mannered and cultured. I really enjoined his objective views on operating systems in general. I learned a thing or two. This a welcome relief from the Free BSD interview posted days ago. *rolleye*
Regards,
Mystilleef
Lately I have been wondering about HP-UX and the direction that HP is taking with it – it seems like HP is not serious about HP-UX being a desktop workstation or even a mid-sized server. HP dropped GNOME and the tools available through official channels are pretty bad (this does not include the Porting Archives, which are not officially supported).
It is nice that Sun is providing an alternative Unix workstation. I love Linux, but commercial Unix is still more scalable and more reliable – at least for now.
What a classic guy! Nice interview. I think Sun should continue developing Solaris on Sparc for specialised high end computing, but I don’t think Solaris on Intel makes much sense. Are they going to make money on the OS? Or selling Intel hardware? Its much better for them to use Linux on on Intel, but beyond that, Sun needs to do something huge and radical to remain in the long-term business. I don’t know what they could possibly do on the hardware end, Intel is already set to win that battle. But they are a great software engineering firm, so maybe they could plan something along those lines. Too bad they don’t make money from java.
i liked it a lot, Slaris = high end, linux = cheap low end that’s the way I see it anyway.
Nice interview.
But looking at the screenshots makes me remember why I’m running Mac OS X. BTW, too bad they dropped all their work on the NEO (OpenStep based) desktop environment and object-oriented kit… Now they seem stuck in the late 80’s, GUI wise.
Gnome 2 for Solaris 9 is available as a download package. Solaris 10 will switch to Gnome completely, Sun has said.
> Now they seem stuck in the late 80’s, GUI wise.
i remember reading that Solaris 10 will be the default desktop. i’m not saying that it will be better than OS X tho
I like this guys attitude. Was polite and gave credit to Linux and its achievments. It would be nice if all the Linux zealots took note.
I’d think about picking up a SPARC processor, if the next versions of Linux ran on it without a hitch. I want kernel 2.6!
Also, it is true that current platforms have evolved by taking on object oriented features such as micro kernels and a layered design. The C language can take on object based functionality (classes) and some object oriented functionality (inheritance). What I don’t like about current platforms though is that they are inflexible. I want no more than one, two, or three developers to be able to write their own platform, or work with distributed objects and quickly assemble a platform from those distributed components. I think that comilers with A.I. the ability of write code, and analyse architecture, recognize architectural designs and patterns, the ability to reconstruct patterns out of binary digits, those are the tools that humans need in order to be able to take on such large development projects as individuals. We need architecture readers and than through understanding design, the human can guide those machines to create new code, and maybe to collaborate with other machines.
RE: linux_baby (IP: —.library.utoronto.ca)
Read The Friendly F*cking Site ( http://www.sun.com )
One doesn’t need to be a rocket scientist to see that the majority of the efforts appeart to be directed towards getting the SUN One Software stack online. That is where the money is made. Money isn’t made selling something as boring and rudamentry as an operating system.
RE: Chris Parker (IP: —.mpan.com)
Are you surprised about HP ignoring HP-UX? I’ve said it again and again, HP like Compaq and Dell are just Microsoft sycophants that simply mass produce Intel servers running Windows for the unwashed masses. Just look at the intellectual property Compaq squandered when they bought Digital and Tandem. Look at the lack of any innovation on the part of HP in terms of their software line up.
RE: Anonymous (IP: —.cg.shawcable.net)
What is wrong with Solaris on UltraSparc and no, Swolaris is nothing more than troll bait.
… that put real effort into the whole med-upper range of the computing market… Sol9 on US3 is beautiful, but alas out of the price range of most of us, but then again Sun isn’t a commodity PC seller…
Sun is the Apple of the high-end market…
That was one of the best interviews I’ve read in a while. Also, Andy seems like a real great guy, a credit to Sun for sure.
Great interview, well thought out and balanced. This guy knows his stuff! On a side note, those screenshots remind me of just how ugly CDE is 🙂
It seems as if everyone is just jumping on to the Linux bandwagon. I for one would love to see Solaris developed on x86-64, Power4, and Itanium 2. If they could accomplish that then a company could roll out any hardware and have the same OS interoperate throughout. I know Linux is getting close to this and NetBSD can pretty much already do this, but I would like to have the choice. I am very fond of Solaris and I think it is currently the best OS out there. I just wish sun would seperate the hardware from the software. I can’t forsee the engineers suddenly deciding to stop making a SPARC version! Other OSs run on the SPARC as is, so they shouldn’t worry about the adoption of the SPARC, it seems to me that it would survive on its own merits (or lack thereof)
Just a thought…
If Sun or Microsoft was smart, than they would join IBM and support Linux. Incorporate any good features of Solaris into Linux and help Linux run on Ultra Sparc. They would make a fortune.
hahaha…
I guess you haven’t seen Tru64 yet 😉 the ugliness is almost unbearable.
When I was at Sun in the 90’s a PowerPC version of
Solaris actually got into Beta around the 2.5 days.
It would have been interesting to see Solaris get
out on other platforms.
They should combine Solaris and Linux and than make a ton of money and do it now before it’s too late, because Linux will bury Solaris.
that trolls and broadband go together like fanboys and PC’s?
OK, I can’t help but say this. I am very tired of all of Linux sycophants preaching about how Linux ownz and rulz all other Operating Systems cuz its “Open Source”… or its the one true OS. It is like a religious statement, they have so much of their self-esteem tied to their OS it is beyond pathetic.
I like Operating Systems for their own merits, not because every magazine and “guru” I know spouts of about its perfection. I love the fact that BSD is developing their KSEs and fine-locking on their threads, I like the fact that we have Linux developers concerened about I/O scheduling. But the point is Solaris has very few flaws and a great deal of Merit. I don’t want to see all of these UNIX branches come back into a single entity. It would totally thwart the idea of many advances occuring in parallel, we would be stuck with one source branch and we would have no choice over what we want to use. Choice and competition in technology is ALWAYS a good thing.
<The following is supposed to be a joke>
If linux had every had Windows success it would have exterminated all other OS-s just because of its unparalleled users zealotry
This was a nice and interesting article.
Not that I actually use Solaris, I do not have a Sparc or a x86 which is supported… but the article was nice.
I read this, don’t think much of it, and then realize I raced bicycles with this guy at Stanford. Hehe… cool. He’s a good guy.
Try it – Solaris/x86 is waycool. Just wish it were qual’d on more boxes. And had better RAID support.
But let’s face it – Linux == RedHat. And their new pricing sucks (as does their quality, but they always seem to sidestep that by saying “it’s a community product.” – if it’s a community product, please send me my checks for fixing the freakin’ bugs).
At the end of the day, HP/UX is dead, and we’re moving to Solaris. I’m a happy campah…
Great article. Enjoyed reading it.
I’ve been using Solaris since the bad old days when it was really flakey. Now, I like it as a solid platform to get real mission critical work done, but I have two small whinges:
1) It seems to be accumulating too much crap. I know I should install the minimum system and add packages I need, but when time is short, I don’t.
2) When something is added to Solaris (apart from features designed for workstations) can somebody in Sun please make sure that it is easy to administer through the serial port connected by some method to a terminal of unknown type.
I don’t know how much Sun hardware sold as workstations vs. as servers but I would think that the future of Solaris is much surer in the server market.
I not suggesting the Sun abandon Solaris workstation customers (I use one myself) but the change of emphasis would be appreciated. I’m cheerfully willing to be shouted down over this, but what do you people think?