One of my popular articles shortly after I joined OSNews in 2001 proved to be “the big *BSD interview” and so it is only appropriate to end my serving at OSNews with a similar theme. Today we are very happy to host a Q&A with well-known FreeBSD developers John Baldwin, Robert Watson and Scott Long. We discuss about FreeBSD 6 and its new features, the competition, TrustedBSD, Darwin etc.
1. Tell us more about SMPVFS and its significance.
John Baldwin: The SMPVFS work is a task to add fine-grained locking to the VFS layer of the
kernel as well as the UFS and nullfs filesystems. The VFS layer provides the
abstractions in the kernel that describe file objects. Each filesystem
provides a VFS “driver” to manage the files on a disk device according to the
design of that filesystem. Adding fine-grained locking to VFS and the UFS
filesystem allows more concurrency in the kernel, especially with workloads
that include disk I/O.
Robert N M Watson: One of the other nice benefits to the SMPVFS work is that with our fully
preemptive 6.x kernel, not holding the Giant lock over the file system
code lets the file system code not only preempt lower precedence kernel
threads, such as background crypto operations or file system operations,
but be preempted by more timing critical code, such as sound card
interrupts, network I/O, and so on. So this isn’t just a win for SMP, but
a win for UP also. The SMP wins are impressive though — Kris Kennaway
has recently been benchmarking package builds, a very VFS-intensive
workload, on 12-CPU sparc systems, and all the scalability we’d hoped for
Scott Long: SMPVFS also reduces contention for storage drivers that are still under the Giant lock and increases the possible parallelism between these drivers and the filesystems above them. Kris’ tests are a very good example of this; even though the SCSI subsystem and most of the ESP driver are still under the Giant lock, performance still scaled well.
2. We recently read that the fix for the Hyper-Threading vulnerability is considered non-trivial. Why is that?
John Baldwin: The issue found with HT is that the two logical CPUs on a single core share
the same caches and as a result there are ways for one logical CPU to spy on
the activities of the other CPU in the same core. The proposed fixes involve
ways of guaranteeing that all of threads on a single core are all allowed to
spy on each other. For example, one policy is that only threads with the
same user ID should be allowed to run together no the same core. The problem
is that right now FreeBSD treats logical CPUs as separate CPUs and schedules
available threads on the first CPU that becomes available. It would be a bit
of work to make the scheduler more aware of logical CPUs and to schedule
threads with respect to UIDs, etc.
Robert N M Watson: It’s worth observing that this is a serious vulnerability across a range
of operating systems, not just FreeBSD. If you allow untrusted users on
the same system as an SSH daemon, you’re at risk, which affects everyone
from desktop users, to ISPs, to military end-users. It’s also a very hard
problem to solve — we’re looking at it from the perspective of improving
the scheduler, bringing in OpenSSL updates to limit timing attacks, and
obviously we’re hoping that CPU vendors take this opportunity to explore
how to harden CPU architectures against this sort of attack. Because this
vulnerability isn’t just about scheduling, crypto, or hyper-threading, a
lot of hard work will have to into a long term solutions.
3. Why is TrustedBSD an important piece of the upcoming FreeBSD 6? How does it compare to SE-Linux or OpenBSD?
John Baldwin: Robert is probably the best to answer this. From my understanding, TrustedBSD
is a superset of SE-Linux as it includes things like ACLs for files as well
as the MAC framework that allows for arbitrary MAC policies to be hooked into
the kernel. One such policy being developed is a port of SE-Linux called
SE-BSD. There are also other policies available for the MAC framework
Robert N M Watson: TrustedBSD elements have now appeared in 4.x, 5.x, and 6.x. 5.x brought
in many of our most significant features — some were infrastructure to
support our goals, and others were security features we’ve been targetting
as primary goals. Supporting TrustedBSD features included OpenPAM, GEOM,
UFS2 with extended attributes, and a lot of kernel and user space cleanup
for access control. It turns out that our extensive SMP work in 5.x was
also very important, as the mature kernel synchronization architecture of
5.x allows us to generate access control decisions in many code paths that
would not easily have supported it in 4.x, such as in software interrupt
paths in the network stack.
The direct feature set in 5.x included the TrustedBSD MAC Framework, which
allows compile-time and run-time extension of the FreeBSD security model,
a set of sample system policy modules, such as Multi-Level Security, Biba
Integrity, and a variey of hardening policies, and also support for Access
Control Lists (ACLs). So the TrustedBSD work was really key to the 5.x
release line, especially if you include the supporting features I listed
In 6.x, many of the experimental features from 5.x are considered
production quality, and extended in a variety of ways. Two of the biggest
“new” projects are SEBSD, a port of NSA’s FLASK/TE implementation from
SELinux and its predecessors (DTOS, FLUX), and support for CAPP security
event audit, which is derived from OpenBSM, which is in turn derived from
Apple’s CAPP Audit implementation. SEBSD will be an add-on distribution
on top of FreeBSD 6.x, and allow the authoring of fine-grained Type
Enforcement (TE) policies similar to those in SELinux. OpenBSM provides
us with a implementation of both kernel event auditing, as well as a
BSD-licensed user space audit library implementing Sun’s BSM audit file
format and service API. Adding support for Audit really fleshes out our
trusted operating system feature set, and NSA’s FLASK/TE provides a
powerful policy language to for tuning system security for specific
applications and configurations.
These are security features that our network appliance, security,
financial, and military consumers will appreciate greatly. They’re also
features that end users will be able to use to customize and monitor
security operation of their systems in a manner currently unsupported by
any other open or closed source operating system.
4. What other new features are we going to see on FreeBSD 6.0?
John Baldwin: Support for WPA security for 802.11. The tty subsystem has been reworked in
preparation for adding fine-graind locking in the future. FreeBSD/i386 and
FreeBSD/amd64 now use the timer in the local APIC to drive the various kernel
clocks. Support for several different CPU frequency drivers such as
SpeedStep and PowerNow. Support for hardware performance monitoring counters
on i386, amd64 and ia64. The if_bridge(4) driver from NetBSD has been merged
in as well. There are lots of other things that I’m sure I’m missing, but
more details can be found at http://www.freebsd.org/relnotes.html in the
release notes documents for FreeBSD-Current.
Robert N M Watson: You’ll also find features like read-only support for reiserfs, and
substantial performance optimization and SMP cleanup from our first design
for fine-grained SMP in 5.x, compiler suite upgrades, significant upgrades
to our 802.11 code to support features like WPA and authentication/crypto
plug-in frameworks, and complete integration of IPv6 into the ipfw2
firewall (previously IPv6 was supported separately). There’s also a
substantial re-write of the libthr threading library by David Xu, which
offers significant performance enhancements for 1:1 threading users.
Scott Long: The APIC change for i386 and amd64 that John mentioned is actualy a very
important feature. Many motherboards, especially newer Athlon64 desktop
boards, cannot even boot FreeBSD 4.x or 5.x, but work very well under
6.0 with these changes.
PowerPC support is something that I consider FreeBSD 6.0’s best kept
secret. Installing FreeBSD/ppc on a Mac is a little cumbersome (though
not much different from NetBSD or Linux), but once installed it works
very well and runs X Windows and most apps. I run it on my MacMini,
and there is effort underway to provideWe are looking at possible
providing install ISOs for it for 6.0.
We are also looking at integrating DomU support for Xen. Xen is a very
exciting piece of technology for both developers and for large
datacenters, so supporting it is a high priority. We are actively
looking for help with porting full Dom0 support so that FreeBSD can be
fully self-hosted in the Xen environment.
5. Are there plans to move FreeBSD’s public version control system to another system, away from CVS?
John Baldwin: Probably not anytime soon. The current CVS + CVSup infrastructure that the
FreeBSD Project has is a great benefit and would require a large deal of work
Robert N M Watson: Revision control is always topic of active discussion in any open source
project, and especially projects in which revision control is so essential
to its operation as in the FreeBSD Project. FreeBSD has some pretty
intensive requirements for revision control — with several hundred active
developers working throughout our CVS tree, not to mention all the
external contributors, the FreeBSD CVS repository sees over 50,000 comits
a year. We have our eyes on where to go in the future, but whatever we
select when the time comes, it will have to support today’s workload, and
tomorrow’s as it continues to grow.
As John alludes to, one of the ways we make CVS go as far as it does is
through the use of extension tools for replicating our CVS repository to
thousands of end-systems, avoiding read-only operations having to go into
a single central repository. In this manner, FreeBSD developers can avoid
hitting the central repository with anything other than commit operations.
Many sub-projects within FreeBSD use other revision control systems to
augment CVS, tracking the central repository, and then merging changes
back to CVS at mature snapshot points for the sub-project. For example,
developers frequently make use of CVS, Subversion, and Perforce — for
TrustedBSD, we use Perforce with cvsup export of our repository, for
6. We hear that some cool bits from Darwin will be backported to FreeBSD (and the other way around). Please tell us more about these… bits and what they do. 😉
John Baldwin: I personally do not have any plans of porting anything from one OS to the
other. It might be interesting to port launchd from OS X Tiger to FreeBSD as
a replacement for init and inetd though.
Robert N M Watson: The TrustedBSD Audit support originated in large part from Mac OS X, and
we really appreciate Apple’s work with us to develop audit support, and
their support in getting it out into open source. One of the outcomes of
this will be our (TrustedBSD’s) continuing maintainership of OpenBSM, a
bundling of the libraries, documentation, and command line tools, which
will be portable across a host of operating systems including FreeBSD,
Darwin, and Linux. This sort of arrangement can be a strong motivator for
companies like Apple to release software under open source — we’re
already preparing bundles of documentation and feature enhancements that
we hope they will be able to adopt back into Mac OS X.
7. Apple announced recently that they are swiching to x86. What does this mean for FreeBSD and other open source OSes?
John Baldwin: I do not currently foresee it causing any changes in the free software world.
Robert N M Watson: Apple’s work on Mac OS X is very impressive — they’ve successfully drawn
from both their extensive experience in UI and application design, and a
host of open source origins, including Mach, BSD, FreeBSD, KDE, FSF’s tool
chain, and Python, not to mention their closed source components, such as
windowing system and application suites. others to create a convincing
and powerful desktop product. Part of the root of their success is in
interacting with and building on open source products — be it FreeBSD
network stack code, or the KDE web browser components. I don’t see this
going away with the move to i386, and hopefully we’ll see even more
contributions back to the open source community.
8. Looking back to the last 3 years, how do you evaluate FreeBSD’s popularity? Do you see FreeBSD’s status declined or getting more popular among users and developers?
John Baldwin: I think it is growing. It seems that every time I sit down to talk to other
FreeBSD folks at conferences I hear about another company that is using
FreeBSD as part of a product or service that they sell. The volume of e-mail
going through the lists also seems to be steadily growing, which can be a bit
of a headache when one is trying to keep up. 🙂
Robert N M Watson: Likewise — more and more companies are adopting FreeBSD as the foundation
for their products, and FreeBSD continues to see wide-spread success in
the web services, ISP business, etc. The FreeBSD Project has never really
done a very good job at advocacy or monitoring its user population in a
formal way — we’re software engineers, and until recently, we didn’t even
have a marketing team. However, the scope and influence of FreeBSD users
in the computer world is easily visible from their contributions to the
project. You don’t have to look far to find out that FreeBSD systems
process over a trillion dollars in banking transactions in the US every
year, at the root name servers, or many of the most significant internet
service providers and web service providers.
One of the things I like to point out about FreeBSD is the longevity we
see in our contributors — we are one of the few open source projects that
can demonstrate a code history going back almost 30 years, and who have
active developers who have worked on that code base for much of that time!
I began using FreeBSD in about 1995, and joined the developer team in 1999
— despite being on the project six years, I consider myself a relative
newcommer. Many commercial software companies would count themselves lucky
if their senior engineers had half that time working on a project. Daily
contributors range from high school students to tenured university
professors, and our developer base continues to grow.
Scott Long: The stats that I always look at are: CVS commit volume, mailing list traffic volume, bug database activity, and new committer activity. 9 new source tree committers have been added in the last six months, which brings the total number up to an impressive 219. This doesn’t count the numerous others who contribute to just the ports and/or the docs tree.
9. Many have commented on the inability of open source desktop environments to provide a coherant, integrated solution that would satisfy modern users used to Mac OS X or even Windows. Where do you pinpoint the problem?
John Baldwin: I think that free software developers are usually driven by churning out code
to solve the problems that they face and building tools for their own use
whether it be for personal use or work related. End users want a product
that solves their needs and those needs are not a subset of a developers
needs. Some developers may try to solve needs that they don’t have, but they
still are not able to solve needs that they don’t know exist. 🙂 Companies
such as Apple and Microsoft employ more than just software engineers. They
also employ folks who understand user interfaces and people who try to
determine what problems consumers want solved.
Robert N M Watson: The open source world is an interesting place — some very important
pieces of innovation in the UI arena have been developed as open source,
from X11 to NCSA Mosaic, and more recently Firefox. Parts of the academic
world performing research into the user experience and human interactions
have long benefited from and contributed open source. However, I do feel
that the current top-to-bottom open source desktop stack is playing catch
up with work by companies like Apple and Microsoft. Part of the
difficulty here has been in developing a complete object model stack, not
to mention a mature application suite. The contribution of the
OpenOffice.org source code by Sun was an important step in fleshing out
the open source desktop, but I think it’s telling that OpenOffice.org
still needs to ship with many basic components that are considered
standard on the commercial desktop systems. Until the open source world
can make it easy to develop integrated desktop applications, we’ll
continue to see a lack of mature ones. All this aside, we’ve come a long
way in the past six years — software products such as X11, X.org, KDE,
and GNOME provide usable user interfaces that are miles ahead of X11 and
fvwm in 1999.
10. How are you getting on with the journalling extension to UFS?
John Baldwin: Ask Scott about this one. 🙂
Scott Long: The UFS Journalling project was listed as a candidate project for the Google Summer of Code. The response from applicants has been overwhelming, and I’m looking forward to working with some very good
people on it. While it might take more time than the Summer of Code timeframe to complete, I expect it to be working and stable in time for FreeBSD 7.0, and possible available for later 6.x releases via patches.
11. Are you paid to work on FreeBSD fulltime or is it a side project/hobby for you?
John Baldwin: I am very fortunate to be paid nearly full time to do FreeBSD work. I
probably spend about 80+% of my paid work time working on FreeBSD.
Robert N M Watson: It varies — I’m a Senior Principal Scientist at SPARTA (previously
TIS/NAI/McAfee Research), and what I work on depends on the customers I
find and the work I find. Over the past few years, much of my work has
been on the FreeBSD or Mac OS X platforms, particularly as relates to work
on operating system security. Other work, such as SMP network stack
support for FreeBSD, I do in my copious (?) spare time. The open source
business world took some bumps with the dotcom crash, but it’s easy to see
that it is experiencing the same recovery that’s visible in the closed
source high tech world. Whereas three years ago it was hard to throw a
stone at an open source event without hitting a recently laid off open
source developer, today recruiters are easily seen on and off the mailing
12. What is your opinion on PC-BSD and OpenSolaris? How’s your cooperation –if any– with OpenBSD & NetBSD these days?
John Baldwin: PC-BSD: It’s great to see someone taking FreeBSD and extending it to be even
more friendly to desktop end-users. OpenSolaris: No opinion as of yet; haven’t had time to look at it.
*BSD: FreeBSD recently imported if_bridge(4) from NetBSD and dhclient(8) from OpenBSD so we certainly have no problem taking code from our sibling projects. I’m not as familiar with any code going in the other direction as I really only have the time and brain capacity to focus on one open source operating system.
Robert N M Watson: Sun has a long history of innovative operating system work, and it’s
really great to see them starting to get parts of Solaris out under an
open source license. Sun has obviously long depended on and interacted
with the open source community, and their ability to work this through Sun
Legal is impressive :-). An interesting question for Sun will be whether
they can build a community around OpenSolaris that extends beyond
Sun-employed developers in the same way that Apple has started to see
success in building a community around Darwin.
A point I was reminded of recently by Mike Smith at Apple is that there
are really two things we mean by open source: we mean the
licensing/distribution of software, but also the community that is built
around it. Key to the success of an open source project is both of these
elements, and creating and maintaining that community requires easily as
much investment as the software development itself. However, once
achieved, the pay-offs for everyone involved can be huge.
With respect to other open source projects — FreeBSD remains both a large
producer and consumer of open source, generating open source foundations
for many other open source projects, as well as consuming the output of
many other open source projects. In the closed source world, competition
is a powerful force for change, as it is in the open source world.
However, in open source world, the opportunities for collaboration and
cooperation are far greater than in the closed source world, so we have
the opportunity to share ideas and code much more easily, and as a result
benefit from that exchange. A point that is sometimes lost when the “Open
Source Community” and “Open Source Software Stacks” are discussed is the
importance of both competition and cooperation in its success. Open
source makes possible a market place of ideas and the incarnation of those
ideas in source code. We’re all better off for the existence of many
competing (and cooperating) software projects, and let’s hope we never
move away from a world where that is the case.