Federico Biancuzzi interviewed FreeBSD Core member Scott Long about the SMPng technology, the current implementation status, future goals, and plans.
FreeBSD’s SMPng
About The Author
Eugenia Loli
Ex-programmer, ex-editor in chief at OSNews.com, now a visual artist/filmmaker.
Follow me on Twitter @EugeniaLoli
The Linux SMP approach appears to be focused mainly on using spinlocks for thread synchronization. This has the benefit of it being fairly easy and straightforward to lock code, but results in CPUs wasting time spinning when they could be doing other useful work.
The premise is that the cost (in serial execution speed, cache and TLB impacts and associated bus traffic) of a spinlock can often be much lower than a blocking lock if they are only held for short times.
In practice this is often true (for Linux).
Most uses of [Linux] sleeplocks also block interrupts while the lock is held, so latency increases.
It is true Linux doesn’t have any sort of interrupt threads or interrupt priorities (it would be largely a waste of time at this stage, since it doesn’t claim to be a hard-rt OS). So holding any spinlock does increase scheduling latency, and holding an interrupt safe spinlock will increase interrupt latency. However, these latencies are watched pretty closely in Linux, and are generally well under a millisecond.
There is ongoing work to implement kernel preemption in Linux to help mitigate these issues, but it seems to often be the source of controversy and problems.
Kernel preemption has been in the Linux kernel for a long time (much of the 2.5 cycle). It has not been a source of problems for a long time.
But regardless of that fact, kernel preemption does nothing to mitigate these spinlock latency issues. There are various works that implement realtime semaphores, convert most kernel locks to semaphores, implement interrupt threads, etc. but this is in order to work toward hard realtime guarantees as opposed to performance or general purpose computing. As such it would only ever be an optional compile time setting.
i read the interview and felt that there was a negative morale coming across. the interview is full of “yes, it works better in xBSD or linux, but we’re working on it…”
freebsd has a great tradition but i really think they are feeling that Dragonfly and Linux and NetBSD has done their things and done them well. i feel they have fallen behind in the momentum behind innovations. (this last sentense is carefully worded because they did innovate – just lost the drive i feel).
hurray! for netBSD.. from the old but reliable friend … now comes a clean mean and very well performing system. congratulations to them for keeping at it while the media hyped freebsd and linux.
so what now for freeBSD? i think it has been upstaged in various domains and needs to rethink what its role is.
ps – i’m a great fan and appreciate all their work. i’m just commenting on objectives. i owe a great debt to all thse open source OSes!
That’s not news, my friend. Mike Smith already talked about the ‘too much politics’ problem when he resigned years ago. What does FreeBSD want to be? Portable? That’s NetBSD’s place. Secure? Beaten by OpenBSD *and* NetBSD. Scalable? Not with that model. The need more smp and vm clued hackers to make that work. Maybe if people like Poul could get their head out of their ass and the core team would admit their piss-poor management in the last 2 years, maybe, FreeBSD could be saved. Or they could just jump ship and join DragonFly
This sounded very Pro linux to me. FreeBSD sounds like it needs a killer app. Something new and fresh and lift that low feeling in the pitt of their guts.
It’s interesting — if you read comments from some of the FreeBSD Project’s leading developers — John Baldwin, Scott Long, Robert Watson, etc, every time they make positive claims about FreeBSD, they’re always careful to couch them in terms like “… for this application”, or “… as measured in the following tests”. The system they build is excellent, but the problem is that at heart they’re too honest. They can’t just pretend the details doesn’t exist — vis “No root compromises!” [in any release of OpenBSD 24 hours after release], or “NetBSD is faster than FreeBSD!” [unless you turn on SMP or run a real-world application, like a web server], “DragonFly has better SMP than FreeBSD!” [as long as you don’t mind waiting four more years for a functional prototype, and assume FreeBSD will stand still in that time], and “Linux is better than Windows!” [if you think not having decent applications makes something better].
If the FreeBSD Project fails because they can’t lie, that is a sorry message for open source. Isn’t the theory that open source lets us, the OS-savvy people, make a better choice?
The premise is that the cost (in serial execution speed, cache and TLB impacts and associated bus traffic) of a spinlock can often be much lower than a blocking lock if they are only held for short times.
In practice this is often true (for Linux).
One thing that I forgot to mention in the interview was the use of adaptive mutexes in FreeBSD. This helps balance the advantages of spin locks and sleep locks. You’re absolutely correct that it makes no sense to a task switch if the lock is only going to be contested for a few ticks. Adaptive mutexes deal with that.
It is true Linux doesn’t have any sort of interrupt threads or interrupt priorities (it would be largely a waste of time at this stage, since it doesn’t claim to be a hard-rt OS). So holding any spinlock does increase scheduling latency, and holding an interrupt safe spinlock will increase interrupt latency. However, these latencies are watched pretty closely in Linux, and are generally well under a millisecond.
A millisecond is a really long time. Think of it as only being able to recieve 1000 packets a second, or do 1000 disk operations per second. That might not be noticable on a desktop, but is completely useless for a server. There are people using FreeBSD to bridge (i.e. recieve, process and send) 2.5 million packets a second. Any latency at all would render that impossible.
Kernel preemption has been in the Linux kernel for a long time (much of the 2.5 cycle). It has not been a source of problems for a long time.
I recall seeing a number of posts on the LKML in the past year from high-level people making very moving arguments against preemption in Linux. But this isn’t a slam against Linux at all. Preemption adds quite a bit of complexity, and it’s something that we had to turn off for FreeBSD 5.3. It’s one of those “last 5%” things that is really hard to do, but gives you an edge in the end. It’s also quite useful for quickly triggering bugs that are otherwise hard to diagnose but bite you at inopportune times (and thus allows you to fix them and make the system more reliable).
But regardless of that fact, kernel preemption does nothing to mitigate these spinlock latency issues. There are various works that implement realtime semaphores, convert most kernel locks to semaphores, implement interrupt threads, etc. but this is in order to work toward hard realtime guarantees as opposed to performance or general purpose computing. As such it would only ever be an optional compile time setting.
If your kernel synchronization policy revolves around spinlocks, then you are right that preemption doesn’t do a lot to help. You’re basically making the case that FreeBSD is right; converting spinlocks to semaphores (i.e. sleep locks), implementing ithreads, etc. FreeBSD is finally getting to the point where Giant is off enough of the kernel that we can start making intelligent observations on how well this strategy works, and we are starting to see a lot of benefit.
I think it is this type of modesty and pragmatism that first attracted me to FreeBSD. I must admit, I do like the goals of DragonFlyBSD, and hope that their successes will be used as a model for FreeBSD’s development, but at the end of the day FreeBSD and its community is more about the things that I like – honesty, patience, confidence, and pragmatism.
Then you must have been using a different FreeBSD. While most of the FreeBSD developers are nice, there are a few vocal ones (mainly des and phk) who have managed to piss off an incredible amount of people over the years.
I loved the article. The interview was great. I didn’t feel like it was political or anything and I dont see why people need to make it like that.
I’ve always liked FreeBSD even if I’m using Debian for my dekstop machine because I like its simple/short way to install packages and to keep it up-to-date.
That said when I installed 5.3 on my notebook the cooling fan ran continuonsly even if at login prompt without X.
I didn’t research why it happened so I installed 4.10 and I didn’t have that issue on the same H/W.
From the interview it seems that 5.3 *doesn’t* perform too well with single-CPU computer.
Precisely:
>These tests were apparently only done on UP, so they show
>the cost of the SMPng locks on FreeBSD 5.x. There is quite
>a bit of work going on at the moment to address this
So it seems correct to say: “if you have a single-CPU then go with 4.10, right ?”
I’m not techinician and I’m a simple end-user.
Eugenia, please, can you stop the HawkisnOS troll (not you tim hawkins) – who now poses as Perez-Gilaberte? See:
http://slashdot.org/comments.pl?sid=125640&cid=10529101
Quite frankly, his rants against des and phk are becoming quite boring. Thanks.
I concur with molnarcs. Eugenia, could you please get rid of the abusive messages of this guy?
This HawkinsOS troll has been uttering insults and personal attacks for months. Even if the people he attacks probably don’t give a damn about it, I don’t think it’s fair to let him be abusive.
I am a user, and I do not concern myself with the dirty laundry you bring out, so I feel that my opinion is just as valid as any.
Guys, what ARE you doing with your boxes? Everyone is like ‘oh, one millisecond is waaay too much!’ – for what? Are you running any scientific calculations, multithreaded 3D rendering or what?! I mean, in Gnome or KDE (which most of people use anyway) on generic set of apps you won’t notice any difference, or almost any difference.
Guys, what ARE you doing with your boxes? Everyone is like ‘oh, one millisecond is waaay too much!’ – for what? Are you running any scientific calculations, multithreaded 3D rendering or what?! I mean, in Gnome or KDE (which most of people use anyway) on generic set of apps you won’t notice any difference, or almost any difference.
Yes you are absolutly right, but what kind of quality you want to have??? An OS that can only run light desktop apps or a really good one where you can do whatever you want to and prety fast?
I said I liked the article. Do you no sense at all? Do you even read http://www.hawkinsos.com? Grow Up
You need to stop being abusive toward me and posting such horrid comments in nonrelated discussion. Why must you bring up things that other people say and blame them on me.
you disguist me.
Somebody has been spamming and trolling in news sites and mailing lists for months, advertising HawkinsOS and defaming FBSD developers at the same time.
Since this guy’s “promotional activity” (provided it’s not you) didn’t do any good to your business, I’m surprised that you are protesting against me and molnarcs, instead of protesting against him.
We just assisted (along with many other people) to the sad show of somebody “promoting” your OS AND trolling at the same time. He went on for *months*, saying he was you. What should we think?
I really suggest that you find this person (again, provided it’s not you), and make him stop.
the advertising and promoting worked! i think i’ll replace all the OSes on my servers and workstations with HawkinsOS – the suppport for it is excellent and it has a history of technical innovation. i know i am in good hands – its management team is very well reputed.
I have no idea who this person is nor do I want to find out who this person is.
I think it’s pretty wrong to fall subject to someone’s impersonation of another.
This is really facinating to me. I have never had anyone do this to me and It makes me feel that someone is afraid of HawkinsOS actually be doing it so to me it makes finishing HawkinsOS more worth-while. It’s all stirred up so much controversy it’s weird. Why would someone go spend their time and effort to do this? I have no idea.
Let me say this though, I have worked with other companies that have had evil people impersonate them and send spam and troll and what not before I have learned that it’s near imposible to track them down unless if you have the funds and resources to do so and my company is very tiny and we don’t have legal resources to do that. Any help from you would be helpful to me.
I honestly thought that a few OSNews readers didn’t like my opinions I posted on OSNews at times and were afraid of HawkinsOS for some weird reason and decided to make those posts and then counter them with that “Take a hike” thing and then blast me on here for their own weird reasons.
All very facinating to me and I don’t know how to stop it. E-Mail me any suggestions.
– end of discussion on that issue (unrelated to this thread)-
thanks
>>The premise is that the cost (in serial execution speed,
>>cache and TLB impacts and associated bus traffic) of a
>>spinlock can often be much lower than a blocking lock if
>>they are only held for short times.
>>
>>In practice this is often true (for Linux).
>
>One thing that I forgot to mention in the interview was the
>use of adaptive mutexes in FreeBSD. This helps balance the
>advantages of spin locks and sleep locks. You’re absolutely
>correct that it makes no sense to a task switch if the lock
>is only going to be contested for a few ticks. Adaptive
>mutexes deal with that.
Sure they have advantages. But they are not the be all and end all solution either.
For example, if it is a short held lock, why not just use a spinning lock? If it is a long held lock, why not use a sleeping lock?
If you don’t know, then you should. If it is really that variable, then any time your adaptive mutex switches to blocking, you not only have had the performance disadvantage of spinning, but also task switching. So it doesn’t only have the best of both worlds, but also the worst.
>>It is true Linux doesn’t have any sort of interrupt
>>threads or interrupt priorities (it would be largely a
>>waste of time at this stage, since it doesn’t claim to be
>>a hard-rt OS). So holding any spinlock does increase
>>scheduling latency, and holding an interrupt safe spinlock
>>will increase interrupt latency. However, these latencies
>>are watched pretty closely in Linux, and are generally
>>well under a millisecond.
>
>A millisecond is a really long time. Think of it as only
>being able to recieve 1000 packets a second, or do 1000
>disk operations per second. That might not be noticable on
>a desktop, but is completely useless for a server. There
>are people using FreeBSD to bridge (i.e. recieve, process
>and send) 2.5 million packets a second. Any latency at all
>would render that impossible.
This is worst case latency I’m talking about here. It is not like the hardware is suddenly doing nothing for 1ms, on the contrary it is running full tilt (probably processing those very packets). But even that is no problem for a server. Modern network hardware has to do a certian amount of buffering and interrupt mitigation anyway.
I’d be pretty interested to see FreeBSD bridge 2.5 Mpps. Any links? I’ve no idea what Linux could do in this regard, but to say it is useless for a server is laughable, especially in the present circumstances.
I’m fairly certian that end to end packet processing in Linux is as fast as FreeBSD, if not faster for UP, and will definitely be faster for SMP.
I know FreeBSD recently was shown to forward 1Mpps. But that is using tricks like fast fowarding that make it uninteresting to most router applications.
>>Kernel preemption has been in the Linux kernel for a long
>>time (much of the 2.5 cycle). It has not been a source of
>>problems for a long time.
>
>I recall seeing a number of posts on the LKML in the past
>year from high-level people making very moving arguments
>against preemption in Linux. But this isn’t a slam against
>Linux at all. Preemption adds quite a bit of complexity,
>and it’s something that we had to turn off for FreeBSD 5.3.
>It’s one of those “last 5%” things that is really hard to
>do, but gives you an edge in the end. It’s also quite
>useful for quickly triggering bugs that are otherwise hard
>to diagnose but bite you at inopportune times (and thus
>allows you to fix them and make the system more reliable).
There is nothing wrong with preemption at all. You may not use it if you wanted maximum straight line execution or throughput, but there is nothing wrong with it.
>>But regardless of that fact, kernel preemption does
>>nothing to mitigate these spinlock latency issues. There
>>are various works that implement realtime semaphores,
>>convert most kernel locks to semaphores, implement
>>interrupt threads, etc. but this is in order to work
>>toward hard realtime guarantees as opposed to performance
>>or general purpose computing. As such it would only ever
>>be an optional compile time setting.
>
>If your kernel synchronization policy revolves around
>spinlocks, then you are right that preemption doesn’t do a
>lot to help. You’re basically making the case that FreeBSD
>is right; converting spinlocks to semaphores (i.e. sleep
>locks), implementing ithreads, etc. FreeBSD is finally
>getting to the point where Giant is off enough of the
>kernel that we can start making intelligent observations on
>how well this strategy works, and we are starting to see a
>lot of benefit.
Well no, because FreeBSD isn’t a hard realtime operating system, so I’m not making that case at all. AFAIKS, the Linux patches to do that will have a prohibitive performance cost for general purpose computing. I’m not saying this can be extrapolated to FreeBSD, but just pointing out that I’m not making your case for you
This is really facinating to me. I have never had anyone do this to me and It makes me feel that someone is afraid of HawkinsOS actually be doing it so to me it makes finishing HawkinsOS more worth-while.
Oh please, I’m sure someone got terrified by HawkinsOS? You can’t be really serious about this… Just stop and think for a moment how unrealistically this sounds. Also, it is quite surprising that having a clone doesn’t bother you so much as to hunt him down. And as I said: there is virtually impossible to have such a persistent enemy without personal gripe against, of which you must have at least some idea. Saying that you have absolutely no idea who might be doing this to you is at least as suspicious (or funny) as saying that des hacked hawskins.com (and did what? Introduced grammatical errors in your texts?). See http://www.des.no
ps. Oh, the “other” hawkins OS guy made that ridiculous claim when someone made fun of the “proffessional and enterpris ready HawkinsOS”.
I didn’t write the testimonial that was a submission from a customer.
That’s sort of horrible of people doing that. We american programmers sometimes don’t speak english that well (Not my major lol) Ofcourse HawkinsOS will remain open source and not closed like many claim and it will obey all licenses that exist in current freebsd distro and other software licneses that will be included.
Thanks!
maybe that will be the next innovation in open-source: honesty from developers? It’ll start a craze
Thank You.
>when I installed 5.3 on my notebook the cooling fan ran continuonsly
To turn off the fan when its not needed do:
# sysctl hw.acpi.cpu.cx_lowest=c1
>when I installed 5.3 on my notebook the cooling fan ran continuonsly
To turn off the fan when its not needed do:
# sysctl hw.acpi.cpu.cx_lowest=c1
Also, if that’s a ThinkPad (T4x, R5x) you’re running, check out http://www.thinkwiki.org/Problem_with_fan_noise as you might need a BIOS upgrade.
Among the modded down comments there’s a post by molnarcs that clarifies a lot regarding the HawkinsOS trolling.
Maybe it got modded down because it’s perceived as off-topic, or personal, I don’t know.
Anyway, I just want to say that I have his same doubts regarding the actual existence of a “clone” (and sure as hell I’m not the only one). I also think that it would have been fair to mod down *every* off-topic post in this thread, and not just *the only one* who tells the whole story.
That’s it, I had to say it.