“People who think SD was ‘perfect’ were simply ignoring reality,” Linus Torvalds began in a succinct explanation as to why he chose the CFS scheduler written by Ingo Molnar instead of the SD scheduler written by Con Kolivas. He continued, “sadly, that seemed to include Con too, which was one of the main reasons that I never [entertained] the notion of merging SD for very long at all: Con ended up arguing against people who reported problems, rather than trying to work with them.” He went on to stress the importance of working toward a solution that is good for everyone, “that was where the SD patches fell down. They didn’t have a maintainer that I could trust to actually care about any other issues than his own.” Update: OSNews user superstoned pointed us to the other side of the story.
I believe Con Kolivas when he says that his patches boost performance on a desktop system, but for the time I have used them I never really felt a huge difference. Maybe they would have if I read his benchmarks first
It sucks to see this kind of arguments happen, but I’m not sure CK had all the right in the world, this time.
Looking forward to 2.6.23
than critisize the scheduler.
LOL. What is wrong with Linux’s ACPI implementation?
it depends. is its the kernels job of fixing hardware quirks/bugs in the ACPI support?
hell, at one time linus showed a strong dislike for ACPI as its kinda like a programing language in its own right. this was stated when people started talking about intels bios replacement (the one that apple use in their X86 macs) that had a similar design.
It’s the kernel’s job to work properly, and to interface well with the hardware. Wether or not ACPI has been implemented badly in BIOSs, you can’t just throw up your hands and cry ‘it wasn’t me’.
maybe so, but dont expect it to magically work fine before the bugs are ironed out…
Having lurked lkml for years… it’s incredible that people still want to do stuff for Linus, when he keeps bashing people like he does after they’ve spent days and nights shipping code to Linux (lot of CK’s stuff have been important drivers for improving the kernel). Sure, Linus says people do whatever they want to do and he reserves himself the right to call people stupid and ugly at will. Fine, but to me, he increasingly comes across as narcissistic sociopath (especially after the google techtalk craze).
I don’t think it’s true that CK didn’t take user reports seriously, but at any rate: give the man some credit, Linus!
As far as SD vs. CFS, my experiments shows that SD is indeed better at extremely cpu intensive workloads, while CFS has an edge on mixed workloads. With a decent plugsched and someone taking on the task to maintain SD (now that CK has quit), we should have both in the kernel.
IMNSHO.
>I don’t think it’s true that CK didn’t take user reports seriously
I don’t think so too. This is the usual procedure “how to kill a critic” if he dares to open his mouth afterwards. Sometimes I think, they are more about politics than actually technology.
Edited 2007-07-28 10:29
Part of being a maintainer is saying no. In fact, good maintainers might say no a lot more than they say yes. When they’re two competing solutions to the same problem, the maintainer certainly has to say no to a least one of them. The goal is not to hurt anyone’s feelings, but to keep the project moving in the right direction for most users.
This isn’t grade school. Just because you spend a whole bunch of time toiling unpaid on some code doesn’t imply that you’re entitled to get your code merged. You don’t get brownie points for being a good guy or trying really hard. Code doesn’t get merged on the grounds that everybody deserves a chance to contribute.
Code gets rejected all the time, and developers almost always take it in stride. Sometimes they make a fuss, and you can’t have that in a mature project. If the developer won’t stop complaining, the maintainer might have to take them down a peg in order to put out the flames.
The maintainers of Subversion did an excellent talk on people management in open-source projects where they stressed the importance of removing poisonous personalities from your project. Sometimes you have show good developers the door if they can’t work appropriately in the context of the project as a whole. At the end of the day, the project is more important than any single developer, however productive and talented they might be.
Maybe SD is better in some respects while CFS is more consistent overall. But the primary reason why Linus chose CFS is because Ingo is a very reliable and experienced maintainer who’s had at least twice as much code rejected than Con has ever submitted. He won’t take his ball and go home, whereas CK apparently will.
Finally, plugsched is a bad idea because scheduler proliferation is evil. There wouldn’t be just two or three schedulers, there would be ten, each one representing a major configuration that needs to be tested. Anyone can write a scheduler that’s optimized for a particular workload, but that defeats the purpose. The scheduler should perform consistently in a variety of workloads.
I hope that SD lives on as a patchset for those workloads that benefit from it. But one solid scheduler is all we need in the mainline kernel. CFS is the right choice for the long haul. We’ll all benefit from the improvement over the O(1) scheduler, so we should be happy.
Con Kolivas’ personality is “poisonous” or his work poisoning the kernel? Are you serious? This guy, more than any one else, brought to attention and a solution to the problems of desktop performance. Linus is human–a high-acheiving one, driven by ego and reputation–and his behavior in this matter is observable in most individuals who acquire national prestige and reputation when they are threatened. They first actively seek to find fault on technical grounds, decline to allow wider testing of alternative ideas and methods (in Con’s case–disallowing entry into the mainline kernel), and when that fails attack the individual as being unstable, untrustworthy adducing any vocalized frustration on Con’s part as proof of their unsuitability.
He’s doing the rounds on the interview circuit, explaining that computers are dumb and boring, and how there’s a vast right-wing conspiracy to make Linux unsuitable for desktop users. He’s telling people that he has solutions to a variety of problems with desktop operating systems, but nobody will listen to him. The fact that the kernel community rejected his work is apparently a slap in the face to Linux desktop users.
This is, at best, bad advocacy. He’s coming off like those whiny kids that go on talk shows defending that movie on how 9/11 was an inside job. Maybe they’re right. I don’t know. But they moan and groan and stomp their feet and nearly break out in tears when the guy from Popular Mechanics tries to calmly refute their theories. It’s impossible to believe them because they just can’t control their frustration.
That’s poisonous personality. It means you don’t get what you want, even when you’re right. It means that other hackers working on desktop performance issues cringe in embarrassment that this guy is representing them in public. This isn’t how Linux kernel developers are supposed to act. The kernel community should not encourage this kind of behavior.
The Rubik’s Cube kid that screwed up solving the cube blindfolded on national television and got really upset. Nobody thinks he sucks at Rubik’s Cube, but he might have more fundamental shortcomings.
EDIT: It was a close call, and a decision had to be made. There wasn’t a definitive argument for either solution, but it doesn’t make sense to merge both. Somebody was going to get pissed off. Linus ended up pissing off the developer who would get most pissed off and who has the more vocal supporters.
Would there be outcry if SD was chosen over CFS? We’ll never know, because in the end, Linus made what he thought was the safe decision. A lot of people are unhappy, and I’m not sure that was avoidable. That’s my final comment on this issue.
Edited 2007-07-28 14:01
You’re extremely rude to Con, and it’s both inappropriate and wrong. I (and many others) don’t care wether SD or CFS got merged – Ingo’s a great guy (and he and Con don’t have a fight) and CFS is good. The problem is the way Con is treated. And that’s not due to Con’s behaviour – the discussion on swap-prefetch is still ongoing (I follow it, partly) and it’s just as bad as it was when he participated. He actually deflamed several discussions.
You seriously don’t know Con, do you? He’s a very kind, responsive person. It’s the kernel-ppl who where behaving very in-crowd and rude, not him. He gave (as good as possible) technical reasons, they rejected him repeatedly for the same, already proven wrong, reasons. I deliberately talk ‘they’ here, because it has not been one person, nor always the same people. Andrew has imho been unresponsive on occasions, but less so on others. Ingo has not always been very supportive or smart, on the other hand – he’s a good guy.
I’m sure email as a way of communication has been to blame as well, not all arguments where meant as rude as they sounded.
All in all, what I’m trying to say is the situation was complex, not black&white, but YOUR account is totally off…
How do I know? I don’t know everything perfectly or something, but I’ve been following Con’s mailinglist for many years, and I’ve seen the good and the bad.
“decline to allow wider testing of alternative ideas and methods (in Con’s case–disallowing entry into the mainline kernel)”
sorry – you consider the mainline kernel an appropriate place for “testing of alternative ideas and methods”?
sorry – you consider the mainline kernel an appropriate place for “testing of alternative ideas and methods”?
It’s not as if Linus himself thinks any differently.
I’m going to take issue with this ‘maintainer’. One, Con had lots of people testing SD, including myself. I myself provided feedback to Con on IRC. Second, your excuse that ‘scheduler proliferation is evil’ is stupid. In fact, we should have more pluggable mechanisms in the kernel to allow for greater testing capabilities. I’m truly ashamed to see Linux development go down the drain in the same direction as closed-like projects do with their closed mind, closed attitude towards developers.
Edited 2007-07-28 18:54
Its the old story of smart people – the smarter they are, something has got to give, they’re either, weird, arrogant, strange, violent or some other trait.
I’ve seen the debates that occur on lkml – and the end of the day its up to you as to whether you wish to contribute and be part of the community. No one is forcing you to join and contribute to Linux – if you don’t like how things are going, there is nothing stopping you from forking the Linux code and creating your own community.
Thats the thing; if there was such a major issue with Linus, someone would have forked Linux, created a parallel community. All things consistant, if there was such a dislike for Linus’s mannerisms, he would have isolated long ago – so I’m assuming he is doing something right (not that I agree with him or how he holds himself).
On a bigger issue, this is the reason, although qualified in IT, I don’t work in it. I don’t work in it because of what I’ve seen and experienced; I definately don’t want to work in a place like Microsoft were I am constantly pushed, prodded and probed by upper management, expecting me constantly on the defensive about every decision being made, constantly having to advocate and explain every possible position made then interrogated.
I can see why so few females are involved with IT – this constant agressiveness thats required. For me, I prefer working in a team as a team rather than simply a bunch of individuals who happen to be under the title of being a group. Peoples egos, unwillingness to discuss in a polite way seems to go way over the head of many people as I’ve seen.
”
I can see why so few females are involved with IT – this constant agressiveness thats required. For me, I prefer working in a team as a team rather than simply a bunch of individuals who happen to be under the title of being a group. Peoples egos, unwillingness to discuss in a polite way seems to go way over the head of many people as I’ve seen.
”
Oh, so it is REQUIRED now? No, people make it that way. It doesn’t have to BE that way. People become aggressive and insulting because they want their way over someone else’s way.
I am guessing that Linus is so famous now that the people who work with him and get their faces slapped actually ENJOY getting their faces slapped by “the great Linus.”
I’ve yet to be in a situation where it has been different from the situation described. I say its a biproduct of men in a group sitting around trying to ‘out piss’ each other. Beating of chests, trying to prove they’re the bigger and better alpha male. Its pathetic to see it in action.
I can assure you, having been in a mixed project of male and female you can really see how the dynamics are different. Rather than the leader trying to make a name for himself she instead encouraged and fostered a team atmosphere.
When I look at the IT world its all about ego’s, beating of chests and hopeing if one makes enough noise, they win the praise of the leader. Just take a look at Microsoft and how their meetings are conducted – and they wonder why the crapest ideas from the biggest loud mouth are chosen rather than a good idea from the quietly spoken gentlemen at the back of the room who doesn’t want to ‘rock the boat’.
I wouldn’t be surprised; there are idiots here who appologise for Stallmans out of touch rantings no end – completely devoid of reality when it comes to delivering solutions for end users.
Btw, it isn’t just limited to Linux – I’ve seen it in a number of projects. Its sad though that for every 1 loud mouth prick there are atleast 9 people behind the scenes working on what matters rather than trying to get their name in lights or have infamy for being the biggest asshole on the block.
You took the wind out of my sails.
You are right tho’, I have seen it at work as well, in design meetings mostly. It’s a shame and I always try to steer away from that kind of thing in my meetings (don’t have them anymore since I work remotely now but…) by trying to at least show support that people speak up and even if an idea is not the best, to try and point out the good in their ideas/comments and downplay the bad bits. In the end, weighing the ideas for the one[s] that provide the most benefits.
Of course you cannot ignore the fact that sometimes they could have very bad effects as well and that should be pointed out…
It’s REALLY nice when everyone thinks along the same lines.
Thanks for your post, it was very good, and a great response to mine which could have been interpreted as a bit inflammatory and perhaps trolling.
Awww 🙂
Right now I’m working part time whilst at university – being a supervisor managing teenagers. Its always good to give a compliment when they do a good job – recognising their contribution.
One of those things that are necessary; rather than putting down an individual and saying they’re wrong, being proactive and saying, “hey, you’re on the right track there, I’ll show you how to make it even better” – positive re-enforcement and correction at the same time.
Having supervising/managing people for almost a decade, the one thing you learn is trying to make the work place an enjoyable place to be rather than it being a burden. If they feel welcome, respected and encouraged – hopefully that will in turn bring out the best in people and as a whole, we all benefit from it.
Linux doesn’t implements ACPI.
Do you know who is the Linux ACPI maintainer? There’re a few programmers being paid by Intel -you know, this little company that invented ACPI- to maintain it.
The fact that the same company that created ACPI can’t create a working implementation should give you an idea of how easy is this task and how beautiful is the x86 world.
the thing is, a guy, which i will not mention the name of here, was a fool.
He tried SD, and found it was not to his liking, for being fair.
what was he doing? and what was his problem?
well, his problem was that the gui wasnt running perfectly smooth when he did several things, which put a somewhat large load on the box. and he refused to renice.
is this a bug? no, and ill explain. SD is supposed to be a fair scheduler, and that means that if you give some tasks equal prioity, they will get equal cpu time if they want it. the specific problem here, was that the time alotted for X simply was not enough for it to perform the tasks he wanted, within the time he wanted, and hence the gui lagged.
Con explained this numerous times, and, popularly said, refused to do anything about it, since this was not a bug, it was in fact the exact expected and sought after behavior.
Now comes along linus, and in his infinite wisdom decides that this means Con is totally unwilling to listen to user input..
the truth of the matter is, torvalds did not understand what was going on, and in his typical lowbrow attitude he simply proclaimed that he is god, and can do no wrong(like has happened before).
and as for CFS, well, CFS never managed to perform as smooth as SD did for 3d under load, despite ingo attempting to fix it alot, he failed. Oh, and ingo also tried to cheat his way out of SD being superior, now i dont want to spoil your hunting on lkml.org cfs thread, but ill give some pointers.
1: cfs thread
2: in the cfs patch, an obscure x86 only io thing which basically ONLY xorg/xfree triggers
3: renicing to -15 (or was it -10?)
oh well..
and the never ending question remains, is it the users job or the systems job of managing tasks?
in either case, CFS (which is also completely fair, just like SD) did make it into the kernel – despite it has the same ‘problem’ (not being unfairly positive to interactive tasks). So it’s pretty silly of Linus to use this ‘problem report’ as an argument.
If you read a little further in the thread, you’ll see some very good and well informed comments, btw.
if linus switched the kernel to con’s scheduler, con would take over as maintainer. but with ingo’s scheduler, linus could keep ingo as the maintainer, a person he have learned to trust over the years iirc.
in the end, its a executive decision so to speak. people will either love it or hate it, but they will all learn to live with it.
Why would that necessarily mean Con Kolivas would take over as the maintainer? Could he not simply maintain the HIS scheduler? And it would seem to be better for Con, given all the other desktop-related things he was up to.
from what i understood ingo was the current scheduler maintainer, but it could very well be that i got something mixed up.
And this should not be happening in Open Source. What happened to “best code.” Ooops….
Seriously I have had this happen to myself. Around 96-97 I ported the Apache server to Windows and wanted to integrate the sources. What happened? I was blasted. Some guy emailed me privately and said the exact same thing happened to him and that I should forget about it.
Now Apache is on Windows, and it was late!
My point is that Open Source should stop making the argument that their software is based on technical merit alone. And that politics plays quite a bit in Open Source.
welcome to humanity…
True, I see your point. But linus should’ve said THAT, and not give silly and/or uninformed arguments.
And besides, if Linus from now on would never add any new maintainers, Liux would die soon… Con has proven himself over the years as a trustworthy person (no matter how Linus is trying to portray him), and it would allow Ingo to spend time on new stuff…
now that i think about it, isnt there a kind of chain of command?
as in, linus should not have gotten directly involved as ingo is the person in change of that area of the kernel?
also, from what i understand the SD patches where getting kinda massive. and linus iirc hates massive patches (see reiser4 for a similar issue). how big was the CFS from ingo?
Yes he “hates” “one” massive patch, but he favours zillions of single patches – that sounds somewhat funny :o)
>how big was the CFS from ingo?
Just politics – don’t even try to reason.
i think the reason for his patch policy is that with a single large patch that touch on multiple files and so on, one is forced to take it all or nothing. but with smaller patches one can apply those that works and fix those that dont.
but now that he has git and seconds in command, i dont know how much it applies any more.
Linus prefers large submissions to be split into a series of smaller patches. You’d understand why if you’ve spent enough time reviewing diffs. It’s purely a pragmatic policy to make things easier for subsystem maintainers, Andrew, and Linus. They have to review ungodly amounts of code, so it’s only fair that hackers take the time to split their patches up into digestible parts.
I’m not sure about the relative sizes of SD and CFS, but I don’t think that this was a major factor in the decision.
Reiser4 had a completely different problem involving duplicated functionality and unusual implementation details for a Linux filesystem. Most of this as since been resolved, but only because other developers took the initiative (eventually). On the other hand, Hans and Con have similar personalities that don’t work well in the kernel community.
It’s important for the kernel community to be comfortable with the code. Consider what happens to the users that rely on this code if its original developer ends up in prison, for example. Is anyone in position to maintain it? What happens if the developer quits because of a dispute over a patch? When code goes into the kernel, there’s a commitment to supporting it for the long haul.
Unfortunately, the kernel community seems to want submissiveness more than originality. The whole novelty of Linux is what draws so many people in.
Con has proven himself over the years as a trustworthy person (no matter how Linus is trying to portray him),
I used to run -ck patchsets fairly consistently, and I appreciate Con’s work, but I don’t understand where people get the idea that he’s “proven himself over the years.” He’s a relative newcomer to kernel development. Until a couple years ago, he didn’t even know how to code. He learned by resolving merge conflicts as a patchset maintainer. It’s a fantastic story of a self-taught developer pulling himself up from his bootstraps.
He’s gotten a good handful of relatively modest patches into the mainline, but he hasn’t maintained any significant chunks of code. He went straight for the jugular with the scheduler and swap prefetch. These aren’t pieces that you can trust to a relatively inexperienced hacker. Not in the Linux kernel, with millions of users, many of them in production environments. Most hackers start by taking ownership of a driver or a non-core kernel service.
There’s so much work to be done in the Linux kernel, and the vast majority is relatively accessible for newer kernel hackers. But Con picked a couple of the very few parts that are typically reserved for seasoned kernel hackers for good reason. For one thing, they know how to navigate the kernel community, grease the wheels, and get the job done. They’ve had all the idealism beaten out of them. They know when it’s time to deviate from the theory.
CK may be a hero to the enthusiast crowd, but he doesn’t have much credibility in the kernel community. He doesn’t appear to be especially trustworthy, either. I’m interested to see what he ends up doing next.
He doesn’t appear to be especially trustworthy, either.
Explain that one, please, butters. I generally like your comments (this one included), but if you make such statements, you better have something to back them up
.
He quit because his patches weren’t merged. How can you trust someone to maintain a crucial part of the kernel if they can’t take criticism? It takes a certain kind of teflon-coated composure, a certain level of well-earned respect, and a certain capacity to be moved by reasoned arguments in order to maintain contentious parts of the kernel.
Maybe if SD were clearly better than CFS, than the decision could be made on technical merit. But it’s a close call, so it comes down to who we want maintaining the process scheduler of the mainline Linux kernel. Ingo has all of the above listed qualities, and he’s maintained the scheduler before. We know he’ll do an excellent job.
Like Thom, I more often than not enjoy your posts but I have to disagree with this line of reasoning.
Yes, Con is, compared to Ingo, green when it comes to kernel hacking but alone the fact that he willingly maintained his own patches outside of the mainline kernel for so long not only means that he is well up for the job, but also puts Linus’ argument about him not being trustworthy to shame.
Pretty much everybody who replied to Linus in that thread stated the same thing, that Linus was talking out of his rear end. Character assassination is a very dirty way of going about dismissing somebody you don’t like.
Hans Reiser is a completely different situation to Con’s. If you look at how Mr. Reiser tended to through tantrums and cry conspiracy, then look at how Con just knuckeld down and did the job, you quickly see two different personalities here. Sure, Con left in a huff after the CFS sheduler was merged instead of SD, which if you believe all the people who replied to Linus is actually an inferior implementation; but frankly if you where slapped in the face, wouldn’t you?
Edited 2007-07-28 13:45 UTC
I can’t blame con for leaving the lkm scene.
Especially now with two small kids and a demanding job as an Anaethesist it will be a relief to have more time left for other things that the linux kernel.
“Hans Reiser is a completely different situation to Con’s”
Baaawhahahahahaha. That’s so true nowadays.
On his own list, he has contributed to more code than you might think. Jens axboe (IO scheduling) and the adaptive-readahead guy have been on his list, working with him on issues in their code. Heck, I myself even tested some of that stuff. Why would they work with him if they though he can’t code?
Besides, Ingo is supportive of the swap-prefetch code as well. And I only ever heard positive comments from the kernel hackers about Con’s code. He did contribute patches to the current (pre 2.6.23) scheduler as well, before he decided it needed a rewrite (and it obviously did, it just took years to convince others).
Please understand all this is NOT against code from others, or other persons. It’s about the political and social part of the kernel. Ingo’s CFS is great, and I’m happy it got merged. It’s merely annoying how Con’s code by default seems to be ignored, just because he’s working on desktop stuff which is hard to measure – esp on 16-core 16 GB hardware.
Nobody is questioning CK’s ability to code. I didn’t mean to imply that. I’m questioning his ability to own the scheduler and deal with the criticism that comes with that job. You don’t want someone with strong beliefs and visible biases in that position. You want someone who can maintain a semblance of impartiality and mature judgment.
To make a political analogy, policy wonks and political operatives don’t run for elected office. They become senior advisers and campaign strategists. It’s a shame that CK couldn’t find a comfortable role in the kernel community, because there’s no question that he has good ideas and a passion for improving performance.
He should be working with Ingo, Jens, and Nick. That’s where he belongs. That’s where he can make a difference, which is the reason he started hacking in the first place.
Con’s work was original. He was the first to rethink and rewrite the scheduler. Why shouldn’t Ingo and Linus accept CK’s work and help refine or tweak it so that the code is in a form that’s suitable for future maintenance and suitable for other applications as well? Instead, he goes and builds his own scheduler incorporating CK’s ideas–and arguable does a worse job of it–at least according to many on the LKML. And then, because of his position of confidence with Linus, he gets it included in the mainline kernel. That is the essence of workplace politics! Who would not be frustrated and angry?
Your analogy with politics is misplaced. The kernel community fancies itself a meritocracy. None of them would appreciate being likened to GW Bush, Karl Rove, or Cheney. They are more similar to a community of scientists–with the same politics.
butters, I think you and I won’t ever agree, and I will be the first to admit that I do not know enough about the technical issues to weigh in with an opinion about whose solution is better, but CK’s handling was plainly wrong and unjust.
> He should be working with Ingo, Jens, and Nick. That’s where he belongs.
Here we go, our caste system spelled out.
Outsiders with good ideas and reasonably good
implementation need not apply, if they
DON’T KNOW THEIR PLACE.
What he ends up doing next? You mean in operating system engineering? My guess–nothing, for a long time. He is a physician and has a family who will probably be happy not to share him with Linux.
>CK may be a hero to the enthusiast crowd, but he >doesn’t have much credibility in the kernel >community. He doesn’t appear to be especially >trustworthy, either. I’m interested to see what he >ends up doing next.
This is what I mean, I see more and more kernel people I know being turned off by Linus and this ‘process sham’. I was going to get into kernel development myself but it’s not worth it. Userspace is more fun, less political.
“””
Con has proven himself over the years as a trustworthy person
“””
How’s that? He *quit* when he didn’t get his way. Compare that to Ingo, who has likely had more code rejected over the years than Con has written, and has never threatened to quit,let alone actually done it.
I’d hardly call that “trustworthy”.
Some examples of major work by Ingo being rejected:
http://lwn.net/Articles/242912/
Though I started out pretty neutral, the more I hear from Con and his followers, the more convinced I become that Linus made the right decision by selecting the more reliable maintainer, and the one who understands the importance of acknowledging when a problem might exist in his code.
Like Linus said:
“”
This is like alcoholism. If you cannot admit that you might have a
problem, you’ll never get anywhere. And quite frankly, the RSDL proponents
seem to be in denial (“we’re always better”, “it’s your problem if the old
scheduler works better”, “just one report of old scheduler being better”).
“”
http://lwn.net/Articles/226963/
This wasn’t about choosing a maintainer perse, nor about SD vs CFS – both are fine schedulers.
What does bug me and others is how the beneficial work of Con got ignored way longer than it should have. His staircase was offering better interactivity to 2.2 and 2.4 – it didn’t get in. We’re at 2.6.23 and finally, suddenly, everybody seems to agree the mainline scheduler sucks (duh!) and something is done about it.
See swap prefetch. Why won’t it get in? If you see the reasons given, they don’t make sense. If you’ve folowed what happend around Con the last few years, you start to wonder if the process in which the kernel is developed is right. I’ve discussed this with Andrew Morton once, and he thinks it’s mostly fine (though not perfect). Well, I wonder. Seeing Con go, I wonder how many did leave without this splash. How many didn’t even get involved at all???
SD is supposed to be a fair scheduler, and that means that if you give some tasks equal prioity, they will get equal cpu time if they want it. the specific problem here, was that the time alotted for X simply was not enough for it to perform the tasks he wanted, within the time he wanted, and hence the gui lagged.
Yes, but then you have to ask yourself, is the CPU scheduler in an operating system that requires users to constantly tweak their nice levels while they change between doing various different activities, each with different load patterns, ideal for the *desktop*?
Is this what would happen, or can you get away with setting them once then leaving them be? (In which case the distro’s can handle it for the most part out of the box.)
Most of the time, I want an operating system (read: not kernel) tuned for low latency user interfaces and for intensive tasks to automatically drop back to make that happen. But then again, I want a system that is smart enough to notice I am not moving the mouse, the keyboard or reading (occasionally scrolling a web page) and then bumps the priority of background tasks appropriately while I don’t need the interactivity.
If Linux is going to use a *strictly* no nonsense fair scheduler and any of the distro’s want to build a solid desktop environment on top (One that doesn’t want to make you tear your hair out) then as well as a nice set of ‘nice’ defaults it’s going to take a lot of effort to enhance user land t the point where it can perform things like automatic renicing with a sense of intelligence relevant and well adapted to the user.
One of the things I think people are also underestimating is tweakaholic syndrome. Much like many Window’s users which are obsessed disabling services and performing registry tweaks (most of which have positive results that are purely in the mind), this could lead to all kinds of posts on support forums like “Renice X to -19 and gnome-vfs to -5 for ultimate performance!!” and a lack of any real empirical study into the problems with the desktop and improving it.
From a technical perspective *I’m not opposed* to simplifying (or dumbing down) of the kernel scheduler to something completely strict and fair and putting more weight behind nice levels. All I’m saying is the user land will have to adapt and i’m not convinced application developers or distro’s have even considered this seriously yet.
Edited 2007-07-28 11:26
You’re underestimate the clever design ideas behind both SD and CFS. You’re experience in terms of interactivity is not likely to degrade, and in certain cases, it’ll get even better. Meanwhile, the reason for the fairness in the first place is that unfairness leads to serious, unfixable problems and cornercases. The specifics are beyond me here, but race conditions like priority inversion (where one high prio process, waiting for a lower prio process, is preventing that same process from running, thus stalling the system) can only be fixed once and for all by fairness.
Oh, and ingo also tried to cheat his way out of SD being superior, now i dont want to spoil your hunting on lkml.org cfs thread, but ill give some pointers.
This is just pure FUD.
Ingo did no “tried to cheat”. He implemented a patch to give more priority to processes trying to use the syscall that X.org uses to do I/O to graphics cards as an experiment to nice to -19 processes doing I/O. It was an “obscure x86-only thing” because that was the way to implement this feature.
He released it to the public so that people could give feedback on how it behaves in their systems. He ANNOUNCED it: http://lkml.org/lkml/2007/4/20/197
– usability fix: automatic renicing of kernel threads such as keventd, OOM tasks and tasks doing privileged hardware access (such as Xorg).
There were reports of people saying that CFS with this patch behaved better than SD thanks to this patch and Con needed to explain that SD wasn’t renicing anything so you needed to renice X or disable the CFS feature to compare both schedulers.
But then Linus said it was a hack and vetoed it: http://lkml.org/lkml/2007/4/23/179 . And then Ingo removed it, despite of the fact that he argues that desktop is smoother with it (http://lkml.org/lkml/2007/4/23/254). Juzguing from the reported smooter experience, I say that it was a worthy experiment.
End of the story. So much for your stupid “cheat” argument.
I really think you are saying this from an outsiders perspective, the fact is, when people download CFS many of them dont look at the thread, many did not even know this! i know this from the threads when they later observed that X had been reniced.
fact is, if mingo would have wanted people to compare to SD with X reniced, he should have simply told them to renice it, not slip it into the patch like this.
Nah, I don’t think this was ‘evil’ on the part of ingo, though it was sure controversial… He’s not like that, imho.
I really think you are saying this from an outsiders perspective, the fact is, when people download CFS many of them dont look at the thread
No, it’s YOU who is saying this from a outsider perspective. New CFS versions were announced at the lkml. There could be several releases in hours if a serious bug or a big improvement was coded. Testers did read the lkml to get notified about new versions and told their experiences replyin to the announcement thread. This was an internal lkml work. There were not “outsiders”.
if mingo would have wanted people to compare to SD with X reniced, he should have simply told them to renice it, not slip it into the patch like this.
The goal of this feature was to renice tasks automatically. If this feature would have been useful and had not been critized and vetoed by Linus and others, Con could have picked it up and released a new SD version with this feature.
Ingo also prefaced his comments with a completely deferential tone arguing against his own code’s making a special case for X as setting a bad precedent and potentially causing trouble in the future. While he clearly was in love with the results of his hack (in his own words), Ingo relegated it to “useful experiment” status and accepted that it was not perhaps the best solution to the problem.
there’s always two sides to every story. Even though Linus can sometimes express very strong confrontational views over philosophical or ideological aspects of open source,it’s nice to see the side of him we rarely do or that’s rarely reported,i.e the pragmatic level headedness that he clearly possesses and has demonstrated many times before in the past.Good articles :>
I think you all should read
http://lkml.org/lkml/2007/7/28/29
(it’s content is confirmed by many in the same thread)
It’s slightly annoying to me to only read Linus’ comment here, without the answers pointing out the obvious flaws in his reasoning.
I feel that, more important than which is implemented (CFS or SD), that desktop interactivity and scheduler fairness become a much more regular focus of those who have the power to make a difference to the base Linux code on kernel.org.
Can a single kernel can be compiled for desktop and/or server priorities, depending on compilation flags? Yes this can already be done but I mean more in the direction of SD/CFS work. I don’t see why not.
Linus needs to know that there is a complaint he himself isn’t addressing. It is that the kernel developers aren’t giving priorities to desktop interactivity and experience. The response here on osnews.com etc have shown that there is public demand for it too. Average Joe’s like us don’t run Linux servers (except maybe on our SOHO routers!) but we do use a desktop for hours a day and that’s what matters to us. Most people are not running servers in corporate server farms. Had the Linux kernel abandoned the desktop experience as a priority? I think it had until Con did his magic.
Maybe once or twice Con couldn’t help or fix an issue but isn’t that what open source software is all about anyway? If one person can’t fix a problem does that disqualify the entire effort or the target goal? I agree with previous posters here that the ‘bug’ Linus complains about not being fixed, wasn’t even a bug, but a feature. If he has issues with Con’s handling of the situation that’s fine with me. I don’t mind… He might by right. So what.
Edited 2007-07-28 11:04
note the point here is NOT which got merged, CFS or SD – it’s how Con was and is handled by the kernel community. He’s a very valuable hacker, and they let him go.
http://lkml.org/lkml/2007/7/28/34
“””
note the point here is NOT which got merged, CFS or SD – it’s how Con was and is handled by the kernel community. He’s a very valuable hacker, and they let him go.
“””
I would argue that even brilliant and talented people are not necessarily “valuable” to the project if they are also poor team players.
I believe that Con will be back, and strongly suspect that this business of “quitting” is a poorly considered stunt.
If and when he returns, I hope that he is a better team player and that both he and his followers have a more positive and constructive attitude. In other words, I hope that he can be truly valuable.
Edited 2007-07-28 14:57
Hmmm. I don’t think he’ll come back anytime soon. He got real sick, you know, for a big part due to the stress of all this.
And he’s not a poor teamplayer. He’s not perfect either, but I do think the majority of the ‘fault’ does lie in the kernel community’s behaviour. Again, not at specific people. But I think a series of incidents clearly points out there’s an underlying problem…
“””
He got real sick, you know, for a big part due to the stress of all this.
“””
If kernel development caused his illness, then perhaps he should not return. Getting high profile pieces of code into the mainline kernel is not for the weak or the thin-skinned. You wouldn’t catch me trying it even if I felt that my skills were up to it.
However, many other kernel developers have code rejected, some trying for many years to get it included, without it affecting their health, and are likely better suited to being long term maintainers for it.
Every exchange that I read, both here and on LKML, increases my feeling that Linus made the right choice. Yes, even the one’s linked by Con’s supporters. Because they so often turn out like your reply, presenting Con’s illness as a reason for merging the code, rather than as a reason *not* to merge the code.
Con’s health is Con’s responsibility and not that of Linus, Andrew, Ingo, or anyone else. The *kernel’s* long term health *is* their responsibility. And I cannot help but feel that it is in good hands.
Edited 2007-07-28 18:34
“He’s a very valuable hacker, and they let him go.”
They didn’t let him go, he left. That is a big difference.
From his departing article he stated that he will get obsessed with some other non computer related project now. He didn’t seem like a very stable coder. Now what would have happened if more of his code was accepted then one piece of code down the line was rejected? Would he have quite like this? You don’t want your main coder on a project to just suddenly drop it.
This is why Linus makes sense in what he was saying. He needs someone who he knows will be there.
And why couldn’t CK just suck it up and start patching CFS? This would be the best way to rise above the politics and support the project you believe in.
“””
From his departing article he stated that he will get obsessed with some other non computer related project now.
“””
Add to that the *nature* of his illness (disc prolapse in his neck), and the fact that he says that he did not leave over SD being rejected ( http://lkml.org/lkml/2007/7/28/179 , originally posted by RJop ) and the fact that he did not choose the article headline (which was originally “Why Linux Failed On The Desktop” and was changed by the author to “Why I Quit” when he objected to it that) and the fact that he is still posting cpu scheduling tips on lkml, at least as of yesterday, and a different picture than what has been primarily discussed begins to emerge.
Edited 2007-07-29 19:47
Indeed. Ow, and ‘sucking it up and start patching SD’- that’s ridiculous. SD went through over 50 (FIFTY!!!) iterations before it became 1.0!!! CFS isn’t at version 0.50 yet, is it? I wouldn’t say Ingo is not as responsive as Con, but Con SURELY isn’t any less responsive than Ingo.
Point is, and he admits this himself, Linus has only a limited view on the whole issue. He doesn’t follow the CK mailinglist, he only got CC’ed on a few emails. He didn’t see the hard work and short response times, but he DID see the one guy giving stupid complains and Con, tired of it, getting a bit pissed off. Once…
Aaah, and don’t just believe me:
http://lkml.org/lkml/2007/7/28/117
This comment might be even more clear than the other one 😉
http://lkml.org/lkml/2007/7/28/34
the latest post on lkml brings up a good point. http://lkml.org/lkml/2007/7/28/67 i wonder what Linus will say.
He ignored most of the post.
http://lkml.org/lkml/2007/7/28/109
Actually, one of said SD merits was its uncomplexity and small size (92K compared to CFS’ 217K, main patches only, applicable to 2.6.22). Additionally, Con ported SCHED_ISO and SCHED_BATCH (different than mainline’s) to SD from his old staircase, which I find very interesting and helpful. However, I blurly recall reading something about people not understanding Con’s SD’s code (which effectively stopped them from contributing). I wonder how understandable CFS’s code is to someone else than Ingo; considering it’s size, I suspect it is even more obfuscated.
As for the mentioned one person complaining about SD, if I recall correctly it was Mike Galbraith, who seemed versed enough in scheduler field, helped Con and then Ingo somewhat, but intently complained about X not getting enough CPU time with high load (I remember something along the lines “X is getting only 33% CPU time, not 66%!!!11”). Con indeed refused to fix those “issues”, attributing it to the fair nature of the scheduler. Well, as I see it, either you add hacks (like mainline scheduler is said to be done) to fix all the cases, or stick to the design. It didn’t seemed that much of an issue, really, and let’s remember that it was still in SD 0.2x times, Con tried hard to make things better (even in his painful state). It’d be nice if computers would read users mind and proceed adequately, but current state is that it is even hard to differientate between X and non-X tasks, let alone foreground and background X tasks (I caught something along the lines on CK’s ML, I’m really no expert, as stated below). You can’t just get everything right with one CPU scheduler. Let’s remember that predictability is said to be a main asset of a fair scheduler. I wonder how Ingo tackled that issue with X not getting enough; I hope the design alone alleviates that, adding hacks would taint its predictability, wouldn’t it? Incidentally, I remember people observing with some amazement that the higher the load with SD was, the _gradually_ slower performing of tasks became. Well, you can’t just get more of the CPU than it offers, isn’t that right?
Regarding Con leaving kernel development: a comparable scheduler got merged, some implementation of dynticks got merged, swap prefetch is continuously refused to be merged, but stable and available to all interested; there’s not much left in -ck patchset (some VM stuff) to bother releasing it, unless he had the incentive, motivation and ideas for some other improvements. That way or another, Con decided to pursue other things, like studying Japanese (ganbatte!). He didn’t just leave before SD was refused to be merged, did he?
My take on the commented article: however I like some of Linus statements and his inclination not to beat around the bush, I truly don’t think this evaluation of Con’s work and attitude holds much truth.
Those are observations of a guy who more or less attentively observed the developments of SD itself and then discussions regarding CFS on CK list, although, earnestly, not versed enough to understand _all_ the technicalities mentioned during that time (vide Mike and deeper background behind his complaints and why it has worked this way, not another).
-edit-
some typos fixed; heck, how do I even manage to write such monstrous posts, not even being a native English speaker.
Edited 2007-07-28 15:22
IMHO this has all gone by and far out of hand. Some things do get in the kernel, some don’t. That’s just the way it is. If you can’t handle that then don’t, you’re free to stop caring. I can’t really say much about CK cos I don’t know him as a person and I see lots of very biased arguments both against him and in favor of him, but yeah, Linus treated him quite coldly. But I’d assume he does that to everyone else too.
I’m just wondering the whole scheduler thing..How many of you people actually can explain what it is a scheduler does and on what grounds is CK’s work better than Ingo’s? I hear that Con’s one is better for desktop and Ingo’s is better for mixed workloads so in my view it might be a good idea to include both and just select either one depending on whether it’s going to be used on a desktop or not. But well, I’ve been using the old scheduler for god knows how long and my desktop works just fine without either of those schedulers…
Sure, the diff between SD and CFS is small, and for me it doesn’t matter. I’m fine with CFS in the kernel, it’s a huge improvement over the previous one. But why, while there was a good alternative, did THAT one stay in that long?
Some things get into the kernel, other don’t. Some get in too soon, others too late. Sure. But shouldn’t we try to improve this process, instead of saying ‘it is what it is, get over it’?
For me, that’s the purpose of this whole discussion. We’re losing valuable code and contributors, yet at the same time code which isn’t mature yet enters the kernel. Acknowledging this is a problem is the first step in solving it.
Won’t even *one* of you offer to maintain the SD patch and fix the inevitable problems? Won’t even *one* of you step up and do what it takes to show that you are serious and will be a reliable long term maintainer? Sign a document committing you to a certain level of support? Or use the credibility which you have already earned with the kernel developers as collateral, if you have such?
No one?
Well, that settles that, doesn’t it?
In addition to code quality and merit, inclusion of code into the kernel requires a degree of earned trust, willingness to consider problem reports, and perceived reliability of maintainership which SD apparently did not, and does not have.
Code inclusion into the mainline kernel goes to those willing to do the work and take on the responsibility.
> Won’t even *one* of you offer to maintain the SD patch and fix the inevitable problems? […]
> No one?
> Well, that settles that, doesn’t it?
Har har, be serious now. What on earth would be the point of trying now?
“””
Har har, be serious now. What on earth would be the point of trying now?
“””
1. Linus has been known to change his mind when presented with a *valid* reason to do so.
2. At worst, it would be less useless than whining on message boards.
Qoute Linus:
“It’s not like we’ve come to the end of the road: the baseline has just
improved. If you guys can show that SD actually is better at some loads,
without penalizing others, we can (and will) revisit this issue.”
If you guys believe so much in the SD scheduler this is a good reason to continue working on it after Con.
Sure beats crying and whining on public forums about how mistreated you were.
The issue is NOT the whole ‘SD vs CFS’ thing. It’s about the way kernel development works, it’s culture.
http://lkml.org/lkml/2007/7/28/186
The issue is NOT the whole ‘SD vs CFS’ thing. It’s about the way kernel development works, it’s culture.
“Dig your culture well” Monster Magnet:P
No. I’m a professional software developer and have on occasion been interested in jumping in somewhere. However, the political environment whereby someone is falsely accused, rejected for years, and then finally after ideas are stolen and credit taken elsewhere, I’m sorry. I wouldn’t touch it. And, I bet there are a lot of very talented people that are in the same boat. Linus and some others need to take a course in human relations management. They are woefully lacking. And, the community suffers because people like me will avoid it like the plague. It is a sacrifice to contribute to the community, and all such efforts should be honored, by treating the people honorably. It’s apparent that not only was Con not treated honorably, he was driven away, purposefully. Linus says we’re armchair complainers, and because of his attitude, we’ll remain so, to the detriment of the community.
“””
No. I’m a professional software developer and have on occasion been interested in jumping in somewhere.
“””
That’s probably just as well. It sounds like you are not really suited to Linux kernel work. I’m not suited to rugby either. 😉
But there is *much* more of a need for you in user space. That’s where the desktop problems lie. We’ve got the staples down, pretty well, but need apps to fill in the… rather large… gaps outside of that. We need people to help eliminate the incredible amount of waste that Dave Jones discovered in his trip into the user-space Wonderland.
You might want to start with something like eliminating “font madness” from your favorite apps.
I’ve said it before in my other Osnews posts, but I’ll reiterate that SD is the fairest scheduler, and it has the best Opengl/Gaming/Compiz/Music performance (i.e. Desktop!).
I am one of those users who actually went out of their way to compare different schedulers.
http://lkml.org/lkml/2007/4/27/1
I did internal testing of every version of SD, and most major revisions of CFS, up until the newest versions.
My experience: SD worked from very early on (since the rsdl days) really well. CFS took a lot of patching to fix my “smoothness” issues. And it’s still not fixed. But I give credit to Ingo for replying to my emails to try and get this problem fixed.
Unfortunately, the fix in CFS was to add “smoothness” and “load average” calculations, including “debit” credits to try and make up for CFS unfair nature and improve desktop experience. CFS has so many tunables in /proc/sys/kernel it leads me to believe there are a lot of hacks built in. “hacks” defined as heuristics designed to overcome CFS’s inherently unfair design (from the users perspective).
Meanwhile SD has only 2 tunables. An interactive flag, and rr_interval. The interactive flag simply gives a small priority boost to user-interacting applications, like a spinning compiz cube with video playing (see my tests). But you don’t even need the interactive flag. It was a late addition to SD. SD works fine without it.
But the time for CFS vs SD discussion is over. Linus has once again put his foot down and chosen CFS, thus killing SD and -ck. I am now forced to use CFS since I don’t have the knowledge to maintain SD for myself (though I am seriously considering learning con’s code just to do this).
I can only hope now that Ingo can fix my performance issues with CFS. I’d rather contribute to getting CFS fixed than complaining about SD being superior since Linus rarely (never?) changes his mind.
Oh, and from the posted lkml discussion:
“No, the response on osnews.com only shows that there are a lot of armchair complainers around.” -Linus
Not true in my case! Just because you post to the lkml vs. Osnews doesn’t mean you are somehow better. It’s not like their is a “Desktop Scheduler Benchmark.” Everybody should test out CFS vs SD (a time consuming activity) and give their subjective opinions.
Edited 2007-07-28 17:41
On the lkml [1] Linus wrote:
So the whole argument about how kernel developers think that the desktop isn’t important is totally made-up crap by Con, and then parrotted by osnews and other places.
Quite unfair characterisation, if you ask me. OSNews has clearly showed both sides of the same coin, as we reported on both Linus’ and Con’s standpoint.
Anyway, I guess Linus only cares about the opinions of long-time, die-hard kernel programmers. If you’re just a user, even a user who makes benchmarks like heckhacker1 above, you are an “armchair complainer” and irrelevant.
His good right, but good to know anyway.
[1] http://lkml.org/lkml/2007/7/28/106
Wonder why Linus doesn’t think OSNews is objective?
Thanks for the link, Thom, I was looking for that particular post.
Frankly, from reading through the LKML, Linus is starting to sound like a kid having a hissyfit. The last time I heard a person throw a tantrum of this proportion when someone questioned they’re decision was at a play school.
If I take all of what I have read from Linus into account, I can’t help but agree with Con’s assessment that the kernel is no longer interested in the desktop and just because Linus says otherwise does not mean that he is in tune with the reality of the situation.
“””
If I take all of what I have read from Linus into account, I can’t help but agree with Con’s assessment that the kernel is no longer interested in the desktop…
“””
What I don’t understand is all of this focus on the relatively minor tradeoffs, often tunable, between server and desktop regarding the kernel. When by *far* Linux’s biggest desktop problems lie in user space.
None of us will find desktop Nirvana in a kernel patch, because that’s not where the problem is.
Butters linked to this earlier, and it is one of my favorite links as well. Read the pdf and weep bitter tears at the true enormity of the situation:
https://ols2006.108.redhat.com/reprints/jones-reprint.pdf
And the story on lwn.net:
http://lwn.net/Articles/192214/
I know that people would like it all to be simple. Apply a magic patch to the kernel and beat MacOSX and BeOS. But it’s not going to happen.
The good news is that many of the specific problems pointed out in the paper have been addressed. The bad news is that the examples in the paper are likely only the tip of the iceberg.
The data presented in the paper was taken from Fedora Core 5 test2, which would make it about 2 years old. It would be interesting to see what similar testing today would turn up.
I agree that user space is a screaming, bloody mess but surely, every little helps. No?
So we have some really crummy code out there in user land, does that mean that we should not look into performance on all fronts?
As for a ‘magic patch’, I agree it wont happen but that still does not negate my above statement.
My comment is more directed at all of those people who think that, because they do not see the Linux desktop as being performant, we need to fork the kernel, get rid of Linus, take away those evil kernel developers’ power and replace them with a dev team by and for the people, blah, blah, blah.
Anything but addressing the real problem where it lives.
I’m interested in Linux kernel development, filesystems, cpu and io schedulers, etc. else I would not have followed this whole CFS vs SD soap opera.
But I don’t kid myself for a moment that a new scheduler can do anything but act as a sort of band-aid.
And as an admin of multiuser XDMCP servers, I have a vested interest in how all this turns out. Anyone who thinks that getting the desktop right is tricky should try managing a 50 user XDMCP server sometime. (Exercise for the reader: Is such a machine a desktop or a server? In the unlikely event of a desktop/server fork, which kernel should it use? Or is there really any fundamental distinction. Think about it.)
I’m cautiously optimistic about the new scheduler. But if I do have problems, I don’t really want some code-proud and complacent developer telling me that I’m just a corner case.
Again, I see where you’re coming from but I don’t agree with you’re implication that Con was ‘code proud’.
The guy came up with a great design but had it rejected, for whatever reason, only then to promptly find out that another maintainer either reimplements his design off his own bat or is told to do so by the powers that be. This design is then merged even though it’s considered by many others to be much more of an ugly hack than the original attempt.
Linus then states his case and is questioned by his own people as to the validity of his arguments. He then goes on to attempt some form of character assassination, which again his own people refute point for point.
Nobody tried to look at the code and see if it could be maintained by the current schedule maintainer, instead it seems like it’s just passed off as some half asses attempt by a green hacker.
I really don’t see much code pride here, just a guy who has been treated very wrongly for trying to help.
Ok, so he walks off in a huff. Being slapped in the face generally has the effect of pissing people off.
You call it character assassination. I call it drawing reasonable conclusions based upon observed behavior.
At any rate, it looks like Con might have given up too soon. He’s no doubt dug himself a bit of a credibility hole to climb out of. But with a positive attitude and improved communication, all hope is not lost for SD. Which means that if anyone has the method and means to run controlled, verifiable, quantifiable tests, now is the time to do it and present the data:
http://lkml.org/lkml/2007/7/28/117
ok, I have no love for Linus as a person. in many of the mailing-list discussions I’ve read he is quick to call people with different opinions morons, which to me makes him bit of an arrogant ass.
that said, I have no doubt of his technical expertise and passion in making the Linux kernel as efficient as possible (which in the end is what really interests me). so I can’t really see him clinging on to a solution if it has been proven to be subpar.
I’m also a firm believer that technical disputes should be settled with technical data. so here’s hoping we’ll see a good thorough study of these two schedulers in action so that we can make our arguments on a purely technical basis rather than our personal feelings regarding Con, Linus or Ingo.
Sure it was ;-). He based his conclusion on one single situation. Pretty much everybody who replied to Linus’ statement has pointed out that maybe he should have checked out what else was going on and how Con was actually more than happy to help with every other question asked.
Linus himself admits that the question asked was a no brainer (a fare scheduler acting fair, who would have thought?) and even goes on to say that maybe he got it wrong. (Link to the same post as above http://lkml.org/lkml/2007/7/28/117)
But check this out. Linus says that Con spouting off about kernel hackers, i.e. himself, gives credit to his conclusions. Con only started blowing a gasket after he had repeatedly been treated like crap.
Go figure.
He based his conclusions upon that portion of SD development that occurred on LKML. You know, that list that is supposed to be used for kernel oriented development discussions, though some seem to mistake it for debian-legal, these days. If Con was receptive to criticism on the ck list, but not so on LKML, then that was a grave error on his part, and was possibly due to the “mind-meld” effect that Linus points out in the link. It’s not as though Linus was the only one who noticed Con’s defensiveness in the face of a perfectly valid regression report:
http://lwn.net/Articles/226054/
None of the schedulers we have seen are actually fair, btw. As Linus, pointing Con in the right direction, suggests to him here:
http://article.gmane.org/gmane.linux.kernel/503916
It’s quite a stretch to say that Con was spouting off about Linus, when his disparaging statements in the apcmag article are quite general in scope:
“”
I think the kernel developers at large haven’t got the faintest idea just how big the problems in userspace are.
“”
At any rate, Con’s gone. Stomped off into the sunset. And perhaps now work can continue, more harmoniously and productively.
Funny thing is, I run Linux in (multiuser desktop) environments which should show up any shortcomings in the scheduler quite clearly. And if you had asked me before this who saga began what I thought Linux needed to improve the desktop, a new cpu scheduler would not even have been on my list. In fact, the only kernel related issue that *ever* causes my users problems is that we are running 50 desktops on 4GB of memory. (Another 4GB is on order.) And while 4GB is fine and dandy most of the time, if a user suddenly requires the maximum allowed amount of memory (256MB per process) the swapping is excessive. I believe that part of the problem is running out of memory in zone normal on that 32bit system. I’ve upped lower memory protection for now and am considering moving the machine to 64 bit soon. So that may be my fault for sticking to 32bit for too long.
In my experience, the existing kernel is great for both desktop and server. It is user space that needs more work.
Edited 2007-07-29 15:32
Looks like we are going to have to agree on disagreeing here. Thanks for the link, very helpful but I still stand behind my voiced opinion as to how this situation was handled.
I’m glad we were able to have a discussion without it turning into a flame war. 🙂
Edit: Cheers Thom 😉
Edited 2007-07-29 16:40 UTC
I’m glad we were able to have a discussion with it turning into a flame war. 🙂
‘Without‘, right?
Well, the way I look at it, no matter what people think of Linus’ style, it has *worked* in the sense that he has pretty successfully led a herd of *very* smart and headstrong cats for 16 years now and is still effective and respected by most kernel devs. No one is poised to take his place. Some people think that he is pompous and arrogant. But it’s really hard to imagine a milquetoast succeeding in that role. He’d make a good thespian, I expect.
And in the final analysis, what matters is the level of respect that he maintains among the kernel dev team. Flamewars between users on OSNews are, in the end, mostly inconsequential in the grand scheme of things.
If the Daleks retroactively erased OSNews completely from the timeline, I doubt it would make much of a difference to anyone but us.
Edited 2007-07-29 17:36
Thom,
You don’t like Linus’ post because it calls _you_ insane. (Admittedly, in Linus’ typical untactful style…)
You linked in a prior post about this topic that you have been advocating splitting the Linux kernel codebase for years. This is your post:
http://cogscanthink.blogsome.com/2005/12/20/split-the-kernel-up/
And Linus writes,
“People are suggesting that you’d have a separate ‘desktop kernel’. That’s insane. It also shows total ignorance of maintainership, and reality. And I bet most of the people there haven’t tested _either_ scheduler, they just like making statements.”
He’s referring to you, but not by name.
I used to work on kernels, and when I read your blog post, I thought the same thing. “Split off a separate ‘desktop’ kernel? That’s ridiculous.”
The only thing that makes a “desktop” special are its workloads. In terms of hardware, desktops and servers use much of the same. Desktop users often want to use “server” hardware on their desktop (e.g., geeks) and server users often want to use desktop hardware (e.g., building commodity servers a la Google). Handling desktop workloads is a big engineering effort, but not one worthy of splitting off the kernel codebase, for crying out loud! It’s basically just the scheduler (and maybe _some_ I/O and memory stuff) that needs enhancement/fine-tuning.
So, answer Linux’s questions, why don’t you?
1. have you ever maintained an open source project of significant size? If so, did you ever encounter a feature you thought was worthy of creating an “internal fork”?
2. Have you tested either the SD or CFS scheduler?
3. Do you enjoy commenting on technology for its own sake?
From my vantage point, I think Linus has you pegged spot-on.
Listen, I’m not a huge fan of Linus and his [lack of] diplomacy, but in this case, you just don’t like the fact that he’s taking a dig at your IMO uninformed statements about what direction this massively complex kernel should take.
I used to work on kernels, and when I read your blog post, I thought the same thing. “Split off a separate ‘desktop’ kernel? That’s ridiculous.”
That’s what came to my mind, too. In the best case scenario, you would end up with something like the Compiz/Beryl situation. Worst case, it would be a total disaster. The CK patchset was already basically a desktop oriented version of the linux kernel, so there’s no need to really create a completely seperate one. I wouldn’t mind seeing two versions of certain subsystems, each oriented towards one type of workflow, but maintaining a completely different kernel would be a nightmare. Linus seems to think that is a bad idea, though, and argues that making a best fit subsystem is better than two different ones.
I don’t see the Compiz/Beryl situation as bad at all. There were problems with Compiz so it forked, and now they’re merging again, once the issues have been resolved. What’s the problem there?
I agree on the splitting kernel thing being insane. What’s the point of splitting the kernel into different branches ? I would really be interested in the concrete reasons. I don’t see any myself. I can see many reasons why not (incompatibilities growing between the source trees being one).
Linus has been saying several times that the desktop is more interesting for him than a server, much before this whole thing about scheduler. But anyway, most of the problemes any OS based on Linux has right now for the desktop user is in user land.
I never said that it would be easy or that it could be done overnight. The Linux kernel is a huge monolithic beast, and splitting it up into three different branches isn’t something you can plan and complete in a few days, weeks, or even months. It’s a process that might take years.
Take a look at Linux running on PDAs and phones (the embedded Linux kernel). You only have to read any review of such a device (Eugenia has written a lot about it) to realise that these devices simply do not perform as well as devices that run specialised embedded operating systems/kernels such as Windows CE/Mobile or Palm devices.
And why to these perform better? Because the development path they followed specialised in their specific use pattern: on a mobile, resource constrained, embedded device. Of course, it’s great that Linux is such a good jack-of-all-trades, but it doesn’t take ten years of die-hard kernel programming experience to realise that specialised products will always do the trick better than generalised products.
A modern SUV such as for instance the Toyota RAV4 also tries to be a jack-of-all-trades in that it needs to handle well on both rough terrain as well as decent roads. The consequence of this is that it will suck on both compared to a proper road car, and to a real off-roader.
It’s the inevitable trade-off, a trade-off that *could* be solved by taking the harsh decision to let the Linux kernel evolve into three different branches, and let is *specialise*. Alowing this to happen, would mean that a programmer working day and night on optimising the kernel for his phone, would not have to take into account that his changes might break stuff on the server or desktop side of things. It allows for greater freedom and more breathing room to merge changes for specific use cases that would otherwise be left out because they would be too use-case specific.
I *never* said it would be easy, nor that it can happen overnight. I also clearly said it will most likely not happen at all – but that doesn’t mean my reasoning behind it is flawed.
I still don’t see any argument for splitting the source tree. By the way, splitting the source tree is trivial: I don’t know git, but it is supposed to do branching really easily.
All you say may well be true (embedded devices not running well on Linux, etc…), but this is nothing which cannot be improved while keeping the same source tree. Building different kernels does not mean not going from the same source tree.
If you split the source tree, what will happen really soon is that the API will be incompatible (except if you merge often, but then, what’s the point of having forks as you suggest ?), and then for example the drivers of the desktop OS won’t work for the server kernel. Basically, the whole thing about breaking API too often will often much more, not much less. Also, some people who may find a nice algorithm for network for the embedded kernel will think “maybe this can benefit other kernels”, but then, the merge will be impossible because of the different source trees.
The key, as Linus said, is modularity: the subsystems are modulars. From the same source tree, you can build small kernels by removing many parts (the 2.6 release helped a lot for this). So you have already the benefits of modularity (which I guess is what you are looking for by splitting the source tree); some things are not modular ? Then make them modular, but in the same source tree.
Thom,
Specialized OSes will _usually_ perform better for their platform than non-specialized OSes (but not always). You need a better argument than performance to specialize.
And for embedded vs. desktop/server, I think the hardware (and physical constraints) are different enough that it may warrant two separate codebases. But for desktop/server, this is certainly not the case.
You may think that the Maemo platform built atop the Linux kernel is not performant, or whatever, but you’re looking at it the wrong way.
Maemo serves as an example of the _power_ of Linux’s single code base. You can run many applications with a simple _recompile_, unmodified. For VM/interpreter-based applications (like Python/Perl), they run by simply copying the script and executing it.
Try that with Linux/Windows/MacOS -> PalmOS, or Linux/Windows/MacOS -> Windows CE.
For crying out loud, you can’t even do Windows -> Windows CE or MacOS -> iPod/iPhone OS that easily, and those are OSes made by the same company!
the more i keep readying things like this. the more anxious i get about open solaris maturing….
though still i wish linux all the best none the less
So solaris is not mature and Linux is?
haha no but open solaris is not mature. Solaris is super mature and a nice solid code base.
From Linus’ post:
“”
[ I realize that this comes as a shock to some of the SD people, but I’m
told that there was a university group that did some double-blind
testing of the different schedulers – old, SD and CFS – and that
everybody agreed that both SD and CFS were better than the old, but that
there was no significant difference between SD and CFS. You can try
asking Thomas Gleixner for more details. ]
“”
More details on this study would be most interesting. If true… and considering that the other evidence we have is merely anecdotal… it seems to me that if people really wanted to get to the truth of the matter regarding the merit of the respective patch sets, there would have been more interest in uncovering the details of this study.
My previous comments about Con’s lack of reliability still stand. But if there is, in addition, no clear winner on performance, despite Con’s big head start, then the proper decision seems even more clear.
I’m working on it.
Good!!! a status update on this whole issue would be good. Maybe include this one:
http://lkml.org/lkml/2007/7/28/186
http://osnews.com/permalink.php?news_id=18350&comment_id=259344
Etcetera. I’m willing to help if you’re writing an article, but (like many) I might be biassed… Contact me if you still want input.
sbergman27: Let’s assume Linus is right and there is no real difference in CFS an SD output. Then let’s look at the size of patches: CFS = 217k, SD = 92k. Which means that SD, compared to CFS, has 125k-sized code with zero bugs and zero maintenance cost. Pretty superior, one can say.
Then you address Linus actions as “proper decision”. Well, considering such bad press for Linux. And ck part of community being simply angry. Not to mention talented developer leaving project (and remember that code can be replaced completely, but people only to some degree — every decent IT project manager knows that). I must say that, so called, “proper decision” looks rather like amusing incompetence.
“””
Well, considering such bad press for Linux.
“””
Outside of the Con Kolivas fan club and perhaps the odd sensationalist headline, this event is being taken as an affirmation that the system is working.
“””
And ck part of community being simply angry.
“””
Let them be angry.
“””
Not to mention talented developer leaving project (and remember that code can be replaced completely, but people only to some degree — every decent IT project manager knows that).
“””
Finally something relevant. Every decent IT project manager knows that an uncooperative, poor team player, who quits when he doesn’t get what he wants, abandoning code which was his responsibility, is more trouble than he is worth. And avoids becoming any more dependent than absolutely necessary upon said individual.
Yes. It was the proper decision. That is becoming ever more clear.
Edited 2007-07-28 21:59
About the press: I beg to differ. Nevertheless, even if bad press is limited to “odd sensationalist headline”, it is bad press after all.
“Let them be angry.” you say. Do you know you are really saying “Let customers be angry”? Even if you don’t see market and profit in background Linus should. Not to mention that is rather above “Joe Average” level community — they can be helpful.
Oh, that “poor team player” argument. You’re plain wrong here. First Con couldn’t be poor team player because he wasn’t team player at all. He was never accepted into the team. Second, if one can not use talented developer I don’t consider him decent. Linus could end up with both Con and Ingo working on scheduler. Two team player instead of one, both with good knowledge of solution. This is what is called “redundancy” or “backup”. And the solution could have been better than both SD and CFS.
But Linus failed.
Bad press here isn’t a problem because nobody cares about this except us supernerds.
“””
Bad press here isn’t a problem because nobody cares about this except us supernerds.
“””
Friendly heads up. You misspelled soapoperanerds. 😉
Are you sure we read the same article here? Your interpretation seems a bit strage to me. I would say there was blame on both sides of this, but I have also been finding Linus’s threads to be more and more defensive as time passes.
I have to add that all of this makes me glad that I have settled on BSD rather than Linux. The ego’s in the Linux world seem to be approaching those of the compitition and we don’t need that. All of the people maintaining this stuff need to step back, set aside their ego’s and put the end user first if they ever want to see Linux supplant Windows as the dominent OS.
Edited 2007-07-28 22:09
Hear hear!
Although the day Linux surplants Windows as the dominant OS too me seem a way off.
Anyway, why would Linux want to surplant Windows? Is that really they’re goal? I would rather they hoped to achieve a better, not necessarily a more popular OS (and some could even argue they have).
Just my €0.02
The problem with Linus is that he thinks it’s a god. May be he’s right choosing CFS over SD, but he always speak like he knows ALL… did you remember their ‘speech’ about the KDE-GNOME desktops?
Linus may be whatever you think it is, but in many cases he acts like the manager at your work: he never wants to discuss about the deep technical problem that’s being solved, instead, he screams in political terms.
“The problem with Linus is that he thinks it’s a god.”
Hmm, well, I don’t think quite so. But it might perhaps be that he has gotten a bit more arrogant over the years (or at least that seems to be what many people feel and which may irritate people?) from the humble chap he may have been in the early days of Linux? Although Linus often opposes Richard Stallman for his political focus and having opinions about this and that, Torvalds himself has may have gotten more political and so more like RMS too.
Then again, people like Linus (or RMS) cannot always help it. In such a position one needs to have opinions and it may simply be quite impossible to fulfill everyone’s high expectations all the time. He is also a big free software “celebrity” and lots of people go ask his opinions about various things all the time and even if he would not always want that… Something that such leaders like Linus need in their position, besides of all other many qualities, is diplomatic skills, and I hope they always remember that too. Having said that, I think that Linus has usually been appreciated for his leader and people skills too.
Edited 2007-07-29 11:42
“The problem with Linus is that he thinks it’s a god.”
I’m not a big Linux fan but that’s just bullshit.
“did you remember their ‘speech’ about the KDE-GNOME desktops? ”
Yes, so what? He spoke his mind, big deal. He’s entitled to have an opinion.
“Linus may be whatever you think it is”
I’m pretty sure Linus is a he and he is the guy who decides what goes into the kernel, like it or not. It’s not a democracy.
“he never wants to discuss about the deep technical problem that’s being solved, instead, he screams in political terms.”
This is so patently stupid that I can’t even think of a comment.
I really don’t understand why Linus engenders this much respect. I like his attitude of not imposing his convictions on others and his idea of using Open Source as a collegial peer-review system rather than a religious movement. But I don’t know why he gets so much respect as an OS developer. He reimplemented an existing OS for which very detailed design documents existed and source code was rather easily available. And Unix was never a particularly feature-rich or hard-to-implement OS. This is part of the reason why it was so successful and why there were so many versions of it.
For someone who I think possesses more talent, consider Alex Ionescu: he did an implementation of the NT kernel for ReactOS by reverse-engineering a closed OS with quite incomplete documentation on the actual semantics and function of the system-call layer. And that guy is only 21 or 22 right now, so he has a long career ahead of him.
Or consider Dave Cutler: NT was constrained by compatibility and portability, but VMS, his previous OS, was a beautiful example of hardware and software meant to run together. Example: DEC had high-performance network cards which used elegant interlocked queue structures to talk with the OS so that the hardware and the host CPU could be operating on the data in parallel without corrupting the shared state. An even better example of Cutler’s innovation was the VMS filesystem: the default FS was fully distributed so that all the VAX machines on a network would behave like one system image with many processors. There was of course a strong failure model around all of this, which is why VMS is still used in banks and other industries as a great fail-safe clustered solution.
What I’m saying through all of this hero-worship is that Linus is just one of many great OS designers. And he is not necessarily the king of the pantheon. Sometimes his techinical decisions are wrong and the insular and self-congratulatory nature of the Linux Kernel developers really doesn’t help them make a good kernel. Take the lack of a mainline kernel debugger as an example of what’s wrong with Linus’ technical direction. Or consider the fact that hibernate does not really work reliably and requires a 22-step process to complete.
I personally like the idea behind Con’s RSDL scheduler of making time allocations long ahead of actually running the task rather than through ad-hoc shifting of tasks in a priority queue. This is cearly the next generation of CPU scheduling, but the LKML demeanor really does not seem to welcome new ideas if they make some older developer’s code look woefully inelegant and inept, unless it’s another older developer who’s making the change. Heck, the O(n) scheduler lasted for way too long before they basically just took on NT’s O(1) design. CFS and RSDL are clearly the next generation… and they should be far more deterministic, which is worth a lot in a core part of the OS.
While I don’t pretend to have had contact with Con in the past, I do think that his reaction to CFS being chosen over SD would certainly support the idea that he has a poor ability to take criticism.
I think, in the end, this is a decision based on both technological (for the end-user) and social (for Linus’ interaction with the kernel hackers) concerns. We will know how right it was a year from now.
Linus certainly could have been a bit more clear in his statement, though…
Sometimes, I can’t help but think that lkml isn’t always the best format for this type of communication.
I think Linus’s post was fairly blunt about what went on. He looked at both schedulers and they seemed about equivalent tech-wise, but he trusts Ingo and doesn’t trust CK. Therefore, Ingo gets what he wants and CK doesn’t.
A lot of people are going to be upset by this, perhaps justifiably, but frankly it is the way the world works.
“.. everyone seems to think I quit because CFS was chosen over SD (hint: it wasn’t).” – Con Kolivas
http://lkml.org/lkml/2007/7/28/179
So Con would have been quit even if Linus had chosen his SD scheduler. If so, I think that was good enough reason to choose CFS scheduler in maintability point of view.
Maybe it wasn’t the sole reason for quitting, but maybe he would have continued if SD had been chosen. Continued to take shit from others and continued working towards a better Linux.
Personally I hope that he’ll move over to the *BSD camps if/when he decides to go back to coding.
I think you can consider the SD/CFS issue (not just the choosing one over another, btw) the final drop.
Ah, I really like the way all these things are openly discussed in public, and how all points of views can be seen, and how also Linus takes part in the discussion. One more big plus for open source / free software as I doubt something like this could happen in closed source world in as open manner. Problems cannot be hidden under carpet even if there was a temptation to do so as someone would notice anyway. The result: bugs and problems tend to get fixed. Never mind if it means a bit arguing and quarrelling at times, it is only human, and may even be needed sometimes in order to clear things up and get problems solved properly…
You’ve got a good point here. I never met anyone who has discussed ANYTHING with Bill Gates, yet I’m mailing with Linus Torvalds 😉
“The problem with Linus is that he thinks it’s a god.”
And the best steering men are on shore:-)
And the best steering men are on shore:-)
Because they don’t want to waste their talents on a broken/sinking ship :p
Might be unfair to the devs involved to put their open private discussions up there to click & read easily for the community .. but this following view from Bill Huey in response to Linus I find very interesting & could lead somewhere .. .
http://lkml.org/lkml/2007/7/28/186
Hard to say who’s right or wrong. I think it’s time to move on there’s so much work to do.
No, it’s time to reflect on the way the kernel development process works and see if it can be improved. Con walking away can be an incident, but it can also be a symptom of a deeper problem. That’s what I’m trying to say to Linus, and that’s what others are trying to say as well.
pharmsen@localhost ~ $ gcc-config -l
[1] i686-pc-linux-gnu-3.3.6
[2] i686-pc-linux-gnu-4.1.2
[3] i686-pc-linux-gnu-4.2.0 *
a simple gcc-config <n0> would switch without a hassle to a different gcc version.
uname -a
Linux localhost 2.6.23-rc1-mm1 #1 SMP Mon Jul 30 12:23:17 CEST 2007 i686 AMD Athlon(tm) 64 Processor 3000+ AuthenticAMD GNU/Linux
pharmsen@localhost ~ $ dmesg |grep NX
NX (Execute Disable) protection: active
Thats why i like to use linux and specifically gentoo.
After following some of these quarrels around and inside the Linux community for a while, I’ve been starting to think that maybe the Linux developer community should have a better organizational structure that could deal with social problems better too? I’m not sure what could be the best way to go, but the point is that the Linux is about a lot more than just the code. The health of the Linux community depends on social and other non-technical decisions too.
Big organizations and companies have decades if not centuries ago noticed the need for some sort of personnel management structures, and usually have dedicated personnel staff too. The need for personnel management becomes all the more important as the importance and size of the organization grows.
Hasn’t also the big Debian community been trying to find new ways to improve the social communication and social health of the project after some difficult social disputes within the project? Besides of already having its social contract and many electorial and other such structures.
Linus Torvalds may usually be quite a fine leader, not only in technical but also in social dealings with the developer community. But he is just a one man capable of making mistakes too. Linus might also be run over by a truck tomorrow. What then? I know that there are some plans for Linux after Linus – but I’m not sure if the plans are very well laid out yet?
And you know what? Dictatorship just happens to be a very primitive form of a government, how ever enlightened and benevolent the current dictator (in this case Linus) might be.
Maybe the Linux developer community should improve its internal social structures a lot in some way or another? At least social problems should be discussed and talked about more besides of technical things (besides, quite often technical problems also have their social aspects – just read the comments by both Linus and Con, and you notice it in this case too).
If the social problems are simply expected to be solved by themselves automatically, without any clear management structures to help solve them, I can see only one result: that such social problems increase in amount, size and difficulty as Linux itself grows in size and importance.
In summary: I would like to see more democracy within the Linux community.
Edited 2007-07-30 20:14
Con and Linus are both right… Con has a narrow focus on making the desktop responsive. Linus dislikes this as it would have issues in other departments where Linux is used. He’s looking for a more general purpose solution to satisfy a greater subset of users/uses.
In fact Linus has a narrow focus by ignoring the specialty tasks that would benefit from a specialized kernel/scheduler. My impression is that he’s still under the belief that there can be one kernel/scheduler that will satisfy everyone; which ‘may’ not be true.
If Linux is to be used as a Game Machine OS (playstation, xbox, like devices) it won’t have a kernel that will accommodate a serious DB server. Similarly if Linux is to be used on a heavy DB server, it’s not going to waste time on graphics or user interface.
So it looks to me that there should be room made for specialty kernels tailored to certain tasks.
However, Linux may already have that from what I’ve been reading from all this. Kernels/schedulers are ‘plugin’able so it should allow someone to hook in their own bits. It just may not be part of the base Linux which some people may be arguing for. In which case Linus may have a point in going with Ingo over Con, but I have no feeling as to whether this is right or not.