A recent discussion on the lkml looked into a reproducable random file I/O regression in 2.6 compared to the 2.4 kernel. Alexey Kopytov posted the benchmark results, attempting to simulate the workload of a database under intensive load. The tests were tried with all I/O schedulers, including the anticipatory, deadline and CFQ, and in all cases 2.4 outperformed 2.6. Read the report at KernelTrap.
Linux: Random File I/O Regressions In 2.6
About The Author
Eugenia Loli
Ex-programmer, ex-editor in chief at OSNews.com, now a visual artist/filmmaker.
Follow me on Twitter @EugeniaLoli
Love watching the kernel dev’s in action. You know there’s gonna be a solution out within a week.
And then a better solution next week.
…in its Enterprise offerings. Imagine the CTO of some Fortune 500 calling Redhat and asking why their server is slower since upgrading to better software.
It’s obvious that the people developing the kernel basically have no idea what they are doing. Ripping out and replacing VM subsystems, readahead algorithms, etc. Hack after hack after hack. With hardly any performance testing. And then it performs worse than previous kernels and we watch the people involved piss into the wind at the reason.
Solaris solved all these problems a decade ago.
”
Solaris solved all these problems a decade ago. ”
thats too good for them. tell the SUN guys that
What, praytell, does this look like to you, if not performance testing? The fact that you can see all the in-progress work is simply a function of the development model. The 2.6 kernel is still early in its life, while the 2.4 kernel is very well-tested and mature. There are bound to be regressions here and there.
That’s cool, maybe you and Sun’s other customer can go have a beer and talk about it. LOL
“Hack after hack after hack” – The OS and browser combo that you posted your comment with.
I use FreeBSD. Design before you code, you dummies!
What, praytell, does this look like to you, if not performance testing? The fact that you can see all the in-progress work is simply a function of the development model.
I don’t know, when someone marks something as stable, and ready for production servers, I expect that they have done basic testing to make sure there are not regressions such as this one which can lead to filesystem corruption and remote root exploits. I suppose I should send a note to my customers- “sorry for the downtime, this is simply a function of the development model.”
walk out of your shell. freebsd 5.x had a lot of regressions too.
“don’t know, when someone marks something as stable, and ready for production servers, I expect that they have done basic testing to make sure there are not regressions such as this one which can lead to filesystem corruption and remote root exploits”
there was no filesystem corruption and remote root exploits in kernel 2.6. btw a stable series means that new features wont be added and its meant towards being stable not that it will be stable immediately on it first few releases.
these regressions mean that under some work load conditions 2.4 performs better. what has that do with downtime.
I don’t know, when someone marks something as stable, and ready for production servers, I expect that they have done basic testing to make sure there are not regressions such as this one which can lead to filesystem corruption and remote root exploits. I suppose I should send a note to my customers- “sorry for the downtime, this is simply a function of the development model.”
My feelings exactly. It might seem like a minor point to some, but damn, both the FreeBSD and Linux people are really bothering me by doing this. It causes needless confusion among the newer users, and it leaves many of them with really bad impressions of the systems. Its one of the many things that soured me on Linux.
I don’t recall reading anything about filesystem corruption in this particular article though…
“. It causes needless confusion among the newer users, and it leaves many of them with really bad impressions of the systems. Its one of the many things that soured me on Linux. ”
like I said before there is a basically a issue of performance not stability or security. so what is your confusion about?
Anybody familiar with Linux knows that a ‘stable’ release isn’t necessarily mission-critical ready. A stable release is made when most of the major changes have been ironed out, and the kernel needs to see a wider level of use and testing. Since Linux doesn’t have dedicated QA people, the first several stable releases are a way to get early-adopters to iron out the bugs for everyone else. If you use Linux in the enterprise, you know to wait until SuSE and RedHat put the 2.6 kernel through their QA process and ship an RHEL or SLES with their version of 2.6. This isn’t anything new — its been this way for a very long time.
Read again. I’m not confused.
“Read again. I’m not confused.”
ok. now can you explain why new users should be confused?
Solaris solved all these problems a decade ago.
Oh riiight. So *that’s* why Solaris can’t scale up to 512 CPUs and 4TB RAM. Better tell those Solaris engineers that time doesn’t stand still for them.
walk out of your shell. freebsd 5.x had a lot of regressions too.
True but it’s not like the FreeBSD team was claiming that 5.x was stable either. Kernel/OSes that claim to be stable should be held to higher standards (IMO).
“Read again. I’m not confused.”
ok. now can you explain why new users should be confused?
Stop it, guys, you’re confusing me…
No one is saying that users should be confused. Just that users won’t be confused because that’s only something that matters for big databases on heavy load. I’m currently using kernel 2.6.3 on my Linux PC, and I could care less about this particular issue. I’m not going to regress to kernel 2.4.
I think Rayiner said it best – kernel 2.6 is ready for general use but it’s not as finely tuned for some specific tasks as the later 2.4 kernels (though it is superior to the 2.4 series on other specific tasks…).
Comparisons with FreeBSD are pretty much meaningless as the two kernels follow different evolution processes.
ok. now can you explain why new users should be confused?
Damn you like to cause trouble. Calling a release production ready while it really is not. WTF is so hard to understand about that?
In fairness, at least the FreeBSD folks have made no such claims, unlike the Linux ones and their ‘legions’ of followers.
True but it’s not like the FreeBSD team was claiming that 5.x was stable either. Kernel/OSes that claim to be stable should be held to higher standards (IMO).
a) this was a small oversight and the test is a very obscure case that seems to have not bitten anyone until now.
b) are you claiming that when FreeBSD 5 goes stable, it will either have no bugs?
c) no regressions vs FreeBSD4? If so, you are a brave man and I would like to make a bet with you for a large sum of money.
Damn you like to cause trouble. Calling a release production ready while it really is not. WTF is so hard to understand about that?
In fairness, at least the FreeBSD folks have made no such claims, unlike the Linux ones and their ‘legions’ of followers.
Damn dude, you like to cause trouble too. Look, it really isn’t hard for someone to have a look at the FreeBSD -stable mailing list and see all the problems people are having with their stable kernel too.
If you think FreeBSD-STABLE has no bugs you must be living under a rock.
Here’s the most recent one I saw. Something about SMP instability introduced into 4.9. A little bit worse than a performance degradation that nobody has seen until now, and nobody has yet seen in a real workload (this one was a benchmark).
http://marc.theaimsgroup.com/?l=freebsd-stable&m=108154379325132&w=…
“Damn you like to cause trouble. Calling a release production ready while it really is not. WTF is so hard to understand about that?
In fairness, at least the FreeBSD folks have made no such claims, unlike the Linux ones and their ‘legions’ of followers. ”
a regression in i/o clearly means its not production ready?.
there are two things you need to be clear about
1) stable doesnt mean production ready
2) nobody is calling linux 2.6 production ready yet
If you think FreeBSD-STABLE has no bugs you must be living under a rock.
Here’s the most recent one I saw. Something about SMP instability introduced into 4.9. A little bit worse than a performance degradation that nobody has seen until now, and nobody has yet seen in a real workload (this one was a benchmark).
Shit I hate having to be defensive. I’ve never claimed FreeBSD to be buggless! Use your fucking head!
Shit I hate having to be defensive. I’ve never claimed FreeBSD to be buggless! Use your fucking head!
And yet you implied that Linux 2.6 was not production ready due to this small performance degradation, and that FreeBSD’s stable releases didn’t suffer from such things. Don’t try to get smart, we both know what you implied.
Sorry, I don’t know what is acceptable by FreeBSD standards, but I’ll take a performance regression in a system that isn’t crashing left and right, thank you.
And yet you implied that Linux 2.6 was not production ready due to this small performance degradation, and that FreeBSD’s stable releases didn’t suffer from such things. Don’t try to get smart, we both know what you implied.
Just restating what I’d read during it’s release, OSDL blurbs and all.
Sorry, I don’t know what is acceptable by FreeBSD standards, but I’ll take a performance regression in a system that isn’t crashing left and right, thank you.
Who has an unstable system? 4.9 users? 2.6.x users?
But whatever. PR was never a strength of OSS folks, and I guess it never will be.
“Just restating what I’d read during it’s release, OSDL blurbs and all. ”
OSDL said 2.6 is not production ready. yes. thats true. but thats not because of this regression. you were implying that and thats simply false.
A system which crashes is unstable
a i/o regression is a performance problem not a stability issue. so it should be pretty clear what we mean here
a) this was a small oversight and the test is a very obscure case that seems to have not bitten anyone until now.
That’s very true but it should at least give a lesson to the hordes of rabid zealots that were biting RedHat for not adopting 2.6 earlier.
b) are you claiming that when FreeBSD 5 goes stable, it will either have no bugs?
c) no regressions vs FreeBSD4? If so, you are a brave man and I would like to make a bet with you for a large sum of money.
Of course I don’t but I’m pretty sure the developers won’t have to fix something because it has gotten too band-aidy. Don’t get me wrong: I love Linux. I just think that they tend to commit stuff without enough testing sometimes.
Just restating what I’d read during it’s release, OSDL blurbs and all.
Well how about you use your fucking head then? Try to think for yourself sometimes.
Who has an unstable system? 4.9 users? 2.6.x users?
Umm…
This 4.9 user has an unstable system:
http://marc.theaimsgroup.com/?l=freebsd-stable&m=108154379325132&w=…
This 2.6 user does not:
http://kerneltrap.org/node/view/3039
“But whatever. PR was never a strength of OSS folks, and I guess it never will be.”
how is that relevant to the topic?
This 4.9 user has an unstable system:
http://marc.theaimsgroup.com/?l=freebsd-stable&m=10815437932513…
This 2.6 user does not:
http://kerneltrap.org/node/view/3039
Umm, let me just clarify something here. I don’t mean to pick on FreeBSD. By most accounts it is one of the most stable systems out there, and the -STABLE series is quite likely to be more stable than 2.6 if nothing else than due to the amount of time it has been out there.
I am just trying to talk sense into the misguided people who think FreeBSD somehow would never suffer from anything like this.
A system which crashes is unstable
a i/o regression is a performance problem not a stability issue. so it should be pretty clear what we mean here
I see where you’re coming from. However, my POV is that there ought to be no regressions from one stable release to another, once the release has been deemed “production ready,” and many folks at OSDL damn well did make that claim.
I guess that by the time Linux 2.6.10 comes out, I’ll have far fewer technical issues with it anyway so what the hell, right?
Umm, let me just clarify something here. I don’t mean to pick on FreeBSD. By most accounts it is one of the most stable systems out there, and the -STABLE series is quite likely to be more stable than 2.6 if nothing else than due to the amount of time it has been out there.
I am just trying to talk sense into the misguided people who think FreeBSD somehow would never suffer from anything like this.
One person from each camp hand picked by you just to make a half assed attempt at proving your misguided point?
Whatever.
I just don’t see the point of directing it at me when I’ve never made any of the wild claims that you seem to be attempting to disprove. I don’t care what you bash or who you bash as long as there is some basis in truth. I try to do the same myself.
I see where you’re coming from. However, my POV is that there ought to be no regressions from one stable release to another, once the release has been deemed “production ready,” and many folks at OSDL damn well did make that claim.
Well, there ought to be no regressions, but sometimes things don’t work out like that. See for example the FreeBSD *stability* regression when moving from 4.8 to 4.9 which is only *minor* stable upgrade.
Of course Linux 2.6 has regressions vs 2.4, and it probably still has regressions that haven’t been found yet. And some regressions are even intentional tradeoffs to improve something else.
I guess that by the time Linux 2.6.10 comes out, I’ll have far fewer technical issues with it anyway so what the hell, right?
Well yeah, obviously the longer it has been out, the less problems it should have. What technical issues do you have with 2.6 as of now, anyway?
” see where you’re coming from. However, my POV is that there ought to be no regressions from one stable release to another, once the release has been deemed “production ready,” and many folks at OSDL damn well did make that claim.”
thats idealistic. when there is a huge amount of changes between releases and when 2.6 hasnt got enough end user testing it is likely to have some kind of issues which will be ironed out over time. this particular problem doesnt make 2.6 not production ready. 2.6 is not considered production ready because it hasnt proved itself on many servers yet. that will take time regardless of the stability or performance issues. basically people something old and trusted for servers.
”
I guess that by the time Linux 2.6.10 comes out, I’ll have far fewer technical issues with it anyway so what the hell, right?”
lets see. SUSE is one of the major commerical distros that shipped a linux distro based on 2.6 and it was not for servers. mandrake is also a home user distro. fedora core 2 which will be released on may 17 is not meant for servers either.
so afaik redhat’s next version will ship later this year and is meant for servers and it will probably run something around 2.6.10 or higher. so you are right in a way. if you are running linux on a production server wait for a few months after a “stable” kernel release on a test server and let your distribution ship with a 2.6 kernel if you really want to be on the safer side. this applies to all the kernel releases and new software on the server
Well, there ought to be no regressions, but sometimes things don’t work out like that. See for example the FreeBSD *stability* regression when moving from 4.8 to 4.9 which is only *minor* stable upgrade.
Yeah, that wasn’t cool.
Of course Linux 2.6 has regressions vs 2.4, and it probably still has regressions that haven’t been found yet. And some regressions are even intentional tradeoffs to improve something else.
I’m not entirely convinced that the top guys do nearly as much regression testing as they ought to. Having no direct contact with them I can’t confirm that, but the obvious regressions dicovered during our own (@work) testing seems to indicate that they were lax.
What technical issues do you have with 2.6 as of now, anyway?
Very little that you haven’t heard me say before.
One person from each camp hand picked by you just to make a half assed attempt at proving your misguided point?
My point was that FreeBSD has regressions, even stability problems, even in stable releases (ie. deemed production ready by the release team). I found one such regression. That *proves* my point, why do I need to find more examples? You obviously need to brush up on your formal logic skills.
And so now you admit you DO think my point is misguided. So you DO think FreeBSD has no regressions. I conclude you are living under a rock. Please crawl back under it.
And so now you admit you DO think my point is misguided. So you DO think FreeBSD has no regressions. I conclude you are living under a rock. Please crawl back under it.
Sorry, after reading your last response you actually seem quite reasonable. I apologise for this statement, please try to ignore it.
The software should be tested far more than it has been before the first ‘stable’ release in any given series is made. Sure all software has nearly uncountable bugs, but I’d love to see you find major performance or stability issues in (for example) a Solaris release. Very heavilly tested I’ve heard, and in all my experiences with it, and those of others I’ve talked with who use it at work, can’t find too terribly much wrong with it.
Kindof lacking in the driver department though, to be fair.
Very little that you haven’t heard me say before.
Actually, I don’t think I have heard you… please elaborate.
Sorry, after reading your last response you actually seem quite reasonable. I apologise for this statement, please try to ignore it.
Nobody is perfect. I can forget as easilly as I can be irritated
At any rate, I’ll have to continue this later, yet another long day ahead of me. Have a good night all.
“I’m not entirely convinced that the top guys do nearly as much regression testing as they ought to. Having no direct contact with them I can’t confirm that, but the obvious regressions dicovered during our own (@work) testing seems to indicate that they were lax. ”
the fact that this regression has not been wide spread seems to be indicate to me some fairly obscure workload stuff. i dont think anybody here really has the expertise to comment on the level of regression testing required. you are free to form your opinions thou. i dont share them with you
a) this was a small oversight and the test is a very obscure case that seems to have not bitten anyone until now.
A multithreaded pread() is not a “very obscure case”. I’m currently developing an application which provides network file service. The foremost type of application which would utilize this approach is a database, where you have multiple reader threads which may need to access the same file on disk through the same file descriptor.
c) no regressions vs FreeBSD4? If so, you are a brave man and I would like to make a bet with you for a large sum of money.
Clearly this is not the case… there’s considerable overhead added by the kernel mechanisms that have been added in FreeBSD 5.x, especially on UP systems, but that is in the process of being mitigated by new kernel features which are not yet active per default in a -RELEASE, namely KSE threading and the ULE scheduler.
However, I agree it’s ridiculous to look negatively upon Linux because new kernel features are exhibiting problematic performance in a specific case when overall they are performing much better than their predacessors, especially if the problematic case can be remedied by something as simple as proper locking, as one kernel developer noted.
Overall though, I’m wary of the attitudes exhibited by a number of people on this list. Many seem to want to completely rewrite large portions of the I/O subsystem because they dislike the current implementation, because it seems it’s not ideal and has a history of being problematic. I have no problem with them rewriting large portions of the I/O subsystem so long as their changes are merged only into the upcoming kernel development branch, and are *NOT* merged into the mainline 2.6 kernel, especially if the current implementation can be fixed by something as simple as adding a semaphore. This is a legitimate problem where those advocating FreeBSD can take issue… Linux, at least in the 2.4 series, has had problems with major alterations to the kernel codebase without proper testing, resulting in headaches for sysadmins everywhere. This is a mistake I hope we don’t see repeated in the 2.6 kernel series.