Linked by diegocg on Mon 16th May 2016 21:37 UTC
Linux

Linux 4.6 has been released. This release adds support for USB 3.1 SuperSpeedPlus (10 Gbps), the new distributed file system OrangeFS, a more reliable out-of-memory handling, support for Intel memory protection keys, a facility to make easier and faster implementations of application layer protocols, support for 802.1AE MAC-level encryption (MACsec), support for the version V of the BATMAN protocol, a OCFS2 online inode checker, support for cgroup namespaces, and support for the pNFS SCSI layout, and many other improvements and new drivers. Here is the full list of changes.

Order by: Score:
Er, you mean 4.6, not 4.5.
by sbike on Mon 16th May 2016 21:49 UTC
sbike
Member since:
2007-08-17

Er, you mean 4.6, not 4.5.

Reply Score: 2

21 million LoC
by Kebabbert on Mon 16th May 2016 22:30 UTC
Kebabbert
Member since:
2007-07-27

It is quite interesting that a single kernel has 21 million lines of code. It could be a problem.
https://en.wikipedia.org/wiki/Criticism_of_Linux#Kernel_performance

The entire Windows NT, including graphics, etc, was 4-5 million LoC.
https://en.wikipedia.org/wiki/Source_lines_of_code#Example

I wonder how big FreeBSD is?

Edited 2016-05-16 22:30 UTC

Reply Score: 2

RE: 21 million LoC
by panzi on Mon 16th May 2016 23:04 UTC in reply to "21 million LoC"
panzi Member since:
2006-01-22

Yeah, but for Windows you have to install all sorts of drivers. For Linux they are (mostly) in the kernel, but as modules, only loaded when needed.

Reply Score: 8

RE: 21 million LoC
by Lennie on Mon 16th May 2016 23:25 UTC in reply to "21 million LoC"
Lennie Member since:
2007-09-22

As someone else already mentioned, most of it is optional, just drivers, modules for different filesystems you don't use, network protocols, etc.

You can actually build as a small kernel for embedded, even though it has been growing efforts to reduce it do exist:

https://lwn.net/Articles/608945/
https://tiny.wiki.kernel.org/

From the FAQ:
"A minimal kernel takes up hundreds of kilobytes"

Reply Score: 5

RE: 21 million LoC
by jessesmith on Mon 16th May 2016 23:38 UTC in reply to "21 million LoC"
jessesmith Member since:
2010-03-11

Seven years ago, FreeBSD had about 9 million lines of code total, according to this Wikipedia entry: https://en.wikipedia.org/wiki/Source_lines_of_code

Not sure how much of that is the kernel, but I'm guessing a fairly large chunk.

Reply Score: 4

RE: 21 million LoC
by Alfman on Mon 16th May 2016 23:58 UTC in reply to "21 million LoC"
Alfman Member since:
2011-01-28

Kebabbert,

It is quite interesting that a single kernel has 21 million lines of code. It could be a problem.


IMHO this "bloat" is a direct consequence of Linus (and others) ideologically opposing stable kernel ABIs. If we had stable ABIs most of the driver code could be migrated into another repository and downloaded on the fly instead of including the source code for all drivers ever in the kernel. The kernel itself would undeniably be leaner & cleaner this way.

I build linux kernels several times a year, I quite dislike the lack of organization and think it's a mess. It's neat that linux can support hardware from the 90's, but all the linux drivers since the dawn of time are merged in with no metadata. Having inadequate kernel documentation means that I have no idea if an option even applies to any hardware sold in the past decade. "Users who don't know whether they need this can choose 'no'"...WTF!? The reason I don't know is because the documentation sucks! Some of you will probably disagree, which is fine, but it is my honest opinion that while it technically works, it is disappointing that Linux does not do this better.

If I were in charge of the kernel, I'd put into place a mandate that all drivers need to be in a friendly database with proper metadata including all supported PCI and USB ids, all known manufacturers and OEM numbers, dates of manufacture that a user might use to visually identify their hardware. Obviously normal users aren't expected to build their own kernels, but frankly better organization would be helpful even for techies & system administrators like me.

Kebabbert, sorry this was a bit tangential to your post. It's just one of those things that keeps bugging me ;)

Edited 2016-05-17 00:04 UTC

Reply Score: 4

RE[2]: 21 million LoC
by dpJudas on Tue 17th May 2016 01:37 UTC in reply to "RE: 21 million LoC"
dpJudas Member since:
2009-12-10

IMHO this "bloat" is a direct consequence of Linus (and others) ideologically opposing stable kernel ABIs. If we had stable ABIs most of the driver code could be migrated into another repository and downloaded on the fly instead of including the source code for all drivers ever in the kernel. The kernel itself would undeniably be leaner & cleaner this way.

Moving the driver code to another repository has little saying on how "bloated" the kernel is. This is 2016, having a few megabytes of source code you never ever look at is hardly a problem. Either you're working on a specific driver, or you're working on the core of the kernel.

Only in the case of where you're changing the ABI does it become relevant whether the drivers are included or not. In this case, having them right at your fingertip becomes very important as its the only way to determine the impact of your change. Now, you want a stable ABI as the solution to this problem - which means that the ABI cannot be changed at all and thus drivers included or not has no relevancy.

Personally I'm also in favor of a stable ABI, but not for how "bloated" the kernel is. As for those config options you complain about, if it was a stable ABI you'd *still* not know if it was needed or not until you knew exactly what your drivers were going to do. I'm pretty sure the Linux project accepts pull request for documentation improvements btw. ;)

Reply Score: 3

RE[3]: 21 million LoC
by malxau on Tue 17th May 2016 04:07 UTC in reply to "RE[2]: 21 million LoC"
malxau Member since:
2005-12-04

Moving the driver code to another repository has little saying on how "bloated" the kernel is. This is 2016, having a few megabytes of source code you never ever look at is hardly a problem.


It's not just about source code volume. Obviously you have to download it, extract it, but then you have to compile it, and because you just "changed" (compiled) so many things you have to reboot. But think about it - USB is an unloadable kernel module. So with a clean ABI for modules, could the headline USB 3.1 10Gbps support be added without a reboot if Linux weren't so strongly monolithic? What about adding a new filesystem like OrangeFS?

Heck, if all drivers and driver updates were independent of kernel releases, would kernel releases still be noteworthy?

Reply Score: 3

RE[4]: 21 million LoC
by dpJudas on Tue 17th May 2016 12:25 UTC in reply to "RE[3]: 21 million LoC"
dpJudas Member since:
2009-12-10

It's not just about source code volume. Obviously you have to download it, extract it, but then you have to compile it, and because you just "changed" (compiled) so many things you have to reboot.

Okay, so for those not working on the kernel its a couple of minutes waiting and a reboot. Boohoo!

For those working on the kernel, it is a one time couple of minutes. Then those .o files are built and it won't rebuild anything from drivers again unless you are working on changing the ABI.

But think about it - USB is an unloadable kernel module. So with a clean ABI for modules, could the headline USB 3.1 10Gbps support be added without a reboot if Linux weren't so strongly monolithic?

I think you're misunderstanding me. I'm not against a more clean ABI, but the funny thing is, with a stable ABI the "bloat" of the core of the kernel would increase. Having to support legacy ABI features because you can't remove them again increases the workload for those working on the kernel. In any case, the size of the kernel source code (because it includes drivers) has absolutely no saying on how bloated Linux is or not.

Reply Score: 7

RE[3]: 21 million LoC
by Alfman on Tue 17th May 2016 04:14 UTC in reply to "RE[2]: 21 million LoC"
Alfman Member since:
2011-01-28

dpJudas,

Moving the driver code to another repository has little saying on how "bloated" the kernel is. This is 2016, having a few megabytes of source code you never ever look at is hardly a problem. Either you're working on a specific driver, or you're working on the core of the kernel.


I get what you are saying, but a few megabytes is way off. Try 730MB with 501MB in c & h files (this latest version is probably even larger).

Only in the case of where you're changing the ABI does it become relevant whether the drivers are included or not. In this case, having them right at your fingertip becomes very important as its the only way to determine the impact of your change.



People who want all the drivers should always be able to get them, I don't have a problem with that at all. However the vast majority of users neither need nor want that. When I first started building linux kernels, I found it confusing as heck and it's only gotten worse. I'm more experienced now and I've built some tools to help automate things, but I find the lack of better organization shocking given the scale and importance of the OS.


It's already a few years after Linus called the kernel bloated, so what's the plan to reorganize the kernel into more manageable pieces? Maybe we can just keep ignoring it because hardware, but the idea of accumulating this "stuff" indefinitely without a plan makes me picture the mountains of garbage in the movie 'Idiocracy'.



Now, you want a stable ABI as the solution to this problem - which means that the ABI cannot be changed at all and thus drivers included or not has no relevancy.

Personally I'm also in favor of a stable ABI...


Yea, I'd like to see a compromise between long term and constantly evolving ABIs. We may have talked about this before. The reason I blame the lack of an ABI for driver bloat is because without a stable ABI the drivers and kernel are so highly dependent on specific versions of each other that they basically need to be bundled together.

Just to clarify this point, hypothetically if each windows build was allowed to be driver incompatible with the prior build (ie using an unstable ABI), then it would no longer be possible/practical to distribute drivers independently of the operating system (say on a cd). The requirement to bundle all drivers with the OS would lead to a much more bloated OS than today. Adding ABI stability allows for separation of drivers and kernel. I think linux could do stable ABIs better than windows - if only linux leaders were willing to make the ideological shift, but I see no sign of that happening.


As for those config options you complain about, if it was a stable ABI you'd *still* not know if it was needed or not until you knew exactly what your drivers were going to do. I'm pretty sure the Linux project accepts pull request for documentation improvements btw. ;)


Users submitting documentation could help some but I feel it really should be a mandatory part of the process for mainline inclusion from the get-go rather than be an afterthought. Also, using text search in *-config is lacking, having complete info in a structured database somewhere would be so much better than what is there now.

Reply Score: 1

RE[4]: 21 million LoC
by dpJudas on Tue 17th May 2016 13:09 UTC in reply to "RE[3]: 21 million LoC"
dpJudas Member since:
2009-12-10

I get what you are saying, but a few megabytes is way off. Try 730MB with 501MB in c & h files (this latest version is probably even larger).

Okay, so it is a gigabyte of data you never have to look at. You make it sound like this is big in this decade.

People who want all the drivers should always be able to get them, I don't have a problem with that at all. However the vast majority of users neither need nor want that.

One mans bloat is another mans feature. The drivers you need are not the same as I need. Unless you design some elaborate download system that automatically cherry-pick the drivers you need, the easiest way forward is to just download 1.5 gigabytes containing all the drivers.

Stable ABI or not, you'd probably want to just include the open source drivers in the kernel source code distribution.

Users submitting documentation could help some but I feel it really should be a mandatory part of the process for mainline inclusion from the get-go rather than be an afterthought.

Open source development doesn't work this way. The ones doing the pull requests really are in charge of what gets changed, so if you deny their patch based on boring requirements there is some chance they won't do another. Those are the breaks when you're not paying the developer to do the work.

Reply Score: 2

RE[5]: 21 million LoC
by Alfman on Tue 17th May 2016 13:58 UTC in reply to "RE[4]: 21 million LoC"
Alfman Member since:
2011-01-28

dpJudas,

Okay, so it is a gigabyte of data you never have to look at. You make it sound like this is big in this decade.


It is big. I feel Linux can do better, these are very low hanging branches.


One mans bloat is another mans feature. The drivers you need are not the same as I need. Unless you design some elaborate download system that automatically cherry-pick the drivers you need, the easiest way forward is to just download 1.5 gigabytes containing all the drivers.


Again, if you want all the drivers ever, then go ahead and get them, I just feel this is a non-argument against improving the organization and build process in ways that would benefit the majority of users.

Stable ABI or not, you'd probably want to just include the open source drivers in the kernel source code distribution.


I'd certainly make them available, but once you've got a stable ABI you'd be able to manage them independently of the kernel, which is a real benefit. For starters it eliminates a ton of "churn" where lots of code needs to be updated just to be remain compatible with the latest kernel. Maybe we can ignore that because someone else is doing that work, however the tight bundling leads to other kinds of problems. On a NAS server, I've been forced to stick with an old kernel because of regressions in an SATA driver. I eventually debugged and patched the latest kernel so I could upgrade, but with an ABI, there's far more flexibility to run the latest code that works correctly without encountering an "all or nothing" dilemma that's implied by the current solution.



Open source development doesn't work this way. The ones doing the pull requests really are in charge of what gets changed, so if you deny their patch based on boring requirements there is some chance they won't do another. Those are the breaks when you're not paying the developer to do the work.


Oh come on now, with proper leadership I know good documentation can get done, look at the BSDs. This kind of initiative needs to come from the top, otherwise the quality of documentation will remain patchy and inconsistent.

Reply Score: 2

RE[6]: 21 million LoC
by kwan_e on Thu 19th May 2016 08:32 UTC in reply to "RE[5]: 21 million LoC"
kwan_e Member since:
2007-02-18

For starters it eliminates a ton of "churn" where lots of code needs to be updated just to be remain compatible with the latest kernel.


I wonder if there could be a tiered solution. Older, less actively maintained drivers can be moved to a stable ABI, while the drivers whose need to take advantage of the latest kernel improvements outweighs the cost of churn can remain on the unstable ABI.

Oh come on now, with proper leadership I know good documentation can get done, look at the BSDs. This kind of initiative needs to come from the top, otherwise the quality of documentation will remain patchy and inconsistent.


The difference is that Linux gets thousands of LOC added and removed per day. Git was basically invented to handle that. The goalposts are shifting constantly.

Reply Score: 2

RE[7]: 21 million LoC
by Alfman on Thu 19th May 2016 16:54 UTC in reply to "RE[6]: 21 million LoC"
Alfman Member since:
2011-01-28

kwan_e,

I wonder if there could be a tiered solution. Older, less actively maintained drivers can be moved to a stable ABI, while the drivers whose need to take advantage of the latest kernel improvements outweighs the cost of churn can remain on the unstable ABI.


I could go for a policy like that.

The difference is that Linux gets thousands of LOC added and removed per day. Git was basically invented to handle that. The goalposts are shifting constantly.



As for GIT, my own preference is that the end result be a structured database because of how easy it becomes to perform powerful adhoc queries. I would accept structured metadata stored in GIT if we could build tools to parse that data - this would obviously require us to enforce some standards.

Most of it would be write-once whenever a new piece of hardware is supported and the most logical time to collect the data is when it's fresh in the author's mind and many people who are actively testing the drivers. Both people deploying linux and those developing it could benefit from this data. Ideally Linus himself would be involved to give the standards credence so others follow.

Reply Score: 2

RE[2]: 21 million LoC
by kwan_e on Tue 17th May 2016 04:10 UTC in reply to "RE: 21 million LoC"
kwan_e Member since:
2007-02-18

If I were in charge of the kernel, I'd put into place a mandate that all drivers need to be in a friendly database with proper metadata including all supported PCI and USB ids, all known manufacturers and OEM numbers, dates of manufacture that a user might use to visually identify their hardware.


I'm pretty sure Linus and others would accept it - if it worked well enough, and there was a technological solution, not just a mandate, and actually improved things.

I would think there's a reason why it hasn't been done - it's time consuming, tedious, and the metadata itself is not always available or correct.

Reply Score: 5

RE[3]: 21 million LoC
by Alfman on Tue 17th May 2016 04:25 UTC in reply to "RE[2]: 21 million LoC"
Alfman Member since:
2011-01-28

kwan_e,

I'm pretty sure Linus and others would accept it - if it worked well enough, and there was a technological solution, not just a mandate, and actually improved things.

I would think there's a reason why it hasn't been done - it's time consuming, tedious, and the metadata itself is not always available or correct.


You know what, this is one of those things I wouldn't mind working on personally. I think there's a lot one could do with a driver database to make kernel/driver optimization & compilation easier and even trivial. New drivers could be compiled with local CPU optimizations and loaded on demand without having to fuss about recompiling the whole kernel, installing it to grub, and booting into it. I can't really afford to do the work without being paid though.

Edited 2016-05-17 04:41 UTC

Reply Score: 2

RE: 21 million LoC
by mmrezaie on Tue 17th May 2016 09:46 UTC in reply to "21 million LoC"
mmrezaie Member since:
2006-05-09

=============================================
Item Lines %
=============================================
./usr 845 0.0042
./init 5,739 0.0283
./samples 8,758 0.0432
./ipc 8,926 0.0440
./virt 10,701 0.0527
./block 37,845 0.1865
./security 74,844 0.3688
./crypto 90,327 0.4451
./scripts 91,474 0.4507
./lib 109,466 0.5394
./mm 110,035 0.5422
./firmware 129,084 0.6361
./tools 232,123 1.1438
./kernel 246,369 1.2140
./Documentation 569,944 2.8085
./include 715,349 3.5250
./sound 886,892 4.3703
./net 899,167 4.4307
./fs 1,179,220 5.8107
./arch 3,398,176 16.7449
./drivers 11,488,536 56.6110
=============================================

Not all the Kernel is going to be used by each hardware node. In many of the cases much more than 80% of the kernel is not going to be loaded in the memory at all. This is why hypervisors make sense take advantage of it and can be nimble and with relatively small footprint on memory.

Edited 2016-05-17 09:47 UTC

Reply Score: 2

RE[2]: 21 million LoC
by chithanh on Tue 17th May 2016 18:03 UTC in reply to "RE: 21 million LoC"
chithanh Member since:
2006-06-18

./fs 1,179,220 5.8107
./arch 3,398,176 16.7449
./drivers 11,488,536 56.6110

Indeed. Picking just the three biggest ones, you only need the filesystem code of filesystems which you actually use. You only need the code for the architecture of your PC. You only need the drivers for the hardware that is actually present in your system.

All other code is not loaded at runtime, and if you compile your own kernel it is not even present as a binary occupying disk space on your system.

LoC count of the kernel is a total non-issue, and those who try to make an issue out of it are either clueless or have an agenda.

Reply Score: 5

RE: 21 million LoC
by abraxas on Tue 17th May 2016 13:09 UTC in reply to "21 million LoC"
abraxas Member since:
2005-07-07

It is quite interesting that a single kernel has 21 million lines of code. It could be a problem.
https://en.wikipedia.org/wiki/Criticism_of_Linux#Kernel_performance

The entire Windows NT, including graphics, etc, was 4-5 million LoC.
https://en.wikipedia.org/wiki/Source_lines_of_code#Example

I wonder how big FreeBSD is?


Linux kernel contains all the drivers for hardware. Windows does not. It should be expected that there are more lines of code in Linux. I'm less comfortable with the fact that with Windows you may be forced to run drivers in kernel space from various third parties.

Reply Score: 7