Linked by diegocg on Wed 16th Mar 2016 08:45 UTC
Linux

Linux 4.5 has been released. This release adds a new copy_file_range() system call that allows to make copies of files without transferring data through userspace; experimental Powerplay power management for modern Radeon GPUs; scalability improvements in the Btrfs free space handling; support GCC's Undefined Behavior Sanitizer (-fsanitize=undefined); Forwarded Error Correction support in the device-mapper's verity target; support for the MADV_FREE flag in madvise(); the new cgroup unified hierarchy is considered stable; scalability improvements for SO_REUSEPORT UDP sockets; scalability improvements for epoll, and better memory accounting of sockets in the memory controller. There are also new drivers and many other small improvements.

There are also new drivers and many other small improvements. Here is the full list of changes.

Order by: Score:
new syscalls
by uridium on Wed 16th Mar 2016 11:10 UTC
uridium
Member since:
2009-08-20

Nice. The copy_file_range() syscall looks interesting, especially for nfs.

Should be interesting.

Reply Score: 2

If Linus ever...
by Johann Chua on Wed 16th Mar 2016 11:45 UTC
Johann Chua
Member since:
2005-07-22

...mandates a stable ABI for the kernel, then Hell Will Freeze Over™.

Reply Score: 1

RE: If Linus ever...
by Bill Shooter of Bul on Wed 16th Mar 2016 12:59 UTC in reply to "If Linus ever..."
Bill Shooter of Bul Member since:
2006-07-14

Don't tell anybody, but that announcement is only 16 days away !!!

Git your hell parkas here:

http://www.amazon.com/Browning-Hells-Canyon-Primaloft-Parka/dp/B018...

Reply Score: 2

RE: If Linus ever...
by Alfman on Wed 16th Mar 2016 14:11 UTC in reply to "If Linus ever..."
Alfman Member since:
2011-01-28

Johann Chua,

...mandates a stable ABI for the kernel, then Hell Will Freeze Over™.


He always says he doesn't want to be locked into a fixed kernel API/ABI, which I understand. Personally I've always thought a good compromise would be to stabilize those for features between major versions.

Honestly, end users won't be too bothered one way or another, but when I used to maintain some kernel code it was a problem that I needed to adapt to API changes almost every new kernel. IMHO, there was no good reason for it.

What I found especially frustrating was that I frequently had to fix problems with 3rd party out of tree modules (ie for aufs) to make them compatible with the linux flavor of the week. I enjoy working at the kernel level but I lack the resources to keep up with arbitrary changes that didn't benefit me. If the kernel were stable for 6 months-1 year then I could have spent a day or so per year instead of per month or less.

Reply Score: 3

RE[2]: If Linus ever...
by darknexus on Wed 16th Mar 2016 14:29 UTC in reply to "RE: If Linus ever..."
darknexus Member since:
2008-07-15

I've always thought a good compromise would be to stabilize those for features between major versions.

Sounds like the BSDs. It'd be a nice way to go about it, except that Linus then has no control over what Red Hat, SuSE, etc will change anyway. It'd be a step forward, but I doubt it'd be enough. Red Hat, SuSE, and Debian at least, might honor the stability but others like Canonical will do whatever they damn well please regardless of what Linus might do. It's a monster at this point.
I do agree though: it is the user and developer facing APIs and ABI that need to remain stable. What goes on internal to the kernel itself isn't nearly as important as our interfaces to it are. The ABI is important however, not just the API. The need to recompile external drivers needs to end.

Edited 2016-03-16 14:30 UTC

Reply Score: 1

RE[3]: If Linus ever...
by cb88 on Wed 16th Mar 2016 14:58 UTC in reply to "RE[2]: If Linus ever..."
cb88 Member since:
2009-04-23

Or stabilize between LTS releases... this might make backports to LTS alot easier as well if the API changes at the LTS release.

Edited 2016-03-16 15:00 UTC

Reply Score: 1

RE[2]: If Linus ever...
by kwan_e on Wed 16th Mar 2016 14:30 UTC in reply to "RE: If Linus ever..."
kwan_e Member since:
2007-02-18

Isn't Linus' (and the kernel community's) view that things should be userspace as much as possible where the ABI there is stable?

And isn't that ironically what most CS purists want - everything in userspace?

Reply Score: 5

RE[3]: If Linus ever...
by ebasconp on Wed 16th Mar 2016 15:18 UTC in reply to "RE[2]: If Linus ever..."
ebasconp Member since:
2006-05-09

Are you suggesting turn Linux into a microkernel? ;)

Reply Score: 4

RE[3]: If Linus ever...
by darknexus on Wed 16th Mar 2016 15:31 UTC in reply to "RE[2]: If Linus ever..."
darknexus Member since:
2008-07-15

Views are one thing. Actions are another. Linus may have that view, but so far there has been no movement towards anything of the sort. Words are cheap. Actions count.

Reply Score: 1

RE[3]: If Linus ever...
by Alfman on Wed 16th Mar 2016 15:55 UTC in reply to "RE[2]: If Linus ever..."
Alfman Member since:
2011-01-28

kwan_e,

Isn't Linus' (and the kernel community's) view that things should be userspace as much as possible where the ABI there is stable?



Linus is obviously the most recognized, but I wouldn't say there's a singular view in the community. I think Linus would agree with you that only the userspace ABIs should be stable. But from everything I've ever read from Linus, he's pretty consistently against moving anything out of the kernel.

For example:
http://www.phoronix.com/scan.php?page=news_item&px=OTYwMA


And isn't that ironically what most CS purists want - everything in userspace?


Ah, the Tanenbaum versus Torvald debate ;)

True CS purists would have been more likely to appreciate Minux. Linux was more pragmatic (at least at the time).

Edited 2016-03-16 16:00 UTC

Reply Score: 2

RE[4]: If Linus ever...
by jessesmith on Wed 16th Mar 2016 16:13 UTC in reply to "RE[3]: If Linus ever..."
jessesmith Member since:
2010-03-11

I agree with what you wrote, but at the end I think you meant to write "MINIX" rather than Minux. The latter is a Linux-based operating system.

Reply Score: 3

RE[5]: If Linus ever...
by Alfman on Wed 16th Mar 2016 17:08 UTC in reply to "RE[4]: If Linus ever..."
Alfman Member since:
2011-01-28

I agree with what you wrote, but at the end I think you meant to write "MINIX" rather than Minux. The latter is a Linux-based operating system.


Haha, yeah.

I don't remember ever hearing about "minux", the linux distro, before:
https://sourceforge.net/projects/minux/

"ISO Image only 24 MB"

That's tiny even by minimalist standards, my own linux build is on a 64MB iso.

Reply Score: 2

RE[6]: If Linus ever...
by zlynx on Thu 17th Mar 2016 20:12 UTC in reply to "RE[5]: If Linus ever..."
zlynx Member since:
2005-07-20

Meh.

I remember booting Linux 2.4 from a pair of floppy disks. That included basic user space with a shell, ls, etc, and the hard disk installer.

I believe 2.0 could boot from one floppy disk.

24 MB is gigantic!

Reply Score: 3

RE[7]: If Linus ever...
by Alfman on Thu 17th Mar 2016 21:09 UTC in reply to "RE[6]: If Linus ever..."
Alfman Member since:
2011-01-28

zlynx,

I remember booting Linux 2.4 from a pair of floppy disks. That included basic user space with a shell, ls, etc, and the hard disk installer.

I believe 2.0 could boot from one floppy disk.

24 MB is gigantic!


Yea, but those where the days when hardware conformed to (pseudo) software standards. It's what made OS dev fun and relatively trivial back then. These days, if you want to support lots of hardware, you need hundreds of drivers. All of them do the same thing, but accomplish it differently. If we want to hard code driver support and reverse course on supporting decades worth of software abstractions, then we could go back to 1MB days (many indy operating system projects do it), but otherwise it's not really practical any more.


I get you are being tongue-in-check though, and to that end: the guys today just can't code like we used to ;)

Reply Score: 3

RE[4]: If Linus ever...
by dionicio on Wed 16th Mar 2016 18:53 UTC in reply to "RE[3]: If Linus ever..."
dionicio Member since:
2006-07-12

;) ;)

Reply Score: 2

RE[2]: If Linus ever...
by Kebabbert on Thu 17th Mar 2016 12:39 UTC in reply to "RE: If Linus ever..."
Kebabbert Member since:
2007-07-27

Johann Chua,
"...mandates a stable ABI for the kernel, then Hell Will Freeze Over™.


[Linus] always says he doesn't want to be locked into a fixed kernel API/ABI, which I understand.
"
Frankly, I dont understand this. All other OSes have stable kernel ABIs, without problems. For instance, Solaris have had stable kernel ABIs for decades, and ordinary software runs back to Solaris 2.11 unmodified. Oracle guarantees this.

Linus says the drawback with frozen ABI/API is that innovation will lag behind. That is not correct. Look at Solaris, it has brought us innovations such as DTrace, ZFS, SMF (systemd has copied this), Containers, Crossbow, etc etc. It IS possible to have stable kernel ABI/APIs and still innovate. Solaris is the most innovative OS today, everybody else copies or ports the Solaris tech. For instance, DTrace:
-Mac OS X has ported it
-FreeBSD has ported it
-QNX has ported it
-IBM AIX has copied it ("ProbeVue")
-VMware has copied it ("vProbes")
-Linux has copied it ("Systemtap")
-Linux has ported it ("dtrace" in Oracle linux distro)
-etc
All the major OSes have copied or ported DTrace. It is the big killer. Not ZFS. And systemd is a copy of Solaris SMF, but done wrong. As BTRFS is done wrong when copying ZFS. It is possible to innovate with frozen kernel api/abi - if you know what you are doing.


The problem Linus Torvalds faced, when he released Linux as a university student after reading Tanenbaums book, was that Linus did not have experience of how to make kernels. So he tried this and that. It would be a mistake to freeze kernel abi/apis, as a unexperienced programmer. So he kept it open. Today Linus has the experience, so he should be able to freeze the abi/apis today. In the beginning he knew nothing, his Linux code was very amateurish.

Linus has said that trial and error is superior of design. Trial and error has evolved humanity from animals. So Linus has said officially that Linux will evolve by trial and error and become superior to all other OSes. "Linux has no design and will never have", he said. It is better to experiment all the time, he says.

This has the drawback that large parts of the kernel is rewritten all the time. And new code is never stable. Whenever some part of the Linux code matures, it gets thrown out. So the Linux kernel is a constantly moving target. Making it very difficult for device drivers to update their drivers. This is why the Linux kernel model is broken, as you can yourself see from all forum threads "Ive upgraded the kernel and now my soundcard/modem/etc does not work".

This makes LTS a broken concept too. If you want to upgrade or install new software, you likely need new library. The new library will trigger upgrades of other software, that requires new libraries, that triggers upgrade of other software. In an avalanche you have upgraded everything and left LTS. The only way to use LTS is to never upgrade nor install new software. That does not work.

Reply Score: 1

RE[3]: If Linus ever...
by Alfman on Thu 17th Mar 2016 14:22 UTC in reply to "RE[2]: If Linus ever..."
Alfman Member since:
2011-01-28

Kebabbert,

It IS possible to have stable kernel ABI/APIs and still innovate. Solaris is the most innovative OS today, everybody else copies or ports the Solaris tech. For instance, DTrace:


Yes of course. Innovation and stable APIs are two different variables, one doesn't follow from the other.

The problem with stable APIs, over the long term, is that they get stale regardless of whether they were innovative or not. Some parts of posix have not aged well at all, same goes for win32s and java's class library, c++ libraries, android, etc. Previous assumptions no longer hold, yet because software depends on them now, legacy designs sort of 'cement' an API. What usually happens is an API starts to become redundant and bloated. Panzi's example of copy_file_range() and sendfile() is a perfect example of this. Some historic API assumptions are notoriously problematic. Yet sometimes we are forced to do the best we can with the frameworks that were designed decades back.

I don't have to tell you, there are definite downstream costs when things are never stable, which I can attest to myself. I don't think Linus cares enough to do it, but he ought to compromise and keep things stable within major versions, lets say. So code written for X.Y should at least continue to work on X.Y+1. X.Y may not run on X+1. So kernel developers may need to update once a year or two, it's much better than potentially every release.

Reply Score: 2

RE[4]: If Linus ever...
by Kebabbert on Sun 20th Mar 2016 17:08 UTC in reply to "RE[3]: If Linus ever..."
Kebabbert Member since:
2007-07-27

Kebabbert,

"It IS possible to have stable kernel ABI/APIs and still innovate. Solaris is the most innovative OS today, everybody else copies or ports the Solaris tech. For instance, DTrace:


Yes of course. Innovation and stable APIs are two different variables, one doesn't follow from the other.

The problem with stable APIs, over the long term, is that they get stale regardless of whether they were innovative or not. Some parts of posix have not aged well at all, same goes for win32s and java's class library, c++ libraries, android, etc. Previous assumptions no longer hold, yet because software depends on them now, legacy designs sort of 'cement' an API. What usually happens is an API starts to become redundant and bloated. Panzi's example of copy_file_range() and sendfile() is a perfect example of this. Some historic API assumptions are notoriously problematic. Yet sometimes we are forced to do the best we can with the frameworks that were designed decades back.

I don't have to tell you, there are definite downstream costs when things are never stable, which I can attest to myself. I don't think Linus cares enough to do it, but he ought to compromise and keep things stable within major versions, lets say. So code written for X.Y should at least continue to work on X.Y+1. X.Y may not run on X+1. So kernel developers may need to update once a year or two, it's much better than potentially every release.
"
Certainly. Every other OS than Linux have stable API/ABIs, even OS/2 had it. Linus Torvalds is alone with this driver model, and maybe he is smarter than every other OS hackers, but I doubt so if we look at all the Linux driver problems in every forum.

Regarding freezing the ABI/APIs, yes sometimes they become obsolete but the advantages outweigh the disadvantages in my opinion. If you are a good designer then you can make a good API/ABI that will stand the test of time. For instance Unix is from 1969 and it is so well designed it is the most successful OS model out there. Sure, Unix has it's quirks, but the Unix designers knew their stuff. It is still valid today! So you can design well, if you know your stuff. If you dont, you do like Linus Torvalds, breaking everything all the time.

Regarding freezing the Linux kernel ABI/APIs in between major revisions, that is a very good idea, but Linus dont know where Linux is heading so he can not freeze Linux: "Linux has no design and will never have".

Reply Score: 2

RE[5]: If Linus ever...
by Alfman on Mon 21st Mar 2016 01:39 UTC in reply to "RE[4]: If Linus ever..."
Alfman Member since:
2011-01-28

Kebabbert,

Regarding freezing the ABI/APIs, yes sometimes they become obsolete but the advantages outweigh the disadvantages in my opinion. If you are a good designer then you can make a good API/ABI that will stand the test of time. For instance Unix is from 1969 and it is so well designed it is the most successful OS model out there. Sure, Unix has it's quirks, but the Unix designers knew their stuff. It is still valid today!



You use unix as a good example, but I actually think something like Plan 9 is what unix should have been. Of course it was way too late by the 90s. Not to nitpick on unix, lots of legacy tech persists today with both good and bad legacies. C/C++ header files are just awful. But that's just the way it goes, hindsight changes everything.


Regarding freezing the Linux kernel ABI/APIs in between major revisions, that is a very good idea, but Linus dont know where Linux is heading so he can not freeze Linux: "Linux has no design and will never have".


Yea, I think it would benefit linux to have more planning, but ultimately it would require changes in the community that won't happen if Linus refuses to go along.

Reply Score: 2

"Full list of changes"
by Carewolf on Wed 16th Mar 2016 14:40 UTC
Carewolf
Member since:
2005-09-08

I love how the traditional summary of the most important changes have become the "full list of changes" ;)

Reply Score: 2

copy_file_range() Vs sendfile()
by panzi on Wed 16th Mar 2016 16:58 UTC
panzi
Member since:
2006-01-22

What's the difference between copy_file_range() and sendfile()?

Reply Score: 2

RE: copy_file_range() Vs sendfile()
by Alfman on Wed 16th Mar 2016 22:40 UTC in reply to "copy_file_range() Vs sendfile()"
Alfman Member since:
2011-01-28

panzi,

What's the difference between copy_file_range() and sendfile()?


At a casual glance, it's seems to me they added the ability to seek in the destination and flags.

ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);

ssize_t copy_file_range(int fd_in, loff_t *off_in, int fd_out, loff_t *off_out, size_t len, unsigned int flags);

Sockets never need to seek anyways.
Sendfile already covers the situation of reading a file into a socket.
It seems rare that you'd need to copy data from a socket/file descriptor into a file at a random offset, and do it many times, but in such a case this could save a seek syscall.

The main motivation was probably the new flags:
https://lwn.net/Articles/659523/
COPY_FR_REFLINK asks for the destination file to refer to the existing copy of the data without actually copying it. Some filesystems (Btrfs, for example) are able to share references to file blocks in this way.

COPY_FR_DEDUP is like COPY_FR_REFLINK, but it only succeeds if the destination range already contains the same data as the source. The end result is files that look the same as before, but which are now sharing the data on-disk. It is thus a way of removing blocks of duplicated data within the filesystem.


I kind of wish there was more attention to asynchronous use cases, who knows when these features will become available for AIO development. AIO in linux always seems like an afterthought.

Edited 2016-03-16 22:42 UTC

Reply Score: 2

v 1
by Anonymous on Thu 24th Mar 2016 14:36 UTC