Linked by diegocg on Thu 7th Nov 2013 22:19 UTC
Linux Linux kernel 3.12 has been released. This release includes support for offline deduplication in Btrfs, automatic GPU switching in laptops with dual GPUs, a performance boost for AMD Radeon graphics, better RAID-5 multicore performance, improved handling of out-of-memory situations, improvements to the timerless multitasking mode, separate modesetting and rendering device nodes in the graphics DRM layer, improved locking performance for virtualized guests, XFS directory recursion scalability improvements, new drivers and many small improvements. Here's the full list of changes.
Thread beginning with comment 576452
To read all comments associated with this story, please click here.
OOM Killer
by Alfman on Fri 8th Nov 2013 04:56 UTC
Alfman
Member since:
2011-01-28

Thom, you missed what I'd consider to be one of the bigger highlights: changing the out of memory killer!

https://lwn.net/Articles/562211/

As was described in this June 2013 article, the kernel's out-of-memory (OOM) killer has some inherent reliability problems. A process may have called deeply into the kernel by the time it encounters an OOM condition; when that happens, it is put on hold while the kernel tries to make some memory available. That process may be holding no end of locks, possibly including locks needed to enable a process hit by the OOM killer to exit and release its memory; that means that deadlocks are relatively likely once the system goes into an OOM state.



Linux has never been extremely stable in the past under out of memory conditions, I've always considered the out of memory killer to be major hack for a fundamental problem.

Following a bunch of cleanup work, these patches make two fundamental changes to how OOM conditions are handled in the kernel. The first of those is perhaps the most visible: it causes the kernel to avoid calling the OOM killer altogether for most memory allocation failures. In particular, if the allocation is being made in response to a system call, the kernel will just cause the system call to fail with an ENOMEM error rather than trying to find a process to kill.


About time! The previous behavior of returning a successful mallocs only to have to invoke the linux OOM killer never made any sense. Killing processes heuristically without so much as asking the user first is ugly at best and outright reckless at worst. What will linux kill? How about a document, web browser, X11 session, etc. I've learned to significantly over-provision ram to avoid the OOM, but I've never been pleased about it's design nor even it's existence.

That may cause system call failures to happen more often and in different contexts than they used to. But, naturally, that will not be a problem since all user-space code diligently checks the return status of every system call and responds with well-tested error-handling code when things go wrong.


And that's how it should have always been, by returning errors to user space, we let processes handle these errors gracefully.

void*buffer = malloc(20*1024);
if (!buffer) { this doesn't happen }

http://opsmonkey.blogspot.com/2007/01/linux-memory-overcommit.html

Of course this is a problem in the first place because linux was intentionally designed to overcommit memory. I think in practice the harms outweigh the benefits. Hopefully the changes here in 3.12 will make linux rock solid even under out of memory conditions with overcomit turned off (no more killing of innocent processes).

Edited 2013-11-08 05:05 UTC

Reply Score: 23

RE: OOM Killer
by Savior on Fri 8th Nov 2013 09:04 in reply to "OOM Killer"
Savior Member since:
2006-09-02

What will linux kill? How about a document, web browser, X11 session, etc.


According to my experience, it's almost always the last one (X11)... only seldom had I the luck of it being something harmless, like Firefox. If this update fixes this problem, I'll be very happy.

Reply Parent Score: 3

RE[2]: OOM Killer
by ajs124 on Fri 8th Nov 2013 13:30 in reply to "RE: OOM Killer"
ajs124 Member since:
2013-08-28

huh, it worked pretty reliably for me and killed firefox or its plugin container most of the time.

Reply Parent Score: 2

RE: OOM Killer
by Vanders on Fri 8th Nov 2013 09:33 in reply to "OOM Killer"
Vanders Member since:
2005-07-06

What will linux kill? How about a document, web browser, X11 session, etc.

From experience it's usually sshd, making it as hard as possible to actually get to the machine and clean it up.

Reply Parent Score: 3

RE: OOM Killer
by jessesmith on Fri 8th Nov 2013 16:04 in reply to "OOM Killer"
jessesmith Member since:
2010-03-11

This is excellent, I always considered the default behaviour dishonest. Telling a process it can have memory which isn't available and then killing a process (seemingly at random) always struck me as a terrible idea.

Reply Parent Score: 4

RE: OOM Killer
by Kebabbert on Mon 11th Nov 2013 20:19 in reply to "OOM Killer"
Kebabbert Member since:
2007-07-27

Thom, you missed what I'd consider to be one of the bigger highlights: changing the out of memory killer!

https://lwn.net/Articles/562211/
"As was described in this June 2013 article, the kernel's out-of-memory (OOM) killer has some inherent reliability problems. A process may have called deeply into the kernel by the time it encounters an OOM condition; when that happens, it is put on hold while the kernel tries to make some memory available. That process may be holding no end of locks, possibly including locks needed to enable a process hit by the OOM killer to exit and release its memory; that means that deadlocks are relatively likely once the system goes into an OOM state.



Linux has never been extremely stable in the past under out of memory conditions, I've always considered the out of memory killer to be major hack for a fundamental problem.
"
O_o

How many other design problems does Linux have that has not been fixed in the year 2013? This is RAM overcommit OOM thing is horrendous, and if you read the article, Linux still has the OOM killing thing going on, but less seldom. No wonder people say Linux is unstable...

Reply Parent Score: 2

RE[2]: OOM Killer
by Alfman on Tue 12th Nov 2013 06:31 in reply to "RE: OOM Killer"
Alfman Member since:
2011-01-28

Kebabbert,

"No wonder people say Linux is unstable..."

Ugh, there's a difference between criticizing and trolling, and you just crossed it ;)


"This is RAM overcommit OOM thing is horrendous, and if you read the article, Linux still has the OOM killing thing going on, but less seldom."

Well this is a fair question, and to the best of my knowledge, these changes are supposed to completely clean up the kernel's own handling of OOM conditions by allowing syscalls to return 'no memory' errors instead of blocking while the OOM killer runs.

Note this does not change user space over committing, that can already be disabled separately (sysctl vm.overcommit_memory=2). As long as syscalls were still triggering OOM killer inside the kernel due to syscalls which couldn't return 'no memory' errors, there was still a problem. Now that this is fixed, I would expect that the OOM killer will never be invoked.

Also, to really have a full grasp of the situation, we need to understand the background behind over-commit to begin with, and that has a lot to do with one particular function: fork. Unix multi-process based programs work by cloning a process into children. At the moment the child process is spawned, 100% of it's memory pages will be identical to the parent. In terms of fork, it makes sense to share the memory pages instead of copying them. Depending on what the child and parent actually do, they may or may not update these shared pages as they execute. If and when they do, the OS performs a 'copy-on-write' operation, allocating a new page to hold the modified page.

The OS can now use the pages that would have been used by child processes for other purposes. Using extra pages for caching is *good* since they can be dumped at any time should they be needed. Using them for spawning more processes or additional resources at run time is *bad* (IMHO) because now they cannot be recovered without taking back resources already in use elsewhere (aka OOM-killer).

Unfortunately, there are cases when overcommitment is unavoidable under fork semantics. Consider a large app (ie a database consuming 1GB ram) wanting to spawn a child process to perform a simple parallel task. This child process might only need <1MB of ram, but it's never-the-less sharing 999MB of memory with it's parent for as long as it executes. NOT overcommiting implies the two processes combined need 2GB instead of 1.001GB. And if 2GB is not available, well then the parent process will not be able to spawn any children for any reason.

So, overcommiting is good for fork, which is why linux does it. This highlights the real underlying problem, nearly everybody thinks overcommit is bad, but the truth is many are still huge proponents of fork().

Reply Parent Score: 2