Linked by diegocg on Thu 7th Nov 2013 22:19 UTC
Linux Linux kernel 3.12 has been released. This release includes support for offline deduplication in Btrfs, automatic GPU switching in laptops with dual GPUs, a performance boost for AMD Radeon graphics, better RAID-5 multicore performance, improved handling of out-of-memory situations, improvements to the timerless multitasking mode, separate modesetting and rendering device nodes in the graphics DRM layer, improved locking performance for virtualized guests, XFS directory recursion scalability improvements, new drivers and many small improvements. Here's the full list of changes.
Thread beginning with comment 576630
To view parent comment, click here.
To read all comments associated with this story, please click here.
RE: OOM Killer
by Kebabbert on Mon 11th Nov 2013 20:19 UTC in reply to "OOM Killer"
Kebabbert
Member since:
2007-07-27

Thom, you missed what I'd consider to be one of the bigger highlights: changing the out of memory killer!

https://lwn.net/Articles/562211/
"As was described in this June 2013 article, the kernel's out-of-memory (OOM) killer has some inherent reliability problems. A process may have called deeply into the kernel by the time it encounters an OOM condition; when that happens, it is put on hold while the kernel tries to make some memory available. That process may be holding no end of locks, possibly including locks needed to enable a process hit by the OOM killer to exit and release its memory; that means that deadlocks are relatively likely once the system goes into an OOM state.



Linux has never been extremely stable in the past under out of memory conditions, I've always considered the out of memory killer to be major hack for a fundamental problem.
"
O_o

How many other design problems does Linux have that has not been fixed in the year 2013? This is RAM overcommit OOM thing is horrendous, and if you read the article, Linux still has the OOM killing thing going on, but less seldom. No wonder people say Linux is unstable...

Reply Parent Score: 2

RE[2]: OOM Killer
by Alfman on Tue 12th Nov 2013 06:31 in reply to "RE: OOM Killer"
Alfman Member since:
2011-01-28

Kebabbert,

"No wonder people say Linux is unstable..."

Ugh, there's a difference between criticizing and trolling, and you just crossed it ;)


"This is RAM overcommit OOM thing is horrendous, and if you read the article, Linux still has the OOM killing thing going on, but less seldom."

Well this is a fair question, and to the best of my knowledge, these changes are supposed to completely clean up the kernel's own handling of OOM conditions by allowing syscalls to return 'no memory' errors instead of blocking while the OOM killer runs.

Note this does not change user space over committing, that can already be disabled separately (sysctl vm.overcommit_memory=2). As long as syscalls were still triggering OOM killer inside the kernel due to syscalls which couldn't return 'no memory' errors, there was still a problem. Now that this is fixed, I would expect that the OOM killer will never be invoked.

Also, to really have a full grasp of the situation, we need to understand the background behind over-commit to begin with, and that has a lot to do with one particular function: fork. Unix multi-process based programs work by cloning a process into children. At the moment the child process is spawned, 100% of it's memory pages will be identical to the parent. In terms of fork, it makes sense to share the memory pages instead of copying them. Depending on what the child and parent actually do, they may or may not update these shared pages as they execute. If and when they do, the OS performs a 'copy-on-write' operation, allocating a new page to hold the modified page.

The OS can now use the pages that would have been used by child processes for other purposes. Using extra pages for caching is *good* since they can be dumped at any time should they be needed. Using them for spawning more processes or additional resources at run time is *bad* (IMHO) because now they cannot be recovered without taking back resources already in use elsewhere (aka OOM-killer).

Unfortunately, there are cases when overcommitment is unavoidable under fork semantics. Consider a large app (ie a database consuming 1GB ram) wanting to spawn a child process to perform a simple parallel task. This child process might only need <1MB of ram, but it's never-the-less sharing 999MB of memory with it's parent for as long as it executes. NOT overcommiting implies the two processes combined need 2GB instead of 1.001GB. And if 2GB is not available, well then the parent process will not be able to spawn any children for any reason.

So, overcommiting is good for fork, which is why linux does it. This highlights the real underlying problem, nearly everybody thinks overcommit is bad, but the truth is many are still huge proponents of fork().

Reply Parent Score: 2

RE[3]: OOM Killer
by Alfman on Tue 12th Nov 2013 19:08 in reply to "RE[2]: OOM Killer"
Alfman Member since:
2011-01-28

To the extent that fork still gets use and it's cons aren't going away, I'd be very interested in seeing a simple compromise by making over-commit *explicit*.

Such as: no application will ever be overcommitment unless it explicitly requests it or it's process is configured for it by the administrator. By default, when a process requests memory from the kernel, the kernel will keep that commitment (anything else is bad practice). But in cases where overcommitment is still desired or needed, it can be done explicitly.

So, for example, when a 1GB database tries to call fork, it could tell the kernel that overcommiting the 1GB child process would be preferable to being denied with a 'no memory' error by default. If the overcommitment turns into an OOM condition, then only those processes who elected over-commit would be killed, which seems pretty fair.

This could be implemented pretty easily in linux by adding a new flag to the existing clone syscall. Anyone else have thoughts about this?

Reply Parent Score: 2