I have a love-hate relationship with Sun’s NFS. Since it was so prevalent, it’s a go-to for getting stuff on and off the classic UNIX workstations I love to explore, but at the same time, it also never seems to work right away. However, the technology NFS was designed to replace was apparently quite a bit worse. Sun sold diskless workstations before NFS, which used something called nd (network disk). The problems with nd stem from a limitation of SunOS at the time. Since SunOS only provided support for a maximum of eight partitions per physical disk, nd offered the ability to create subpartitions, of which you had to manually create and remember the start and end sectors.
That’s a recipe for problems. But wait, there’s more!
For extra bonus problems, you might run out of available partitions to use on your server disk because you needed all of the available ones for regular filesystems and your swap area. If you were in this situation you could take the dangerous but necessary step of specifying your network disks using the special ‘c’ partition (cf dkinfo(8)), which was conventionally used to provide access to the entire disk. This was extra dangerous because you had to make sure that the nd disks you specified weren’t overlapping into any regular partitions that you were using, since as nd(8) says, nd itself did no sanity checking. If you said sectors X to Y were network disk X, that’s what they were, and goodness help you if some of them were also something else.
↫ Chris Siedenmann
And this isn’t even everything. Every part of this sounds horrid, and I can totally understand seeing NFS as a godsend compared to nd. It’s depressing that we’re in 2026 now, and the basic task of sending a file from one computer to another over your own network often still a total clusterfuck.
I think you mean NFS.
In my current setup at home I did use NFS for a while but LInux seemed to really not like when NFS shares were slow to respond. On the other hand, SMB/CIFS in userspace works exceptionally well, though I haven’t tried benchmarking it vs. kernel CIFS or NFS.
Drumhellar,
I concur. While in principal the kernel should be able to do everything the userspace driver can, in practice linux drivers are designed around synchronous ABIs rather than asynchronous ones, which means they’re not elegant at being interrupted mid-sequence. We’re left with kernel threads that cannot be canceled and all the kernel structures used by those thread are locked until the thread returns. This causes huge problems if the thread is stuck waiting on buggered network IO that isn’t coming. You end up with network mounts that cannot be unmounted, blocked processes that cannot be killed… The official mitigation was to add timeouts to network operations, which default to a few minutes. Although this works, it’s a hack that introduce notorious delays with network file systems on linux for decades. The same problem exists with faulty USB media locking up mount points and processes until the USB media is physically yanked out, which is bad practice because it’s meant to be unmounted first, but that is an impossibility when the device is locked by a stuck thread…oh well.
FUSE drivers don’t have this problem, you can kill the backing process and the kernel cleans up after it nicely.
Alfman,
I’d say we (the computing community) made a huge mistake around mid-1990s by switching to preemptive multi-tasking from cooperative ones.
But fortunately the better paradigm is making a comeback.
Aren’t these the old ones? Since late 2000s, Linux has async APIs in the kernel, which depend on queues, and are not blocking. I’m not sure whether all older drivers were updated, but moderns ones no longer have uninterruptible sleep.
And I seem to recall them celebrating removing that “one main kernel lock” (or whatever they called it) a while ago.
Yes, “D-state” and you cannot kill a process.
But is there any operating system that will let you kill a process that holds I/O buffers?
I mean what is the alternative?
sukru,
Not sure if you’re talking about different subsystems, networking primitives have select/poll/epoll/etc. But all the file system drivers in linux are still using the threaded/blocking APIs in kernel and there’s no consistent async API that works with all file descriptor types. Yet the POSIX API requires every FD to be asynchronous, consequently the POSIX AIO API on linux required a very awkward compromise…every AIO request on linux is actually spawns up a blocking userspace thread. The fake asynchronous API is implemented using signals from blocked threads. Granted they use thread pools, but this design irks me a lot because the whole point of AIO is to get rid of threads and their blocking issues, improving scalability and state management. But on linux it’s just become another layer of bloat that has all the disadvantages. Truly implementing POSIX AIO on linux would require all the synchronous code in linux to be rewritten.
Windows NT has nonblocking APIs (since the beginning?). Linux Torvalds was either unaware or didn’t care about AIO in the early days of linux and changing it now would require very significant rewrites.
One kernel lock is forgivable for a single core kernel, but as more SMP cores are added it balloons the overhead. The solution was create context specific locks, which probably should have been done in the first place but at least we got there.
I’m not sure how much work you’ve done with asynchronous programming, but it solves this problem entirely because every blocking point is in a consistent and well defined state. This means you can naturally add new state transitions (including “kill”) and there is no problem. This is very difficult to handle well with blocking threads.
It requires thinking about things a bit differently but I am a proponent of asynchronous code. It motivated me to create my own async library.. I really love C#’s asynchronous primitives and I miss having them in C/C++. Over time I’ve grown to favor compilers with high level features that not only boost productivity but also significantly lower the risk of human error. These features have merit in the kernel too, but I won’t be holding my breath for linux to get them, haha.
Alfman,
Okay, I’m going in the wrong order (see comment below)
Anyway…
The issue is more fundamental, and not affected by sync or async I/O.
1 – Say you have initiated a “zero copy” read from a file
2 – The filesystem driver maps this to a USB read
3 – The USB driver prepares a DMA operation targeting your buffer.
4 – The hardware gives “all okay”, DMA transfer completes
5 – The USB driver returns success, which is then picked up by filesystem
6 – Which returns success to your program
Now in step (2) the ownership of your buffer is transferred to the kernel, which cannot be returned until (6). There is also another dependency between filesytem and USB drivers which depends on (5), which depends on (4) to complete.
Say at point 3-4 transition, you issue
“kill -9 $PID”
The OS has the wait until (6) before it can actually issue that signal. It does not matter if this is sync (your entire program is waiting) or async (just that operation is waiting).
Hence the program is now in D-State
If there is a timeout or explicit hardware failure (yanking the USB plug) (4) ends with error, and everything is released
Otherwise more or less all operating systems will keep the broken “alive” to simplify bookkeeping. Killing the process will literally mess up internal kernel structures that depend on those buffers.
(I asked Gemini, it says Windows will remove the process from the list, but not from memory. Its resources will still be alive until that I/O operation ends)
sukru,
We might have to test a specific example to see what actually happens. However in principal virtual memory takes care of this transparently.
With normal/blocking file IO, the data is copied between kernel and userspace memory anyway. So the process can be killed immediately (assuming the kernel is designed to handle it) because the IO doesn’t take place in userspace memory.
With O_DIRECT file IO, it is mandatory for data to be page aligned so the kernel can use memory mapping instead of copying to/from kernel buffers. The process can be killed immediately whether or not the data has been read from or written to disk, the page will continue to exist in the page cache after the process is killed – obviously one less process will reference it.
With memory mapped files, a process can just be killed normally without blocking. Killing it merely removes the processes’ memory maps, but dirty pages are still in the system cache. These will get written normally even after the process dies.
The sendfile, splice, io_submit APIs avoid userspace memory altogether, a program using these can be terminated immediately. Obviously the implementation must decide whether or not to finish the IO but regardless the process can be killed immediately.
I understand how it works in linux, but these are OS specific quirks and not blueprints of how things must work in general.
It’s not that processes are inherently unkillable though. I would not have implemented things this way, but these limitations got cemented deep into linux foundations from the start.
I’m reading a paper now that says windows terminates immediately. This is a lie mind you, but I’m trying to make a point that alluding to information doesn’t carry the same weight as providing that information. I shouldn’t be expected to agree with something without knowing the question asked or the answer given.
Just to be clear though, I’m not asserting that windows or linux actually do terminate in the middle of IO only that it’s technically solvable with an asynchronous design. I don’t know that windows file system drivers are fully asynchronous under the hood.
sukru,
I didn’t mean to drop this. Can you elaborate? I’m curious what examples you have.
Alfman,
The modern paradigm is “async”, or the traditional “co-routines” (or if you look at implementation details “fibers”)
For a long running job that is not CPU bound, you’d run what you can and yield back to the scheduler for the next task that is waiting.
In the past, this was what Windows and other GUI operating systems did. But did not work well in practice across applications, since one badly written one could stall the entire system.
(This is different than CPU bound tasks that need to run non-stop until completion)
Now… almost all modern languages do this internally in applications. It gives us significantly better performance for I/O bound tasks. Or things that need micro/nano-seconds to run.
The desktop is still pre-emptive, though. But truly performance critical tasks like servers or real time audio processors just “pin” CPU cores.
Hence, concurrency and parallelism is properly separated.
sukru,
Ok, you’re talking about co-routines at the application level, which I agree are a form of asynchronous programming. Although I don’t immediately associate cooperative multitasking with co-routines. Cooperative multitasking was something that applied across the operating system whereas co-routines are typically within a single process (are there any counter examples to this?).
I also think of co-routines as being stackless state machines, a stack only exists for transient calculations such that coroutines can share the same stack without stepping on each other. This is the type of async that I am a proponent of. However it’s true that some languages implement coroutines using ordinary stacks, making them nearly identical to regular multi-threading except for sharing the thread – it’s more of a hybrid.
https://langdev.stackexchange.com/questions/697/what-are-the-benefits-of-stackful-vs-stackless-coroutines
Oh yea – sometimes my wifi card goes stale (no IO for like 30-60 seconds, once or twice per day). It happens eventually that I turn my laptop off during that period, and then I’m stuck waiting for remote mounts to unmount.
Ugh. Rage.
Was this NFS v3 or v4? I’ve found that the updates they made in v4 are a godsend. Since it’s TCP-based instead of UDP-based and uses a single port, it’s much more resilient because it can definitively know when the remote end has disconnected rather than waiting for timeouts and retries.
That is fairly typical behavior when NFS falls back to v3, which is the de facto baseline when the client and server are not configured correctly.
NFSv3 is ancient and carries a lot of historical baggage: stateless operation, awkward permission handling, and external locking through separate services. Those limitations are a common source of hangs and stale locks, especially on noisy/unreliable networks like WiFi.
SMB/CIFS is generally a much better designed remote file protocol than NFSv3 and deliver more robust and predictable behavior.
In the eighties, with nd we mapped raw blocks over physical networks. A typo in sector offsets could corrupt the filesystem of another user …
NFS moved from blocks to files using RPCs. VFS translates local system calls into remote ones. Still, I spent long hours trying to map OS permissions, stateful locking semantics, and caching.
Here is breakdown of SMB vs NFS https://visualitynq.com/resources/smb-vs-nfs/ explaining how the two protocols solved those constraints.