I have a love-hate relationship with Sun’s NFS. Since it was so prevalent, it’s a go-to for getting stuff on and off the classic UNIX workstations I love to explore, but at the same time, it also never seems to work right away. However, the technology NFS was designed to replace was apparently quite a bit worse. Sun sold diskless workstations before NFS, which used something called nd (network disk). The problems with nd stem from a limitation of SunOS at the time. Since SunOS only provided support for a maximum of eight partitions per physical disk, nd offered the ability to create subpartitions, of which you had to manually create and remember the start and end sectors.
That’s a recipe for problems. But wait, there’s more!
For extra bonus problems, you might run out of available partitions to use on your server disk because you needed all of the available ones for regular filesystems and your swap area. If you were in this situation you could take the dangerous but necessary step of specifying your network disks using the special ‘c’ partition (cf dkinfo(8)), which was conventionally used to provide access to the entire disk. This was extra dangerous because you had to make sure that the nd disks you specified weren’t overlapping into any regular partitions that you were using, since as nd(8) says, nd itself did no sanity checking. If you said sectors X to Y were network disk X, that’s what they were, and goodness help you if some of them were also something else.
↫ Chris Siedenmann
And this isn’t even everything. Every part of this sounds horrid, and I can totally understand seeing NFS as a godsend compared to nd. It’s depressing that we’re in 2026 now, and the basic task of sending a file from one computer to another over your own network often still a total clusterfuck.
I think you mean NFS.
In my current setup at home I did use NFS for a while but LInux seemed to really not like when NFS shares were slow to respond. On the other hand, SMB/CIFS in userspace works exceptionally well, though I haven’t tried benchmarking it vs. kernel CIFS or NFS.
Drumhellar,
I concur. While in principal the kernel should be able to do everything the userspace driver can, in practice linux drivers are designed around synchronous ABIs rather than asynchronous ones, which means they’re not elegant at being interrupted mid-sequence. We’re left with kernel threads that cannot be canceled and all the kernel structures used by those thread are locked until the thread returns. This causes huge problems if the thread is stuck waiting on buggered network IO that isn’t coming. You end up with network mounts that cannot be unmounted, blocked processes that cannot be killed… The official mitigation was to add timeouts to network operations, which default to a few minutes. Although this works, it’s a hack that introduce notorious delays with network file systems on linux for decades. The same problem exists with faulty USB media locking up mount points and processes until the USB media is physically yanked out, which is bad practice because it’s meant to be unmounted first, but that is an impossibility when the device is locked by a stuck thread…oh well.
FUSE drivers don’t have this problem, you can kill the backing process and the kernel cleans up after it nicely.
Alfman,
I’d say we (the computing community) made a huge mistake around mid-1990s by switching to preemptive multi-tasking from cooperative ones.
But fortunately the better paradigm is making a comeback.
Aren’t these the old ones? Since late 2000s, Linux has async APIs in the kernel, which depend on queues, and are not blocking. I’m not sure whether all older drivers were updated, but moderns ones no longer have uninterruptible sleep.
And I seem to recall them celebrating removing that “one main kernel lock” (or whatever they called it) a while ago.
Yes, “D-state” and you cannot kill a process.
But is there any operating system that will let you kill a process that holds I/O buffers?
I mean what is the alternative?
sukru,
Not sure if you’re talking about different subsystems, networking primitives have select/poll/epoll/etc. But all the file system drivers in linux are still using the threaded/blocking APIs in kernel and there’s no consistent async API that works with all file descriptor types. Yet the POSIX API requires every FD to be asynchronous, consequently the POSIX AIO API on linux required a very awkward compromise…every AIO request on linux is actually spawns up a blocking userspace thread. The fake asynchronous API is implemented using signals from blocked threads. Granted they use thread pools, but this design irks me a lot because the whole point of AIO is to get rid of threads and their blocking issues, improving scalability and state management. But on linux it’s just become another layer of bloat that has all the disadvantages. Truly implementing POSIX AIO on linux would require all the synchronous code in linux to be rewritten.
Windows NT has nonblocking APIs (since the beginning?). Linux Torvalds was either unaware or didn’t care about AIO in the early days of linux and changing it now would require very significant rewrites.
One kernel lock is forgivable for a single core kernel, but as more SMP cores are added it balloons the overhead. The solution was create context specific locks, which probably should have been done in the first place but at least we got there.
I’m not sure how much work you’ve done with asynchronous programming, but it solves this problem entirely because every blocking point is in a consistent and well defined state. This means you can naturally add new state transitions (including “kill”) and there is no problem. This is very difficult to handle well with blocking threads.
It requires thinking about things a bit differently but I am a proponent of asynchronous code. It motivated me to create my own async library.. I really love C#’s asynchronous primitives and I miss having them in C/C++. Over time I’ve grown to favor compilers with high level features that not only boost productivity but also significantly lower the risk of human error. These features have merit in the kernel too, but I won’t be holding my breath for linux to get them, haha.
Alfman,
Okay, I’m going in the wrong order (see comment below)
Anyway…
The issue is more fundamental, and not affected by sync or async I/O.
1 – Say you have initiated a “zero copy” read from a file
2 – The filesystem driver maps this to a USB read
3 – The USB driver prepares a DMA operation targeting your buffer.
4 – The hardware gives “all okay”, DMA transfer completes
5 – The USB driver returns success, which is then picked up by filesystem
6 – Which returns success to your program
Now in step (2) the ownership of your buffer is transferred to the kernel, which cannot be returned until (6). There is also another dependency between filesytem and USB drivers which depends on (5), which depends on (4) to complete.
Say at point 3-4 transition, you issue
“kill -9 $PID”
The OS has the wait until (6) before it can actually issue that signal. It does not matter if this is sync (your entire program is waiting) or async (just that operation is waiting).
Hence the program is now in D-State
If there is a timeout or explicit hardware failure (yanking the USB plug) (4) ends with error, and everything is released
Otherwise more or less all operating systems will keep the broken “alive” to simplify bookkeeping. Killing the process will literally mess up internal kernel structures that depend on those buffers.
(I asked Gemini, it says Windows will remove the process from the list, but not from memory. Its resources will still be alive until that I/O operation ends)
sukru,
We might have to test a specific example to see what actually happens. However in principal virtual memory takes care of this transparently.
With normal/blocking file IO, the data is copied between kernel and userspace memory anyway. So the process can be killed immediately (assuming the kernel is designed to handle it) because the IO doesn’t take place in userspace memory.
With O_DIRECT file IO, it is mandatory for data to be page aligned so the kernel can use memory mapping instead of copying to/from kernel buffers. The process can be killed immediately whether or not the data has been read from or written to disk, the page will continue to exist in the page cache after the process is killed – obviously one less process will reference it.
With memory mapped files, a process can just be killed normally without blocking. Killing it merely removes the processes’ memory maps, but dirty pages are still in the system cache. These will get written normally even after the process dies.
The sendfile, splice, io_submit APIs avoid userspace memory altogether, a program using these can be terminated immediately. Obviously the implementation must decide whether or not to finish the IO but regardless the process can be killed immediately.
I understand how it works in linux, but these are OS specific quirks and not blueprints of how things must work in general.
It’s not that processes are inherently unkillable though. I would not have implemented things this way, but these limitations got cemented deep into linux foundations from the start.
I’m reading a paper now that says windows terminates immediately. This is a lie mind you, but I’m trying to make a point that alluding to information doesn’t carry the same weight as providing that information. I shouldn’t be expected to agree with something without knowing the question asked or the answer given.
Just to be clear though, I’m not asserting that windows or linux actually do terminate in the middle of IO only that it’s technically solvable with an asynchronous design. I don’t know that windows file system drivers are fully asynchronous under the hood.
Alfman,
Async or regular, the kernel will have to have a page (or set of pages) associated with the process, temporarily owned by the driver to keep it alive.
The simplest way of doing this is using the process.
And this is the part where can of worms come in
In order a page to be pinned, an operating system will have to associate it with something. The easiest way of doing this is using the process, which already have the associated structures.
And it is more than just the destination buffer for I/O, there are callback, cleanups, and other housekeeping associated… again with the process in practice.
Yes, but this is also the easiest and safest way to handle “D-state”
Otherwise Linux and other kernels will have to invent another structure that looks exactly like a process (except being able to run)
Again, is it much easier, and safer, just to use the process, but mark is as dead. Think of it like a database making a row deleted.
This seems to be the same on Windows (from https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-terminateprocess?source=recommendations)
sukru,
I didn’t mean to drop this. Can you elaborate? I’m curious what examples you have.
Alfman,
The modern paradigm is “async”, or the traditional “co-routines” (or if you look at implementation details “fibers”)
For a long running job that is not CPU bound, you’d run what you can and yield back to the scheduler for the next task that is waiting.
In the past, this was what Windows and other GUI operating systems did. But did not work well in practice across applications, since one badly written one could stall the entire system.
(This is different than CPU bound tasks that need to run non-stop until completion)
Now… almost all modern languages do this internally in applications. It gives us significantly better performance for I/O bound tasks. Or things that need micro/nano-seconds to run.
The desktop is still pre-emptive, though. But truly performance critical tasks like servers or real time audio processors just “pin” CPU cores.
Hence, concurrency and parallelism is properly separated.
sukru,
Ok, you’re talking about co-routines at the application level, which I agree are a form of asynchronous programming. Although I don’t immediately associate cooperative multitasking with co-routines. Cooperative multitasking was something that applied across the operating system whereas co-routines are typically within a single process (are there any counter examples to this?).
I also think of co-routines as being stackless state machines, a stack only exists for transient calculations such that coroutines can share the same stack without stepping on each other. This is the type of async that I am a proponent of. However it’s true that some languages implement coroutines using ordinary stacks, making them nearly identical to regular multi-threading except for sharing the thread – it’s more of a hybrid.
https://langdev.stackexchange.com/questions/697/what-are-the-benefits-of-stackful-vs-stackless-coroutines
Alfman,
Async is to Threads in the same process, whereas “Event Loop” is to Processes in the operating system level.
It is the same division of concepts.
One side is co-operative (async and event loops)
One side is pre-emptive (threads and modern schedulers)
And this makes no sense, except some syntactic sugar.
To be complte:
Async co-routines actually need kernel support
Both Linux and Windows has a concept of “Fibers” (Linux came from Google’s implementation, UMCG). Without those the performance benefits are not there.
A Thread is a group of Fibers woven together.
That concept is very beautiful if you think about it.
sukru,
I wouldn’t get too stuck on specific terminology since different connotations can be a barrier to shared understanding. I like having more context to keep everyone one the same page,
What doesn’t make sense? That fiber and coroutine implementations can have their own stacks?
https://github.com/vzvca/libfiber
The obvious reason some languages & libraries go this route is because they can use a plain old compiler to write standard multithreaded code and simply change the way threads are scheduled. A more sophisticated compiler can do away with the stack saving the context in a state machine, but it’s harder to implement because conventional compilers don’t do it.
That’s not true though, coroutines are NOT a function of the kernel, any kernel that actually has non-blocking syscalls will do.
Google UMCG is one userspace implementation and created syscalls to replace futex.
https://nanovms.com/dev/tutorials/user-mode-concurrency-groups-faster-than-futexes
Google threw their hat in the ring, but remember there were tons of asynchronous implementations, some purely asynchronous and wouldn’t even benefit from google’s patch. I can concede that older unix APIs were very inefficient and scaled poorly. “Select” needed to be called very frequently and built a list of descriptors for each invocation. This suffered from tons of overhead both in userspace and kernel space. However these have long since been replaced by new mechanisms like epoll, which largely solves the problem.
https://www.man7.org/linux/man-pages/man2/select.2.html
https://www.man7.org/linux/man-pages/man7/epoll.7.html
The new syscalls are extremely efficient today and I use them for asynchronous real time programming all the time. Multiple threads and stacks became optional in this programming model (of course depending on implementation). I do accept that coroutines are nice syntactic sugar to make asynchronous programming more friendly with modern programming languages. However asynchronous programming has been around long before before me, google, and probably you too. Google’s implementation is just one of many and not that special.
Alfman,
I might have exaggerated a little, but the bulk of the benefits from async comes from having as little state as possible. An entire stack is heavy.
C#, C++, Swift, Rust, … all abstract this in their compilers. Thanks to LLVM, rewriting an iterative style function into async coroutines by splicing at async/await/poll/value boundaries is “almost” trivial (for compiler writers)
No need to get stuck on specific implementations. Google’s was pioneer, and even if there are newer ones, it does not change that fiber implementation is entirely capable of serving global scale traffic (which is hard to do without proper kernel support)
“This is the way”
Yes, the future is async. Except for trivial programs, anything that requires efficient resource use, and high performance would do what you do.
sukru,
I agree, getting rid of the stack can be beneficial. A stack per async operation or fiber is expensive. However I just wanted to say that there is plenty of variation in async implementations and some languages like go & java still allocate dedicated stacks according to the link I sent about this earlier.
Google implemented their own implementation with UMGC of course, but the techniques they used were not new or unique.
Sure I’ll agree with this. I’ve been advocating for async programming for so many years, long before languages had things like coroutines. It’s only more recently that modern languages embraced the syntactic sugar to make asynchronous programming more palatable than coding async routines by hand.
Oh yea – sometimes my wifi card goes stale (no IO for like 30-60 seconds, once or twice per day). It happens eventually that I turn my laptop off during that period, and then I’m stuck waiting for remote mounts to unmount.
Ugh. Rage.
Was this NFS v3 or v4? I’ve found that the updates they made in v4 are a godsend. Since it’s TCP-based instead of UDP-based and uses a single port, it’s much more resilient because it can definitively know when the remote end has disconnected rather than waiting for timeouts and retries.
That is fairly typical behavior when NFS falls back to v3, which is the de facto baseline when the client and server are not configured correctly.
NFSv3 is ancient and carries a lot of historical baggage: stateless operation, awkward permission handling, and external locking through separate services. Those limitations are a common source of hangs and stale locks, especially on noisy/unreliable networks like WiFi.
SMB/CIFS is generally a much better designed remote file protocol than NFSv3 and deliver more robust and predictable behavior.
In the eighties, with nd we mapped raw blocks over physical networks. A typo in sector offsets could corrupt the filesystem of another user …
NFS moved from blocks to files using RPCs. VFS translates local system calls into remote ones. Still, I spent long hours trying to map OS permissions, stateful locking semantics, and caching.
Here is breakdown of SMB vs NFS https://visualitynq.com/resources/smb-vs-nfs/ explaining how the two protocols solved those constraints.