Unix Archive

I’ve been working on developing an operating system for the TI-99 for the last 18 months or so. I didn’t intend this—my original plan was to develop enough of the standard C libraries to help with writing cartridge-based and EA5 programs. But that trek led me quickly towards developing an OS. As Unix is by far my preferred OS, this OS is an approximation. Developing an OS within the resources available, particularly the RAM, has been challenging, but also surprisingly doable. ↫ UNIX99 forum announcement post We’re looking at a quite capable UNIX for the TI-99, with support for its sound, speech, sprites, and legacy 9918A display modes, GPU-accelerated scrolling, stdio (for text and binary files) and stdin/out/err support, a shell (of course), multiple user support, cooperative tasks support, and a ton more. And remember – all of this is running on a machine with a 16-bit processor running at 16MHz and a mere 16KB of RAM. Absolutely wild.

It may be arcane knowledge to most users of UNIX-like systems today, but there is supposed to be a difference between /usr/bin and /usr/sbin; the latter is supposed to be for “system binaries”, not needed by most normal users. The Filesystem Hierarchy Standard states that sbin directories are intended to contain “utilities used for system administration (and other root-only commands)”, which is quite vague when you think about it. This has led to UNIX-like systems basically just winging it, making the distinction almost entirely arbitrary. For a long time, there has been no strong organizing principle to /usr/sbin that would draw a hard line and create a situation where people could safely leave it out of their $PATH. We could have had a principle of, for example, “programs that don’t work unless run by root”, but no such principle was ever followed for very long (if at all). Instead programs were more or less shoved in /usr/sbin if developers thought they were relatively unlikely to be used by normal people. But ‘relatively unlikely’ is not ‘never’, and shortly after people got told to ‘run traceroute’ and got ‘command not found’ when they tried, /usr/sbin (probably) started appearing in $PATH. ↫ Chris Siebenmann As such, Fedora 42 unifies /usr/bin and /usr/sbin, which is kind of a follow-up to the /usr merge, and serves as a further simplification and clean-up of the file system layout by removing divisions and directories that used to make sense, but no longer really do. Decisions like these have a tendency to upset a small but very vocal group of people, people who often do not even use the distribution implementing the decisions in question in the first place. My suggestions to those people would be to stick to distributions that more closely resemble classic UNIX. Or use a real UNIX. Anyway, these are good moves, and I’m glad most prominent Linux distributions are not married to decisions made in the ’70s, especially not when they can be undone without users really noticing anything.

The most unlikely subsystem of contention is definitely the init system used by Linux, with most popular distributions opting for systemd, while a vocal minority prefers to use something else. Neither of these two groups are wrong or right, as we live in a free world and different people have different needs and desires. Personally, I don’t think there’s a more utterly pointless and meaningless debate than this, and people who make the init system they use their entire personality more often than not come across as really, really sad. It’s a tool; use the one you like and move on with life. A brand new init system was recently released by Leah Neukirchen, who among a ton of other things, contributes to Void Linux. It’s called nitro, and it’s a “tiny process supervisor that also can be used as pid 1 on Linux”, and it also can be used on FreeBSD supervised by FreeBSD’s init. There’s some overlap with runit here, so Neukirchen published a blog post detailing the differences between the two, which should help in getting a better understanding of what makes nitro stand apart. While both use a directory of services managed by small scripts, nitro seems to opt for a more contained, monolithic approach, as it keeps everything in a single process. On top of that, Nitro contains some new features runit doesn’t have. The focus seems to be on integrating a few capabilities that on runit require hacks, but on nitro are just built-in, like “support for one-shot ‘services’, i.e. running scripts on up/down without a process to supervise (e.g. persist audio volume, keep RNG state)”, running service directories multiple times, and more. Nitro also maintains its runtime state in RAM and provides an IPC service to query it, meaning it can be run on read-only filesystems without special configuration. There’s a lot more information in Neukirchen’s blog post, including a look at some of the current limitations of Nitro. I highly suggest reading it, and perhaps we will see nitro as another valid alternative to the popular systemd.

The first prototype was ready in just six months. By October 1986, the project was announced, and in January 1987, the first NEWS workstation, the NWS 800 series, officially launched. It ran 4.2BSD UNIX and featured a Motorola 68020 CPU. Its performance rivaled that of traditional super minicomputers, but with a dramatically lower price point ranging from ¥950,000 to ¥2.75 million (approximately $6,555 to $18,975 USD in 1987). Competing UNIX workstations typically cost closer to ¥10 million (around $69,000 USD). NEWS caught on quickly in universities and R&D labs, where cost sensitive researchers needed real performance. The venture team had invested ¥400 million into development (about $2.76 million USD), and remarkably, they recouped those costs within just two months of launch. That same year, Sony introduced a lower cost version called POP NEWS (PWS 1550). With a GUI shell named NEWS Desk, a document sharing format called CDFF (Common Document File Format), and a focus on Japanese language desktop publishing, PopNEWS aimed to make UNIX more accessible to general business users. Targeted at the Desktop Publishing market, it showed Sony’s desire to bridge consumer and professional segments in ways no other UNIX vendor was trying at the time. ↫ Obsolete Sony’s Newsletter I’ve been fascinated by Sony’s NEWS workstations, and especially the NEWS-OS operating system, for a long time now. Real hardware is hard to find and prohibitively expensive, but some of these Sony NEWS workstations can be emulated through MAME. Sadly, as far as I can tell, you can only emulate NEWS-OS up to version 4.x, as I haven’t been able to find any information about emulating version 5.x and the final version, 6.x. If anyone knows anything about how to emulate these, if at all possible, please do share with the rest of us. What’s interesting about Sony’s UNIX workstation efforts from the ’80s and ’90s is that they played an important role in the early development of the PlayStation. The early development kits for the PlayStation were modified NEWS workstations, with added PlayStation hardware. To further add to the importance of the NEWS line for gaming, Nintendo used them to develop several influential and popular first-party SNES titles, which isn’t surprising considering Nintendo and Sony originally worked together on bringing a CD-ROM drive to the SNES, which would later morph into the PlayStation as Nintendo cancelled the agreement at the last second.

There’s a number of backups of old DECtapes from Dennis Ritchie, which he gave to Warren Toomey in 1997. The tapes were eventually uploaded, and through analysis performed by Yufeng Gao, a lot of additional details, code, and software were recovered from them. A few days ago, Gao came back with the results from their analys of two more tapes, and on it, they found something quite special. Getting this recovered version to run was a bit of a challenge, and only aap’s PDP-11 emulator is capable of running it. To even get it to run in the first place, Gao had perform quite some intricate steps, but eventually he managed to build an image that can be downloaded and booted on aap’s PDP-11 emulator. The image in question, as well as some more details, can be found on the GitHub page.

What might be somewhat more surprising though considering its research origins is that Unix almost since the very beginning had a comprehensive set of online reference documentation for all its commands, system calls, file formats, etc. These are the the manual- or man-pages. On Unix systems used interactively, the man-pages have historically always been installed, space permitting. The way the manual pages have evolved and how they are used has changed over the decades. This set of posts is intended to give people unfamiliar with them an overview, as well as offer a review to seasoned users. ↫ Alex Bochannek Right in this first article in the series there’s an interesting observation I never stopped and thought about: because the original creators of UNIX were writing the content of man pages with the very tools they were creating for UNIX, it led to a virtuous cycle. “Unix tools were used to document Unix, improving the documentation tools themselves as well.” I tend to use the internet now to learn how specific tools and commands work, but having such detailed man pages built right into the operating system was a huge deal pre-internet.

Update: there’s a fork called heirloom-ng that is actually still somewhat maintained and contains some more changes and modernisations compared to the old version. The Heirloom Project provides traditional implementations of standard Unix utilities. In many cases, they have been derived from original Unix material released as Open Source by Caldera and Sun. Interfaces follow traditional practice; they remain generally compatible with System V, although extensions that have become common use over the course of time are sometimes provided. Most utilities are also included in a variant that aims at POSIX conformance. On the interior, technologies for the twenty-first century such as the UTF-8 character encoding or OpenType fonts are supported. ↫ The Heirloom Project website I had never heard of this before, but I like the approach they’re taking. This isn’t just taking System V tools and making them work on a modern UNIX-like system as-is; they’re also improving by them adding support for modern technologies, without actually changing their classic nature and the way old-fashioned users expect them to work. Sadly, the project seems to be dead, as the code hasn’t been altered since 2008. Perhaps someone new is willing to take up this project? As it currently stands, the tools are available for Linux, Solaris, Open UNIX, HP-UX, AIX, FreeBSD, NetBSD, and OpenBSD, but considering how long the code hasn’t been touched, I wonder if they still run and work on any of those systems today. They also come in various different versions which comply with different variants of the POSIX standard.

As an online discussion grows longer, the probability of a someone mentioning macOS is a UNIX approaches 1. In fact, it was only late last year that The Open Group announced that macOS 15.0 was, once again, certified as UNIX, continuing Apple’s long-standing tradition of certifying macOS releases as “real” UNIX®. What does any of this actually, mean, though? Well, it turns out that if you actually dive into Apple’s conformance statements for macOS’ UNIX certification, it doesn’t really mean anything at all. First and foremost, we have to understand what UNIX certification really means. In order to be allowed to use the UNIX trademark, your operating system needs to comply with the Single UNIX Specification (SUS), which specifies programming interfaces for C, a command-line shell, and user commands, more or less identical to POSIX, as well as the X/Open Curses specification. The latest version is SUS version 4, originally published in 2008, with amendments published in 2013 and 2016, which were rolled up into version 4 in 2018. The various versions of the SUS that exist, in turn, correspond to a specific UNIX trademark. In table form: Trademark SUS version SUS published in: SUS last amended in: UNIX® 93 n.a. n.a. n.a. UNIX® 95 Version 1 1994 n.a. UNIX® 98 Version 2 1997 n.a. UNIX® 03 Version 3 2002 2004 UNIX® V7 Version 4 2008 2016 (2018 for roll-up) When you read that macOS is a certified UNIX, which of these versions and trademarks do you assume macOS complies with? You’d assume they would just target the latest trademark and SUS version, right? This would allow macOS to carry the UNIX® V7 trademark, because they would conform to version 4 of the SUS, which dates to 2016. The real answer is that macOS 15.0 only conforms to version 3 of the SUS, which dates all the way back to the ancient times of 2004, and as such, macOS is only UNIX® 03 (on both Intel and ARM). However, you can argue this is just semantics, since it’s not like UNIX and POSIX are very inclined to change. So now, like the UNIX nerd that you are, you want to see all this for yourself. You use macOS, safe in the knowledge that unlike those peasants using Linux or one of the BSDs, you’re using a real UNIX®. So you can just download all the tests suites (if you can afford them, but that’s a whole different can of worms) and run them, replicating Apple’s compliance testing, seeing for yourself, on your own macOS 15 installation, that macOS 15 is a real UNIX®, right? Well, no, you can’t, because the version of macOS 15 Apple certifies is not the version that’s running on everyone’s supported Macs. To gain its much-vaunted UNIX certification for macOS, Apple cheats. A lot. The various documents Apple needs to submit to The Open Group as part of the UNIX certification process are freely available, and mostly it’s a lot of very technical questions about various very specific aspects of macOS’ UNIX and POSIX compliance few of us would be able to corroborate without extensive research and in-depth knowledge of macOS, UNIX, and POSIX. However, at the end of every one of these Conformance Statements, there’s a text field where the applicant can write down “additional, explanatory material that was provided by the vendor”, and it’s in these appendices where we can see just how much Apple has to cheat to ensure macOS passes the various UNIX® 03 certification tests. In the first of these four documents, Internationalised System Calls and Libraries Extended V3, Apple’s “additional, explanatory material” reads as follows: Question 27: By default, core file generation is not enabled. To enable core file generation, you can issue this command: sudo launchctl limit core unlimited Testing Environment Addendum: macOS version 15.0 Sequoia, like previous versions, includes an additional security mechanism known as System Integrity Protection (SIP). This security policy applies to every running process, including privileged code and code that runs out of the sandbox. The policy extends additional protections to components on disk and at run-time, only allowing system binaries to be modified by the system installer and software updates. Code injection and runtime attachments to system binaries are no longer permitted. To run the VSX conformance test suite we first disable SIP as follows: – Shut down the system.– Press and hold the power button. Keep holding it while you see the Apple logo and the message “Continue holding for startup options”– Release the power button when you see “Loading startup options”– Choose “Options” and click “Continue”– Select an administrator account and enter its password.– From the Utilities menu in the Menu Bar, select Terminal.– At the prompt, issue the following command: “csrutil disable”– You should see a message that SIP is disabled. From the Apple menu, select “Restart”. By default, macOS coalesces timeouts that are scheduled to occur within 5 seconds of each other. This can randomly cause some sleep calls to sleep for different times than requested (which affects tests of file access times) so we disable this coalescing when testing. To disable timeout coalescing issue this command: sudo sysctl -w kern.timer.coalescing_enabled=0 By default there is no root user. We enable the root user for testing using the following series of steps:– Launch the Directory Utility by pressing Command and Space, and then typing “Directory Utility”– Click the Lock icon in Directory Utility and authenticate by entering an Administrator username and password.– From the Menu Bar in Directory Utility:– Choose Edit -> Enable Root User. Then enter a password for the root user, and confirm it.– Note: If you choose, you can later Disable Root User via the same menu. ↫ Apple’s appendix to Internationalised System Calls and Libraries Extended V3 The second conformance statement, Commands and Utilities V4, has another appendix, and it’s a real doozy (the indicate repeat remarks from the previous appendix; I’ve removed them for brevity): Testing Environment Addendum: The third and fourth conformance statements have

One of the innovations that the V7 Bourne shell introduced was built in shell wildcard globbing, which is to say expanding things like *, ?, and so on. Of course Unix had shell wildcards well before V7, but in V6 and earlier, the shell didn’t implement globbing itself; instead this was delegated to an external program, /etc/glob (this affects things like looking into the history of Unix shell wildcards, because you have to know to look at the glob source, not the shell). ↫ Chris Siebenmann I never knew expanding wildcars in UNIX shells was once done by a separate program, but if you stop and think about the original UNIX philosophy, it kind of makes sense. On a slightly related note, I’m currently very deep into setting up, playing with, and actively using HP-UX 11i v1 on the HP c8000 I was able to buy thanks to countless donations from you all, OSNews readers, and one of the things I want to get working is email in dtmail, the CDE email program. However, dtmail is old, and wants you to do email the UNIX way: instead of dtmail retrieving and sending email itself, it expects other programs to those tasks for you. In other words, to setup and use dtmail (instead of relying on a 2010 port of Thunderbird), I’ll have to learn how to set up things like sendmail, fetchmail, or alternatives to those tools. Those programs will in turn dump the emails in the maildir format for dtmail to work with. Configuring these tools could very well be above my paygrade, but I’ll do my best to try and get it working – I think it’s more authentic to use something like dtmail than a random Thunderbird port. In any event, this, too, feels very UNIX-y, much like delegating wildcard expansion to a separate program. What this also shows is that the “UNIX philosophy” was subject to erosion from the very beginning, and really isn’t a modern phenomenon like many people seem to imply. I doubt many of the people complaining about the demise of the UNIX philosophy today even knew wildcard expansion used to be done by a separate program.

While we’re out here raising funds to make me daily-drive HP-UX 11i v1 – we’re at 59% of the goal, so I’m starting to prepare for the pain – it seems you can actually run older versions, HP-UX 10.20 and 11.00 to be specific, in a virtual machine using QEMU. QEMU is an open source computer emulation and virtualization software, first released in 2003 by Fabrice Bellard. It supports many different computer systems and includes support for many RISC architectures besides x86. PA-RISC emulation has been included in QEMU since 2018. QEMU emulates a complete computer in software without the need for specific virtualization hardware. With QEMU, a full HP Visualize B160L and C3700 workstation can be emulated to run PA-RISC operating systems like HP-UX Unix and compatible applications. ↫ Paul Weissman at OpenPA The emulation is complete enough that it can run X11 and CDE, and you can choose between emulating 32bit PA-RISC of 64bit PA-RISC. Devices and peripherals support is a bit of a mixed bag, with things like USB being only partially supported, and audio not working at all since an audio chip commonly found in PA-RISC workstations isn’t supported either. A number of SCSCI and networking devices found on HP’s workstations aren’t supported either, and a few chipsets don’t work either. As far as operating system support goes, you can run HP-UX 10.20, HP-UX 11.00, Linux, and NetBSD. Newer (11i v1 and later) and older (9.07 and 9.05) versions of HP-UX don’t work, and neither does NeXTSTEP 3.3. Some of these issues probably stem from missing QEMU drivers, others from a lack of testing; PA-RISC is, after all, not one of the most popular of the dead UNIX architectures, with things like SPARC and MIPS usually taking most of the spotlight. Absolutely nothing beats running operating systems on the bare metal they’re designed for, but with PA-RISC hardware becoming ever harder to obtain, it makes sense for emulation efforts to pick up speed so more people can explore HP-UX. I’m weirdly into HP-UX, despite its reputation as a difficult platform to work with, so I personally really want actual hardware, but for most of you, getting HP-UX 11i to work properly on QEMU is most likely the only way you will ever experience this commercial UNIX.

Today, every Unix-like system can trace their ancestry back to the original Unix. That includes Linux, which uses the GNU tools – and the GNU tools are based on the Unix tools. Linux in 2024 is removed from the original Unix design, and for good reason – Linux supports architectures and tools not dreamt of during the original Unix era. But the core command line experience in Linux is still very similar to the Unix command line of the 1970s. The next time you use ls to list the files in a directory, remember that you’re using a command line that’s been with us for more than fifty years. ↫ Jim Hall An excellent overview of some of the more ancient UNIX commands that are still with us today. One thing I always appreciate when I dive into an operating system closer to “real” UNIX, like OpenBSD, or a actual UNIX, like HP-UX, is just how much more logical sense they make under the hood than a Linux system does. This is not a dunk on modern Linux – it has to cater to endless more modern needs than something ancient and dead like HP-UX – but what I learn while using these systems closer to the UNIX has made me appreciate proper UNIX more than I used to in the past. In what surely sounds like utter lunacy to system administrators who actually had to seriously administer HP-UX systems back in the day, I genuinely love using HP-UX, setting it up, configuring it, messing around with it, because it just makes so much more logical sense than the systems we use today. The knowledge gained from using BSD, HP-UX, and others, while not always directly applicable to Linux, does aid me in understanding certain Linux things better than I did before. What I’m trying to say is – go and load up an old UNIX, or at least a modern BSD. Aside from being great operating systems in their own right, they’re much easier to grasp than a modern Linux system, and you’ll learn a lot form the experience.

“Dell UNIX? I didn’t know there was such a thing.” A couple of weeks ago I had my new XO with me for breakfast at a nearby bakery café. Other patrons were drawn to seeing an XO for the first time, including a Linux person from Dell. I mentioned Dell UNIX and we talked a little about the people who had worked on Dell UNIX. He expressed surprise that mention of Dell UNIX evokes the above quote so often and pointed out that Emacs source still has #ifdef for Dell UNIX. Quick Googling doesn’t reveal useful history of Dell UNIX, so here’s my version, a summary of the three major development releases. ↫ Charles H. Sauer I sure had never heard of Dell UNIX, and despite the original version of the linked article being very, very old – 2008 – there’s a few updates from 2020 and 2021 that add links to the files and instructions needed to install, set up, and run Dell UNIX in a virtual machine; 86Box or VirtualBox specifically. What was Dell UNIX? in the late ’80s, Dell started a the Olympic project, an effort to create a completely new architecture spanning desktops, workstations, and servers, some of which would be using multiple processors. When searching for an operating system for this project, the only real option was UNIX, and as such, the Olympic team set out to developer a UNIX variant. The first version was based on System V Release 3.2, used Motif and the X Window System, a DOS virtual machine to run, well, DOS applications called Merge, and compatibility with Microsoft Xenix. It might seem strange to us today, but Microsoft’s Xenix was incredibly popular at the time, and compatibility with it was a big deal. The Olympic project turned out to be too ambitious on the hardware front so it got cancelled, but the Dell UNIX project continued to be developed. The next release, Dell System V Release 4, was a massive release, and included a full X Window System desktop environment called X.desktop, an office suite, e-mail software, and a lot more. It also contained something Windows wouldn’t be getting for quite a few years to come: automatic configuration of device drivers. This was apparently so successful, it reduced the number of support calls during the first 90 days of availability by 90% compared to the previous release. Dell SVR4 finally seemed like real UNIX on a PC. We were justifiably proud of the quality and comprehensiveness, especially considering that our team was so much smaller than those of our perceived competitors at ISC, SCO and Sun(!). The reviewers were impressed. Reportedly, Dell SVR4 was chosen by Intel as their reference implementation in their test labs, chosen by Oracle as their reference Intel UNIX implementation, and used by AT&T USL for in house projects requiring high reliability, in preference to their own ports of SVR4.0. (One count showed Dell had resolved about 1800 problems in the AT&T source.) I was astonished one morning in the winter of 1991-92 when Ed Zander, at the time president of SunSoft, and three other SunSoft executives arrived at my office, requesting Dell help with their plans to put Solaris on X86. ↫ Charles H. Sauer Sadly, this would also prove to be the last release of Dell UNIX. After a few more point release, the brass at Dell had realised that Dell UNIX, intended to sell Dell hardware, was mostly being sold to people running it on non-Dell hardware, and after a short internal struggle, the entire project was cancelled since it was costing them more than it was earning them. As I noted, the article contains the files and instructions needed to run Dell UNIX today, on a virtual machine. I’m definitely going to try that out once I have some time, if only to take a peek at that X.desktop, because that looks absolutely stunning for its time.

The ELF object file format is adopted by many UNIX-like operating systems. While I’ve previously delved into the control structures of ELF and its predecessors, tracing the historical evolution of ELF and its relationship with the System V ABI can be interesting in itself. ↫ MaskRay The article wasn’t lying. I had no reason to know this – and I’m pretty sure most of you didn’t either – but it turns out the standards that define ELF got caught up in the legal murkiness and nastiness of UNIX. After the dissolution of the committee governing ELF in 1995, stewardship went from one familiar name to the next, first Novell, then The Santa Cruz Operation, then Caldera which renamed itself to The SCO Group, and eventually ending up at UnXis (now Xinuos) in 2011. In 2015, the last maintainer of ELF left Xinuos, and since then, it’s been effectively unmaintained. Which is kind of wild, considering ELF is a crucial building block of virtually all UNIX and UNIX-like operating systems today. The article mentions there’s a neutral Google Group that discusses, among other things, ELF, but that, too, has seen dwindling activity. Still, that group has reached consensus on some changes; changes that are now not reflected in any of the official texts. It’s a bit of a mess. If you ever wanted to know the status of ELF as a standard, this article’s for you.

I needed a bit of a break from “real work” recently, so I started a new programming project that was low-stakes and purely recreational. On April 21st, I set out to see how much of a Unix-like operating system for x86_64 targets that I could put together in about a month. The result is Bunnix. Not including days I didn’t work on Bunnix for one reason or another, I spent 27 days on this project. ↫ Drew DeVault Bunnix’ creator, Drew DeVault, has quite a bit of experience with writing operating systems, as they’re also the creator of Helios, an experimental microkernel operating system. Bunnix is remarkably capable for a 30-day project, and comes with support for both BIOS and UEFI boot, and it’ll boot on real hardware too. It doesn’t have USB support though, so if you’re going the real hardware route, you’ll need to take that into account for mouse and keyboard input. Bunnix has a relatively solid set of drivers, taking the short development time into account: among other things, there’s PCI, AHCI block devices, serial ports, framebuffers, and ext4 support. The kernel supports a virtual filesystem, a /dev filled with block devices, a terminal emulator, and more. Bunnix is single-user for now, so it doesn’t enforce file permissions, but DeVault states it should be relatively easy to implement multiuser support. A unique characteristic of Bunnix is that’s written mostly in Hare, complemented by some C. Hare is a relatively new programming language, which we touched on late last year when it was ported to OpenBSD. Implementing file systems proved to be one of the difficulties during development, partly due to Hare. I also learned a lot about mixing source languages into a Hare project, since the kernel links together Hare, assembly, and C sources – it works remarkably well but there are some pain points I noticed, particularly with respect to building the ABI integration riggings. It’d be nice to automate conversion of C headers into Hare forward declaration modules. Some of this work already exists in hare-c, but has a ways to go. If I were to start again, I would probably be more careful in my design of the filesystem layer. ↫ Drew DeVault DeVault’s post about Bunnix gives a lot more insight into the development of Bunnix, so I’d highly suggest to head on over to read more. Do note that DeVault considers Bunnix “done”, in the sense that the learning experience is over, and aside from a few random developments here and there, they won’t be doing any work on it anymore.