I’ll be setting my system up on a Windows machine, but you could just as easily set it up on a macOS or Linux machine. When I first attempted this install, I used VMware Workstation 14 Pro for Windows. After going through most of the installation process I couldn’t get OpenStep in to a color screen mode no matter how much I tried. I eventually found out that VMware Workstation doesn’t support the proper VESA display modes, and try as you might you’ll get an error on boot that says “VESA Mode Not Supported.” Fortunately, Virtual Box – which is cross-platform between Windows, macOS, and Linux and completely free – does support the proper VESA mode and works great for the installation of OpenStep.
Still, there are some quirks when setting up your VirtualBox VM initially. The VM must have 1 processor with 1 core, 64MB of RAM, and a 2.0GB hard disk on an IDE controller. You will need to configure one IDE CD-ROM drive and one floppy drive. The hard drive must be at IDE 0:0 (Primary Master), and the CD-ROM drive must be at IDE 0:1 (Primary Slave). You will have to edit the properties of the VM upon creation to ensure you adhere to these standards. All other default options of the VM during the creation wizard can be left alone.
NextStep is a lot of fun to explore and play around with, since you can clearly see the early days of Mac OS X in there, for obvious reasons. It’s definitely worth it to set an evening apart and follow this tutorial.