Today, I wanted to show you one of the most fascinating and surprising operating systems ever created. It’s not another Unix, Linux, or Windows. It is an architecture that went its own way and proved that systems engineering design can look completely different.
I’m talking about IBM’s child, which for many might be synonymous with “boring banking systems,” but in reality, is one of the most uncompromising projects in IT history. While we get excited about abstraction and virtualization today, thinking we are discovering new lands, this system was doing it decades ago. Imagine a system that doesn’t know the concept of a “file” in the way we understand it. A system where everything is an object, and all disk and operational memory form one vast, flat space. If you are looking for proof that true engineering doesn’t need buzzwords to blow you away, I invite you to read on.
↫ Kamil Pytliński
Ever since watching Clabretro’s detailed video about getting IBM i to work on his own IBM POWER hardware and then remoting into them, I’ve been obsessed with running IBM i at home. It feels like the final boss of operating systems to dive into and explore, hidden in the deepest, darkest trenches of the ocean of technology. Everything about IBM i feels alien, complex, convoluted, opaque, and overwhelming, and you can probably dedicate your entire career to working with this platform and somehow still learn new things about it every day.
There’s something brutalist about IBM i, and I so desperately want to bang my head on its concrete walls.
In college I had a class on SQL/400 and the instructor had an AS/400 server in the classroom that we all logged in to using real hardware terminals. He was a huge IBM guy, and had been studying and teaching the platform pretty much since its inception, and he loved it when his students had the same “a-ha!” moments he had himself experienced when learning the system.
And yes, it felt alien even to someone like me who had previously operated data terminals at a government office job.
I started my third 3rd career based on AS/400: My former employer had only experience with small European banks and Oracle and then failed miserable when the largest Indonesian Bank showed up with some real data volume on an AS/400. Their ETL just went belly up after 2 hours with less then 1 mln records extracted or so.
I was the only boots on ground, there was a long weekend in a boring hotel in Jakarta and on Tuesday, the whole 40 mln records were written into compressed Loader files in less than 20 minutes. That was the start of my own company in 2010 and this ETL module is still the backbone of everything I am doing. Thank you IBM!
It is definitely nice to have your own little cozy niche 🙂
In my one encounter with IBM i when I had to pull some data from it via ODBC , at first it spewed “gibberish” and then I remembered that it is still EBCDIC not ASCII to this day , because IBM platforms predate ASCII standard. Definitely memorable moment.
In the government job I mentioned in my other post, the county IT department was in the process of moving our systems from AS/400 to Wintel, and their big pain point was the EBCDIC to ASCII conversion, it was the longest and most frustrating part of the changeover. That was before my time as a sysadmin, I was a terminal operator for the county emergency services at the time, so I was on the peripheral. But the IT guys and I got along great, naturally, and they shared a lot with me about the process. It actually helped prepare me for my later college classes on that system.
Morgan,
I’m guessing that their solution was to implement an EBCDIC application on windows? If so, then wow that seems like a lot of work. I wouldn’t recommend doing it that way. it’s easier to convert the data into a native character set and create a normal windows application with standard ASCII/UTF16. I’ve actually worked on a lot of mainframe conversion projects and by and large the the EBCDIC conversion was handled transparently by the same middleware that was providing access to the mainframe. This was in the 2010s and the tooling for this was already mature. Things may have been harder without the middleware though.
Actually as far as I remember, they said they were doing the data conversion first to ASCII then to MS SQL Server on NT 4.0. During the transition they changed out our hardware terminals for Windows NT boxes that booted directly into a terminal emulator to continue access to the old system while the changeover was taking place. I was just a computer hobbyist at that time and was just a data terminal operator at that job, so I didn’t have a lot of insight into the whole thing, only what my friends in IT told me, plus it was 26 years ago so my memory is fuzzy on the whole thing.
I do recall them explaining to me what EBCDIC even was and why it was so different from ASCII, and they had a pretty low opinion of x86 in general. It wasn’t their choice to move to Wintel, it came from the county commissioners.
Morgan,
Yep, most likely they FTP’d the EBCDIC data from the mainframe, converted it to ASCII, then imported it into SQL.
IMHO EBCDIC isn’t a huge issue if it’s converted early. The conversion itself is usually done looking up values in a character array, which requires a trivial amount of code. The difficulty is in reengineering the whole software stack around completely different primitives from those that mainframe devs are used to, especially the way we store data records. Mainframes pass around low level data structures, which makes them very fast whereas something like SQL requires a lot more code to encapsulate and marshal data between processes.
Yeah. I find old school mainframe software interesting to study, although personally I like higher level databases. Interestingly today I find myself torn between relational SQL databases and “nosql” object stores. SQL’s main shortcoming is that everything has to be translated into ridged table structures, which is frustrating when so many data sources are JSON objects with their own hierarchies. It’s tedious to convert between these.
A bit off topic, but recently I’ve been trying out different JSON document stores. I find myself missing SQL’s sophisticated queries and indexes. Ideally the database would work with JSON primitives but not sacrifice SQL capabilities. The no-sql databases I’ve tried fall short. My last project I stored json data inside of an SQL db to emulate a hybrid solution using mariadb’s json_* functionality. It’s helpful that mariadb lets us create virtual columns to expose the data within the json objects as SQL query fields. Unfortunately mariadb’s implementation is poorly optimized because it’s json data types are just strings and not proper object stores. Also mariadb is incomplete, for example not being able to create an index on virtual date fields. Technically there is no reason for this, it should just work, but it’s an unimplemented case.. Also mariadb lacks json subqueries (which mysql supports), Because it’s so sophisticated, SQL has hacks to get around this, but it is extremely inefficient potentially multiplying the amount of json decoding by hundreds for each record.
I’m very tempted to build my own SQL/no-sql hybrid database to address these shortcomings, but I don’t think any clients would want to pay for it and unfortunately paying the bills is a factor for me.
My father was a salesperson for one of the big pharma for 20+ years, handing surgical equipment. Their sales logging system and inventory ran on AS/400 all this time. I remember as a small kid going to the office he shared with other 4 sales folks and their assistant. They would dial into AS/400, log sales in, print the current prices and inventory for customers.
Then modernity came in. Firstly they got a 386 laptop, but they still had to go to the office to connect, so folks still preferred to use the terminal connected to the modem (it had a huge keyboard, probably IBM). Then they got a Pentium laptop with Windows 95 (the domain selection dropdown list showed MANY entries), a portable infrared DeskJet (the most adorable thing ever) and a CSD connection over the phone. Same deal. Dial into the server, get the inventory for customers, place orders, but this time they could do it live together with the customers and live check for discounts for volume purchases, tweak orders on the fly, etc..
I was already 15 when they piloted some web ui thing and everyone hated it, so they went back to directly interfacing with the AS/400 via terminal emulation. My dad knew the keyboard shortcuts better than the names of his children. He could interface with the thing FAST. He would complain about Windows going belly up, or not being able to log into shared drives (which started affecting his work), but the AS/400 thing always seemed to work. They had now a fancy automation that would generate pretty Microsoft Word documents instead of just printing out proposals straight from AS/400. Looked fancier, but same information, and crash prone. He would generate a document from AS/400 and then go to a shared drive, open the document in Word, and print it.
So yea… now we have this idiocy of every X months deploying new tools, retraining staff, migrating data, instead of just relying on what works. Not everything has to have fancy animations, print amazing fonts, be a web app. Sometimes we just want to get the job done.
He retired a few years ago and they were still using the same system.
I do find the IBM i architecture really interesting, and I’m glad you posted this, but… what a bizarre read. This person talking about the OS sounds like Trump talking about himself. Everything (including things like calling a startup script QSTUPGM) is “brilliant,” and it’s so incredible that you could upload files from a MERE PC that only has double or quarruple the compute of a low end POWER server.
Lots of conclusions that the way i does things is the only way to be able to accomplish them just because they’re done on i (processing millions of bank transactions MUST require a lack of filesystem hierarchy and be run in a single-tasking batch slot, OBVIOUSLY. If it could be done any other way, of COURSE banks would stop using COBOL)
something something kool aid, interesting read but I feel dirty
That’s because you only know the “PC” world and you can only think in those concepts where IBM’s “i” is so totally foreign to you that it flies past you without any understanding.
It doesn’t matter how fast a PC is because a PC can only handle so many jobs at once, can only account for so many applications and users at once. It’s not the speed of one machine but of the whole which is many times faster than a PC can ever be for any application or batch job.
Basically it isn’t a matter in intellect but a matter of talking in a totally different language that you’ve never been exposed to, no matter how smart you are, you aren’t going to understand.
A great and élégant platform, as soon as you learn it, other OS seems messy and a pile of dubious stuff.
You can try the OS freely on pub400.com ! Not a virtual machine, you will get a login on a recent IBM i
Download IBLM client access, or another 5250 emulator like tn5250 to gain access via terminal
I’ve rewritten what I’m writing about this about 30 times now.
I read through the whole article and because that is something that I was part of as a programmer for a third of my career, everything he talked about “just made sense” and I totally understood it. Keep in mind that this was written to give a taste to people who have never been exposed to “i”. But it was a walk through history for me which I really, really enjoyed.
For a long time OpenVMS was my final boss of operating systems. Now that I can play with a virtualized VMS in my computer, it feels like I need to find another final Boos. Tandem (now HPE) Nonstop OS looks sufficiently inaccessible to be final Final Boss.