Submitted by Shells have been around forever and, for better or for worse, haven’t changed much since their inception. Until NuShell appeared to reinvent shells and defy our muscle memory. It brought some big changes, which include rethinking how pipelines work, structured input/output, and plugins.
We wanted to learn more about NuShell so we interviewed both of its creators: Jonathan Turner and Yehuda Katz.
I haven’t used nushell, but I am glad they are addressing long term limitations with unix piping not handling structured data in a good way. While the ability to stream data from one tool to another is a very powerful feature in unix, it’s too text-centric. Nobody should have to grep/cut/sed/awk/etc the data from text output when dealing with data. All of our tools really ought to output (and input) structured data directly! I want to be able to have SQL-like power for the command line!
A structured shell is a nice start, but ideally all tools (and not just the shell) could adapt support for a universal structured data standard such that all data created and consumed could be handled as structural data rather than as text. I called for something like this many years ago on osnews.
Weren’t you complaining about Journald saving logs in a binary format at some point?
Good luck getting people/companies to store data in any sort of sane format. I’ve run across lots of data that is technically structured. Maybe it should be a standard, like part of POSIX. 😛
Next, as soon as I suck the data into Python, Perl, C, or whatever it conforms to the structure I want, and the source doesn’t matter. Of course at that point, it’s for machines and not humans. Basically, human-centric design is holding us back. Narrow parts of the system should be human-centric, but assuming humans should be able to survive in all parts of the systems is holding us back. It’s an antiquated view.
I’m pretty irritated with “convenience” features getting in the way today (fighting with an ORM).
Flatland_Spider,
Yes because it doesn’t compliment the operating system’s native toolset and makes the data less accessible.
I agree, but it’s one of those things that should have been standardized decades ago, it’s so much harder to get there today.
Specialized tools have evolved way past simple text processing. I love the analytic power of SQL for example. I only wish that I could use that power everywhere and that every unix application could be a standard structured data provider. Had unix standardized on tables or full blown hierarchical data instead of lines of text, we would be working with higher level abstractions. Of course I can create a database by hand, but I can’t get it’s benefits out of the box across every single unix application without first having to parse & import lines of text.
I’ve done things the unix way for decades, whether it’s using character delimiters, writing regexes, or what have you, but it takes time and effort to get the data in a more usable form, It just feels wasteful to parse text back into a structured form when it came from a structured source inside of software in the first place. Ideally with better standards & tooling the structure could be preserved and rendered/manipulated by the shell or other more sophisticated tools.
I’m not convinced it’s about human consumption versus machine consumption. A table structure doesn’t favor computers over humans, it’s merely an abstraction. The underlying representation can be binary or text, but that’s not what I’m referring to.
Take the output of tools like top, ps, iptables, etc, the output as unstructured lines of text is less suitable than a tabular structure for both computer and human consumption because the table abstraction really is a better fit than lines of text. While the human visual cortex is adept at recognizing patterns in lines of text, this quickly breaks down when data grows beyond cell sizes or starts to wrap, not to mention it’s bloody tedious to work with tabular data in a plain text editor. Obviously spreadsheets can dramatically increase the efficiency of working with tabular data, IMHO it’s too bad our software IO conventions have stuck with lines of text.
Merits aside, I agree with you it’s unlikely that we’re going to see any mainstream convergence, we’ve been doing it this way for too long!
True. More people would be receptive if there was more tooling around it, and if it could be used as a log aggregator.
Text is only universal because it was the most human friendly encoding. People were expected to operate these things, and now that systems are managed with more automation, it gets in the way.
Modern reality is that most systems are managed by, or should be, tools, and we could strip the human centric parts out to get leaner systems which are easier to manage.
We have to start somewhere, and it is an upgrade to local syslog.
Apache Drill and the OSquery project might be of interest to you.
I’m not onboard the SQL everywhere train. Something like Binary JSON or MsgPack would be preferable storing for general data. Something easy to serialize and loose enough to be flexible. After that, it’s easy to filter the fields.
I think an OS would need to be data centric, for lack of a better term, to achieve any sort of consistency. The ideas is kind of like of creating logging pipelines with Fluentd or Logstash and database.
We create a new word processing document. File > New… and pick the type “Word Processing Doc”, or we clone a template. Right click “Word Processing Doc – Summer Theme” > Clone New Doc. Our Text Editor opens and creates an output channel with the word processing document filter specified.
Filters would be specifications on the data should be mapped and written to disk. They would be global, so any application could work with it. MS Word would only exist as a type of filter.
I’m just salty about CLI programs who’s output is human readable by default instead of machine parsable by default. I mean, the idea is to construct programs out of various parts to do the work for you. Having to build complex parsers because someone wanted to be cute and pretty print the data rather then something a little raw and ugly yet easy to parse is annoying, as you well know.
This is mainly specific to *nix being text based. Data being binary based would be different, but that’s not reality. People planning on the output of their tools being consumed by other tools would be a nice start.
*nix is a great programming environment, but people forget that and add creature comforts which don’t bring any value to the machine, just the humans who want to cosplay as sysadmins.
We’re not too far apart. I’m just angrier and more militant about it. 🙂
Yes, exactly.
I’m convinced the problem is being designing computer systems for humans first. It’s a bad way to think, and it leads to things like using a text file as a database because, “We can just open the file and read it.”
The sysstat is designed to get around the limitations of all the top-style performance tools. Which goes back to what I’m saying, the output is a little raw and ugly, but it’s easy to transform the data into meaningful results with some other tools. It’s not built for humans. It’s built for machines.
I think we’ll see movement simply due to the size of server fleets these days. Logging into 10 boxes and hand editing a config file is doable. 400 boxes, not so much.
We’re already starting to see points like centralized config management, centralized monitoring, centralized logging, and containers are the genesis of this. Each of these is a specific portal for people to interact with the systems, and completely abstracts the underlying system, and once the system is just a dot on a web page, it doesn’t matter what’s going on in a server as long and the data flows out of it into the C&C systems.
This does require people to think about minimalism and reducing the tech stack rather then shoveling more crap on top of the existing crap.
I want a Shellophone where I can use a CLI and say “Dial 1-800-555-1212” or similar instead of the annoying bad UI/UX versions.
I’m ready for Skynet to get here and decide to wipe out the human race or sequester people to the matrix. That would solve lots of problems, inparticular phone numbers.
They didn’t go far enough.
It’s not part of systemd. 😛
Flatland_Spider,
+1, made me laugh
It looks like a powershell for Linux on first glance. The best part about powershell is the structured data stuff.