Today, we’re introducing a new shell, written in Rust. It draws inspiration from the classic Unix philosophy of pipelines, the structured data approach of PowerShell, functional programming, systems programming, and more.
It’s called Nushell, or just Nu for short. We have a book (¡también se habla Español!). We have a repo.
Strike 1: Always be wary of anyone who pushes their .dotfiles to the official repo.
Strike 2: They re-implement every command. cp, mv, ls, rm. Only they don’t, so if you want ps ax | grep foo then sucks to be you.
Strike 3: Everything you want to add must adhere to a plugin api.
Also, I find it hilarious that the package manager for Rust is called “Cargo”.
Erez,
I think you’re under the wrong impression that they’re simply trying to create a shell that still runs POSIX programs. It’s obvious from their documentation that what they’re going for is a completely new environment, not “yet another way to run POSIX programs”.
Indeed.
Once people mention Powershell and structured data you’re closer to something like a Python REPL then a traditional shell.
And for that there’s already Xonsh.
1: What about people that feel the need to comment without feeling the need to know what they are commenting?
2: Rhetorical question: do you understand what they are aiming for?
3: The commands have to be coded against an API – how quaint! Unix would never have something like that!! /s
Strike 3 isn’t true. You can get things down to a string and bring in any other existing tool to the pipeline party.
Erez,
Maybe you don’t like the actual tools as they stand, do you have any experience with them? I understand why 2 & 3 are necessary to demonstrate the benefits of structured pipes. Sed/grep/awk/cut/etc are very powerful text processing tools, but they create a lot of work just to essentially converting unstructured pipe data into structured abstractions. As powerful as unix pipes are, the reliance on unstructured/pseudo structured text holds us back.
While technically with enough work you could build nearly any sort query using text processing tools, converting to text and parsing out text over and over is inefficient and you’re stuck with fairly antiquated primitives and limited indexing options. Something I’ve long wished unix had was structured data pipes so instead of having to convert everything to/from text we could instead use far richer and more powerful primitives (think SQL levels of power).
This would be an outstanding achievement. I don’t have experience with nushell in particular, but I am glad they’re working on structured pipes. At the same time I understand why it ruffles your feathers that older tools are being demoted, but I don’t think it can be helped: all of the old tools need to be upgraded to take advantage of structured data.
Our legacy designs are so ingrained that I’m rather skeptical of the industry’s ability to shift now. Ideally though an OS could be built from the ground up to take advantage of structured data & objects. This would not only give powerful tools to admins & power users, but it could potentially revolutionize the way we build software itself.
Hmm, I don’t get what’s funny about that? Is it a reference to the game called rust?
“Sed/grep/awk/cut/etc are very powerful text processing tools, but they create a lot of work just to essentially converting unstructured pipe data into structured abstractions. As powerful as unix pipes are, the reliance on unstructured/pseudo structured text holds us back.”
Let me guess, you think Ken Olsen was right about VMS being “all there”? True, Betamax was a superior product. It was also tightly controlled by Sony, and VMS was available only from Digital/HP/Compaq. But look at who won the war in each case, and the “why” becomes obvious: giving customers (users) freedom to choose.
(Before you tell me Microsoft Windows won the desktop, ask yourself: what OS runs on the most devices in the world?)
gus3,
You completely missed the point. He’s not arguing to displace POSIX with proprietary solutions. He’s simply saying there should be better tools for working with structured data in pipes. Structured pipes could all be part of modern way of doing things in POSIX, if enough people wanted to.
There’s no “war”. This is simply about commandline programs and better ways of working with pipes.
gus3,
I’m not sure where you get the impression that I’m in favor of some kind of vendor bias. Ideally we would have open standards that literally everyone can incorporate. The thing is it has to start somewhere. Instead of taking the attitude that technology needs to remain fixed to what unix/posix was in the past, I’d like to see operating systems to evolve past that. On the one hand it’s nice that some tools have been around for so many decades, but on the other hand these tools were built for the dawn of computing when data was much simpler.
Speaking about user choice, maybe the best way to satisfy most people would be to have multimode tools that still output text, but have the ability to include structured metadata. This way the familiar legacy text processing tools would still work and users can migrate at their own pace. This would please me since I do a lot of text manipulation when I’d rather have structured data, however I concede that as with so many other things, uptake could be slow due to the existence of a chicken and egg problem: users don’t want to invest in something that doesn’t have widespread support, and companies/developers don’t want to support something that doesn’t have many users. The best odds for something like this to really take hold would be for an influential brand or person to push for it,
FreeBSD is moving that way, via libxo a “library for emitting text, XML, JSON, or HTML output”. Any program that uses libxo can output data in structured (JSON, XML, HTML) or unstructured (plain text, HTML) ways. There’s a whole slew of programs in the base OS (/ and /usr directory trees; not /usr/local) compile to support it. Supposedly, it makes scripting a lot easier.
https://www.freebsd.org/cgi/man.cgi?query=libxo&apropos=0&sektion=0&manpath=FreeBSD+12.0-RELEASE+and+Ports&arch=default&format=html
Nice! Hopefully more and more programs will allow a damp of their data using xml or json, like postgres and many programs already do.
There are some neat applications that already work on structured data, like xmlstarlet, jq and trdsql but, unluckily, the problem is far from trivial. Even text matching, when dealing with unicode intricacies and locale rules are a pain to work with.
Anyway, we are inching closer to a sufficiently good toolchain.
Pertinent reading: https://gidi.io/2019/03/01/structured-greping-of-structured-logging-using-grep-for-json.html
This is a recurrent subject, isn’t it? We all would love to have a way to deal with structured data going through pipes, but every useful tool in the chain of commands would have to be aware of it.
Microsoft solved it for its intention, handle the registry from powershell, but, at least to me, in a kludge way.
Fact is, unix solved the data flow problem using the least common denominator and it works for the immense majority of cases, and for specialized cases there are explicit tools for SQL, xml and whatnot.
Anyway, to properly handle the situation we should have a system standard IPC/RPC method by now, and we still lack it.
acobar,
To be fair though, the same is true of any posix conventions, there’s nothing inherently superior about the way things were done originally and I don’t think we should relegate ourselves to the past, we should be improving things over time. Take for example unix file modes for user/group/other R/W/X permissions, there are boat loads of situations where that just doesn’t cut it, unix designers acknowledged this and added support for ACL. The end result is not as nice as if unix had been built around ACLs in the first place, but some people must be very thankful to have solutions that aren’t so constrained by legacy conventions.
IMHO the lack of structured data is in a similar boat, a lot of people would benefit and if done properly the text using folks could continue using their text parsing tools without interruption.
Sure, basic tools are a least common denominator, but since when did we make that the goal for unix? Do you want the least common denominator of /bin/sh or do you want one with more powerful features? I know that I certainly make use of more advanced shell features on a regular basis. In the same vein I find VIM to be significantly better than VI. Sure I can get by with the least common denominator, but I find it absurd to limit ourselves to it when better tools are possible.
Love it or hate it, I think D-Bus is that standard for linux.
https://en.m.wikipedia.org/wiki/D-Bus
You might find this link about posix interesting…
https://www.samba.org/samba/news/articles/low_point/tale_two_stds_os2.html
I have this habit of not be clear enough, I think. My point is not that I would not like to have progress, but that what really needs specialized handling already got it and what doesn’t is also already contemplated, even though we have to contort the data passed through. I will keep my eyes on this project, though.
Unluckily, it is momentarily, or so it looks. We are still in search for a standard that all agree upon. You, for sure, already know about it, others here, perhaps, may not and should have a ready about BUS1 and dbus-broker.
Thanks for the link. Can’t image a world without ACL and compatibility, even if not complete.
My wish list still lacks a proper RDP on linux but most of the remote access I do is via ssh, anyway. (Not blaming remmina, xrdp, or x2go, they are fine, but not at the same level as Windows RDP).
acobar,
Or maybe I have a habit of misunderstanding, haha.
As will I.
I disagree with your assertion though. In my line of work I do quite a lot of process automation involving parsing data fields from output and I certainly don’t feel that everything that needs structured data has already got it. More often than not I feel the opposite is true when I have to use a complex combination of text parsing tools to scrape values out of a pipe. I get that is how we’ve always done it in the past, it does work, but having been spoiled with a taste of SQL, it’s extremely frustrating going back to less powerful tools to handle data output. I seriously believe that we could do better.
I agree with that. SSH is great for console and VNC/rdp have some rough edges on linux.
I use VNC mostly for remote VM desktops. I had considered alternatives such as NX and spice although I put that search on the back burner years ago.
I hadn’t used remmina before, I think I’ll give it a shot, thank you!
https://remmina.org/
How are you doing quotes? I haven’t figured that out for the new WordPress site.
“While technically with enough work you could build nearly any sort query using text processing tools, converting to text and parsing out text over and over is inefficient and you’re stuck with fairly antiquated primitives and limited indexing options. Something I’ve long wished unix had was structured data pipes so instead of having to convert everything to/from text we could instead use far richer and more powerful primitives (think SQL levels of power).”
I’ve come to the conclusion that writing a shell for anything more then launching a couple of commands is madness, and switching to a full programming language is a better option. Preferably something compiled language.
Of course, we have the problem of which one to choose, especially if we’re going to be using this early in the boot process or on minimal systems. Perl and Python each have heavy dependencies on interpreters. Go and Java are better for services. Haskell hates people, and Rust, C, and C++ scare people. So maybe Lua or TCL?
Powershell is a nice idea, convert data to objects instead of text streams, but it’s UX is horrible. The UX of traditional shells isn’t bad. Focused tools with minimal options which are relatively easy to use.
“I understand why it ruffles your feathers that older tools are being demoted, but I don’t think it can be helped: all of the old tools need to be upgraded to take advantage of structured data.”
Also, Rome wasn’t built in a day. It’s going to take time.
I’ve had the thought lately to write a service which would return data about the system, so you never have to log into the system just connect to it. Kind of like MOSH mixed with OS Query and config management. Move to the paradigm where machines are nothing more then endpoints, and everything is RPC based.
Try <blockquote> Text you want quoted </blockquote>
That works. I still need to figure out how to escape code, but that’ better.
Flatland_Spider,
Ah yes, the language debate was going to crop up eventually, haha. There’s no reason in principal that all languages couldn’t input/output structured metadata, however I concede it’s much harder to get there today than if it had been part of a POSIX standard from the get-go. I think the naysayers have a valid point about it being hard to change the least common denominator once it’s in place, however it would have been so nice if unix supported structured data pipes from day one and new languages would have naturally incorporated it from day one as well. I doubt there would be any naysayers if it had been done that way in the first place, but alas I can’t change history. 🙁
One aspect that’s given me trouble over the years is quoting/escaping complex strings. for sufficiently complex strings, it’s very difficult to figure out how many backslashes I need and what backslash sequences are supported. Sometimes I wish shells had some kind of smart paste feature to take care of escaping. This is equally problematic for unstructured text output because the text we’re parsing could contain the very delimiters used to separate fields. This is a major pain when it comes to parsing text data, even something as simple as CSV can become extremely complex and error prone using “simple” unstructured text processing tools.
Yeah, we often bump up the “it’s good enough” principal as well, which can preclude changes regardless of the benefits.
This is what I envisioned for my OS. Everything was to be network transparent. Alas, I stopped working on it when I started my career and nobody wanted to pay me to work on it 🙁
> Maybe you don’t like the actual tools as they stand, do you have any experience with them? I understand why 2 & 3 are necessary to demonstrate the benefits of structured pipes. Sed/grep/awk/cut/etc are very powerful text processing tools, but they create a lot of work just to essentially converting unstructured pipe data into structured abstractions. As powerful as unix pipes are, the reliance on unstructured/pseudo structured text holds us back.
In my experience it’s the opposite. If you’re working with structured data like an SQL database, html, xml, or json there are Unix filters to let you manipulate them cleanly, as well as nice libraries in real programming languages that let you build tools in whatever way you choose and dump data to a file descriptor for further work, but a lot of data really is best treated as a bag of bytes. I’ve run into several situations where there was no clean way to express a pattern structurally in such allegedly modern shells and it was easier to dump an intermediate file to then process in a bash pipeline.
Anyone else mis-read the name as “nutshell”? That honestly seems like a better name to me, though it could cause issues if there was ever an O’Reilly book on it: Nutshell in a Nutshell?
I thought of “nutshell” too. 🙂 It’s an opportunity missed. 🙁
Yep, me too. And yes, “Nutshell” sounds better to me. “Nushell” is … boring.
On the other hand, “Nutshell” is probably less googleable.
Kind of like it for processing data, still pretty immature and relies on the “nightly” release of rust. I like the idea of kinda powershell but more posix like and not giant commands strings. Something more complex than just text, but not quite c # objects.
Also read as nutshell.
Plugging my own work here a bit:
https://github.com/guan-jing-ren/native-2-web
It’s not that difficult to implement structured pipes. Even though this project demonstrates end-to-end structured pipes (web browser to web server), the underlying protocol is just a straightforward self-describing serialization and so can be used with any file like thing.
Plugging in new APIs is easy. The demo program is just a straightforward adaptation of C++’s filesystem API using nothing but declaring which C++ classes and their members to write to a stream, and the functions to make available for calling over a stream.
Using c or c++ to do webstuff has historically turned out to be a security nightmare. That’s not to say that it couldn’t be done securely, but that there aren’t many that can do it.
Bill Shooter of Bul,
In my experience, it’s that a lot of “C” or “C++” web stuff continues to be written using manual low level pointer manipulation. They actively refuse to use new stuff.
My library uses new stuff. And it also uses reflection and metaprogramming, all hidden from the library user, so that there’s absolutely no low level stuff required from them. The reflection and metaprogramming innards figures out all the stuff that’s required to send and receive data over files/pipes/sockets/websockets. There’s not even a risk of introducing a parsing bug from a third party library because there’s no XML or JSON or other text format. It’s either the data provided matches the expected binary format or it doesn’t.
100% agree, but still not a great idea. There are plenty of articles out there decrying all of the ways even high level new c++ is still dangerous.
Bill Shooter of Bul,
There are plenty of scaremongering articles out there about everything in every language. It’s a new cottage industry – lots of making mountains out of molehills (“if you do this stupid thing, look at the horror you get; bad language, bad!”).
Most criticism of new C++ features comes from issues that affect almost nobody, except for experimental library writers who want to push everything to its limits. For day to day stuff, which includes web stuff, normal uses of new C++ is completely safe.
There’s absolutely no call to use low level stuff if using C++ in areas where scripting languages. You can get the same expressiveness (or more), and less bugs due to compile time testing.