newLISP: the language
The first notable difference is related to functions. First of all, unlike Common LISP (but in a manner similar to Scheme’s), newLISP evaluate the operator part of an expression before applying it to its arguments. Furthermore, lambda expressions are not of a special data type, but actual constants which evaluate to themselves. These are subtypes of the list data type, and they behave like any other first-class obect, allowing you to manipulate them freely.
A notable aspect in this regard is the fact that (again, just like Scheme) newLISP uses the same symbol space for variables, primitives and user-defined symbols. However, unlike Scheme and Common LISP, all variables are dynamically scoped by default. Static/lexical scoping is acheived by enabling an original mechanism called contexts, which offer an arguably simpler and more transparent method of defining namespaces and building modules. Contexts are, however, more than just funky names for a namespace: contexts define lexically closed spaces as first-class objects, inside of which namespace scoping is dynamic; newLISP allows mixing lexical and dynamic scoping, allowing programmers to use contexts in order to build function factories, functions with state and software objects.
A third (and last) purely language-related detail we will discuss is that of implicit indexing. newLISP developers consider this to be a logical extension of LISP’s evaluation rules — an opinion with which it’s hard to disagree. Essentially, it boils down to replacing this:
(set 'myList '(1 1 2 3 5 8 '(13 21))) (nth 4 myList) => 5
with this:
(myList 4) => 5 (myList '(6 1)) => 21
To those who are unfamiliar with LISP and are left wondering why is array indexing such a big deal, a quick reminder: in LISP, lists (which are not the same as arrays) are a fundamental data type. Even a LISP program is a list that can be freely manipulated. Therefore, list manipulation really is a big deal in newLISP.
Implicit indexing (which is optional) does indeed look fairly Lispish, and it does seem a logical extension. However, given newLISP’s shared symbol space, its advantages can be debated. On one hand, the syntax is certainly a lot cleaner (especially in the second case, which would otherwise have to be written as (nth 1 (nth 6 myList))). On the other hand, unless the name makes it clear that you are accessing an element of a list and not calling a function, this formula can create confusion.
Two are the notable implementation features that we will discuss: the structure of lists and garbage collection. Although the differences between newLISP and other dialects are multiple, both in terms of architecture and in terms of syntax, we’ll only examine those that have greater practical importance.
In most LISP dialects, a list is implemented as a set of dotted pairs (sometimes called cons cells). Ignoring a great deal of details, a cons cell has two parts, called car and cdr for historical reasons. A list is made out of cons cells where the car (the “head” of the list) contains an atom — pedantry aside, a value –, and the “cdr” (the “rest” of the list) contains a pointer to the next dotted pair. More atoms glued together as dotted pairs form a list.
The reason of this otherwise fairly complicated scheme are mostly historical. newLISP does not have dotted pairs. Instead, every cell contains an object as car and a pointer to another object if the cell is in a list. Therefore, the behavior of cons cells is different in newLISP compared to other dialects — and extra care should be taken by those developers who like to abuse conses. The overall practical impact is fairly low though, since in most cases you do not deal directly with dotted pairs.
The part which does have a serious practical impact is garbage collection. Historically, this has been a major area of research, and (ironically enough) LISP is one of the languages that started it all. Modern Common Lisp implementations have efficient, but fairly complicated garbage collection schemes.
newLISP implements a simpler mechanism, where each object is only referenced once (ORO) — the only exceptions being symbols and contexts. This way, garbage collection becomes trivial: every object that is not referenced by a context or symbol reference can be recycled once the evaluation reaches a higher execution level. By means of a fairly simple mechanism of optimization so as not to spend too much time recycling cells, newLISP’s garbage collection system offers a faster evaluation time and a much smaller memory and disk footprint compared to other LISP interpreters.
The tradeoff for this is that, in order for the system to work, arguments have to be passed by value, and at this point you can hear C programmers leaving the room. This is only one side of the truth — since newLISP does actually allow arguments to be passed by reference, but it is not straightforward (it requires the use of contexts). Whether this is too much to endure really depends on who you’re asking and on the application you are building though. In most of the cases newLISP was built for, the overhead is really negligible, and the tradeoff is clearly worth it in terms of simplicity and size. However, some types of programs (such as scientific applications) where passing huge data structures as arguments is inherent, efficiency will require a short detour. Syntactically and practically this is not much of an issue, but it looks unclean.
Practical newLISP
As we mentioned earlier, newLISP aims to be a pragmatic, clean and expressive scripting language, though not in the semi-pejorative meaning of ‘scripting’, but rather as outlined by Ronald Loui in his In Praise of Scripting: Real Programming Projects. In order to achieve this purpose, newLISP has managed a few fairly remarkable features in terms of how it’s implemented and how it can be used.
newLISP runs on several platforms, including the well-known triad of Windows, Linux and OS X. However, it also runs on Solaris, various BSDs and Syllable. It also sports a Java-based GUI server which can use the system’s native widgets for GUI, but it also offers interfaces to GTK, Tcl/Tk and OpenGL for those cases when speed or portability are at stake. Indeed, the GUI server feels somewhat slow — partly due to its architecture, and possibly in part due to, well, Java. The library’s GUI abilities are also fairly limited (although more than enough for what it is meant to do), and there is no GUI builder available. Then again, we can freely argue that newLISP wasn’t really built with this in mind, and the chores of manually setting up the layout aren’t all that tragical when your application needs only a handful of widgets.
Apart from this, newLISP also comes with a handful of extra modules, touching a variety of subjects, from regular expressions and Amazon-AWS EC2 to network sockets and database interfacing. It also has an import function, which allows interfacing with DLLs and shared libraries.
Although all these standard modules are fairly limited in terms of capabilities, they do have the merit of being reliable, simple to use and understand and very well documented. As a consequence, building up further functions as required can be done with reasonable ease. The only current setback is that third-party modules are somewhat behind the changes newLISP has undergone in its transition to the recently-launched 10.0 version. As a consequence, their behavior may include unexpected failures and green dwarfs, as justly signaled on the newLISP homepage.
A very interesting feature of newLISP is its clean and very easy model of distributed evaluation. newLISP can run as a TCP/IP daemon, and through the use of a single function, net-eval, remote newLISP servers can be given instructions to execute. While high performance computing fans are already thinking about how you can parallelize LISP programs more efficiently, those who frequently write small scripts will appreciate the extra flexibility given to them. The implementation may not be particularly efficient in terms of performance, but it is efficient in terms of ease of use and flexibility.
newLISP also takes reasonable steps towards making it easy to distribute applications, which has traditionally been a chore in the LISP world, only comparable to that of propperly managing memory in C. The newLISP executable is small and it has a very tiny memory footprint. It can also be compiled as a UNIX shared library or as a Win32 DLL, so it can be embedded in other programs. The other option of interaction is the use of I/O pipes and network ports. The former is used by BBEdit, and the latter is used by newLISP’s Tcl/Tk frontend, among others.
Invented half a decade ago
yet it was used in the 1980s!!! impressive
edit ‘the quoted typo has now been ammended’
Edited 2009-01-05 19:37 UTC
It wasn’t used just in the 1980s, but the most famous achievements of Lisp’s date back to the 1980s. Actually, the best stuff Lisp was used for date back in the 1960s and 1970s, when it was THE promoter of many techniques (the most important being garbage collection) that are still used today.
After the 1980s, Lisp was somewhat abandoned because its main field (AI research) was itself all but abandoned.
yeah who cares about AI when real humans are as dumb as rocks and watch American Idol or Eurovision?
AI is not even close to abandoned.
Most of its areas have moved out into fields of their own (data-mining, searching and problem solving, etc)
but AI is a very lively field of study.
… of languages to have a look at and to eventually use for something “real-world”, I once again come down to wondering, in 2009, _what_ I actually might want to use LISP for. Sure, it’s an interesting language eventually covering a set of use cases better than other languages, same as Haskell or Erlang or Prolog or Scala or some others. But: Looking at my (narrow) horizon of using Java in a productive environment every day, I see it’s not really the language, the paradigm or the IDE to make this a useful tool – it’s the vast community, the overwhelming set of third-party components and code both open-source and proprietary, the presence of frameworks and platforms like Java EE or Spring, the availability of tools for modeling / MDA, testing, code coverage and so forth, the existence of application servers to allow for deploying, starting, running distributed applications without too much ado that still make me stay with Java rather than considering going for any of the (eventually more interesting) languages. So to say: Where in this toolbox to meaningfully fit in newLISP and friends? Where, in terms of doing productive work, can newLISP be dropped in there getting something done, making it more than just an “interesting tool”?
Thanks for any comments…
K.
A handful of places where Lisp is used can be found here: http://bc.tech.coop/blog/041027.html . Some of the links are dead (the blog post is almost five years old now), but it’s there all right.
I, for one, don’t believe in the rebirth of Lisp, in spite of liking it as I do. Common Lisp is too baroque to be meaningful today, and I don’t think tools like newLISP or Scheme can make good replacements for Java or C# (although without trying it, I can’t call this an informed opinion). However, there are a few features of Lisp that I wish Java — of which I’m a user myself — actually had.
I also think there’s a reasonable niche for Lispish scripting languages like newLisp; they’re dynamic ,expressive, have enough tools to be used for this, they’re portable and allow for quick prototyping, which is exactly what I’m looking for in a scripting language (and what made me try newLisp first, in fact).
Edit: many of the links on that page point to Franz Inc.’s webpage, which sells its own Common LISP implementation, so there’s a lot of marketing mumbo jumbo on them which I think you can safely skip.
Edited 2009-01-05 21:00 UTC
Thanks for the pointer, I’ll surely have a look and browse through these. Actually, I am generally enthusiastic about solving problems using tools that fit them best. I would love to, say, use a functional language to implement the computational core of our business application, or something like Prolog in situations in which business rules are to be implemented / evaluated. Maybe in many cases this would even eliminate the need to have custom DSLs because some language in existence might address the problem same as well.
But: In the end, to me so far it seems that, in terms of overall support and tooling, there are Java and .NET and no real other languages completely on par with both platforms in terms of feature completeness, tooling, adoption and “ecosystem”. So as long as it’s basically about choosing either one of them, throwing in another language to me mainly seems about adding complexity: I need an expert capable of working with this “additional language”. I eventually will have to deal with a broken tool chain, being in need of another IDE, different debugging / profiling tools, different deployment mechanisms and so forth in order to get code implemented in this language handled. Plus, in worst case I end up with making a “standalone” application a “pseudo-distributed” one as I am suddenly in need of enabling communication between code / subsystems implemented in different languages, making use of different runtime environments, … . Maybe, in Java, things like JSR-233 are likely to ease this a little on the long run, but so far I am really really sceptical about that…
Cheers,
K.
I agree; at its current state, Lisp does not make a good replacement for Java and .NET, not in the real world anyway. Most of those that have used Lisp for complex applications — including some of the companies listed on the page I’ve posted earlier — have complained about the lack of proper libraries. Emacs and SLIME make good enough development tools to be used, and now there are also good alternatives, like the two Eclipse plugins I can’t name right now (one was called Cusp I think, I don’t remember the other one). Most LISP implementations also have an excellent FFI and tools that can automatically generate bindings based on header files, but that really doesn’t make a good replacement for well-written and, most importantly, well-documented actual libraries. In fact, most LISP users are a bit uncomfortable with automatically-generated FFI bindings because, when they are automatically generated, they don’t make very Lispish code (and you probably don’t want to write C with extra parenthesis).
Granted, there are enough libraries to cover most major areas right now, but there’s a great deal of inertia. A few years ago when I first used Common LISP, the only reasonable options for GUI that I had were a toolkit called Garnet (big, complex and non-free) and Xlib bindings.
As for the syntax, I actually find it useful. Keeping track of parenthesis is hard if your text editor cannot do that automatically, but most editors do now. It also helps if it can indent your code properly. I never had problems with parenthesis. There are other problems in Lisp as a language, some of them nicely evaded by newLisp though.
It’s hard to say whether these are better or worse times for Lisp. It does get a lot more attention, and (unlike some years ago) there is some proper documentation available, like Practical Common Lisp, but simply being is the spotlight is hardly enough. Unless someone manages to make up a decent, active community that actively contributes code and ideas, like that behind Perl, Ruby or Python, Lisp is really doomed to be just the language that was cool back in the 1980s (which, for large values of 1980s, really was half a decade ago :-D)
The IDE and tool chain part is the easy part, really. If you have some clean code in lisp that translates into a mess in java, then you should use lisp, because managing an IDE and installing a tool chain for lisp is nothing when compared to debugging messy code: THAT is the hard part, and the bigger the code, the harder it gets. For small programs, it’s ok to use a random language, but when your programs have more than 100 000 lines of code, you better choose the right language, no matter how hard it is to install an IDE. When you have more than 1 million lines of code, you spend less time learning a new language and reading clean code than reading bad code in the language you know best. When the code gets really big, even writting a new language from scratch is easier than using a less than perfect language for your project.
Edited 2009-01-06 14:01 UTC
Dude, you really got the point. In so-called “IT” industry, the most worthless thing is the technology. Money is the real boss. Lisp itself is good. It evolves smoothly, in contrast, languages like java or C# are just evolution dead-end. I saw the talk about the future of C#, my impression is that these C# guys is doing the same thing as those C++ people did to C. They are just patching the existing language. And the result is that we end up with a language with awfully complicated syntax and semantics.
Why these guys like C flavor so much? I mean C is good, it is also an evolution dead end, but it is OK for C because C is designed to do low level programming such as writing kernels, device drivers or system programming, and this kind of jobs rarely changes. In contrast, languages like java or C# are different, they are mostly used in business world, and business changes fast.
Edited 2009-01-05 22:02 UTC
C was designed to be a high level portable language, not to do “low level stuff”. It was the C# or Java of its time, when everything was done in asm.
C# isn’t just rolling random features in, if you look at the last few releases it is obviously trying to roll some of the reasons that functional and dynamic languages are so awesome into a statically typed language while still maintaining clarity and performance.
Not only that, but the IT industry that you are disparaging is taking a good look at functional languages like LISP right now. There is a huge interest in Haskell, Erlang, and F# as a way to leverage parallelism in a smart way and to deal with certain types of problems that OO doesn’t handle well.
C is designed to be general purpose high level language, however, it is practically not good at developing large complex user applications, especially, those business applications which have to be fast-written and change greatly from one domain to another. C does not free programmer from taking extra care of low-level things.
No. At the time C was invented, the mainstream high level languages were COBOL, FORTRAN, Algol and PL/I. These languages do not handle e.g. pointers and bit operations as well as C does.
C was invented specifically as a system programming language (so they wouldn’t have to rewrite OS for different architectures).
One of the “friends” is F#. In the .net world, F# is a first class citizen, and can interop with C# assemblies with little effort and no penalty. The idea is that OO does not deal with certain kinds of problems well, while other paradigms deal with them beautifully. Functional languages are phenomenal for recursion, and can function in a parallel manner gracefully, both things that are very ugly in OO languages.
I would argue that the IT industry is very interested in functional (and dynamic) languages, are are just trying to find the best way to use them right now. I haven’t been in the Java world for about 7 years now, but in the .net world functional languages are getting a lot of buzz.
Symbolics, Symbolics, Symbolics Symbolics Symbolics!
Sorry. Since we’re talking about Lisp I just had to get that out of my system.
Also, I believe OpenGenera is still available for Tru64 UNIX.
sorry, my wrong
Edited 2009-01-05 21:24 UTC
Clojure is much more interesting than newlisp. http://clojure.org/
And actually enjoyed it. But I am glad the author pointed out some of the glaring differences and how they might be confusing (indexes, eg.)
Overall, my impression of it was good too, fun language, really.
wislist -> wishlist
betweem -> between
I wonder if times are getting better or worse for LISP. If you listen to Paul Graham and some other LISP guys times are getting better because other languages slowly adopt features from LISP.
I, however, believe times are getting worse for LISP because its ecological niche is getting occupied by other languages.
The good news is that programming Ruby or Python is probably less painful for a LISP guy than programming C.
The bad news is that the incentive to make the jump to LISP from these languages is also smaller.
“whose fans look almost with despise at other languages rediscovering it” should probably read:
“whose fans almost despise other languages …”, or
“whose fans look almost with despite at other …”
Where 1st is more natural, and 2nd closer to original sentence structure.
Sorry to grammar/vocab nazi on you …
VERY high performance games of the previous generation of consoles…
http://www.gamasutra.com/features/20020710/white_02.htm
http://www.allgame.com/cg/agg.dll?p=agg&sql=4:3946
As most students of LISP will tell you, LISP does NOT stand for LISt Processing. It stands for Lots of Idiotic Stupid Parenthesis.
My favorite:
Perl => Pathologically Eclectic Rubbish Lister
(for the record: I love Perl)
I’m glad to see a renewed interest in LISP. I have always enjoyed the language and what can be done with it. The article and the comments have prompted me to finally register to OSNews after years of being just an anonymous guest.
Yes, all the parentheses are tricky to keep track of and pair properly so that the Read-Eval-Print Loop can handle the programmers’ thoughts as intended. However, this is not a major handicap once one has reached some fluency in the language.
One strength of the language is that Code and Data have the same fundamental structure. This has been used extensively in AI Research, including studies on AI programming languages and the creation of purposely defined LISP dialects.
I’ve done my LISP programming using muLISP on a PC (8088 @ 5 MHz, 512 kb memory, two floppies, no hard drive) years ago. One weakness I encountered back then for the specific case I was programming was the difficulty to handle time-stamped data. The data structures I came up with were likely not the best but I managed. I wonder if the younger dialects have pre-defined functions for this.
Although all data processing problems can be handled by any computing language, there are some pairings which are easier to mentally work with than others.
By the way, MDL (a.k.a. Muddle) from which current LISP draws some of its constructs was used for the original mainframe iteractive fiction game Zork. The coders went on to form Infocom and ported Zork to the early microcomputers via a virtual machine (Z-Machine)and a stripped-down dialect of MDL called ZIL(Zork Implementation Language).
Clojure which has been mentioned here as being much more interesting than newLISP is also a LISP dialect implemented in the Java Virtual Machine…..Many fundamental similarities between the old and the new although names, purposes, and underlying data processing power are vastly different.
They seem too complicated. They take a straightforward concept and make it very complex by adding a lot of first-class features.
Has anybody used http://www.iolanguage.com/?
It is a pure OO language with a prototype based system like JavaScript. It is a really pleasure to work with!