How is Lisp useful?

Note: this was originally a post I wrote on Lemmy in answer to the question what type of problems do you solve using Lisp?. The post got to be a bit too long, and I am re-publishing it here as a proper blog post. I am also including some of a post I wrote on Mastodon which touched on some of these same issues.

So to answer the question: I have known about Common Lisp and Scheme for years, but only recently started using them. This is the story of the 3 Lisp dialects that I use.

Emacs Lisp

I use Emacs and Emacs Lisp to manage my tens of thousands of text files, I write Emacs Lisp scripts to automate simple tasks like searching for pieces of information, formatting it, and outputting it to a report that I might publish on my blog or send in an e-mail. I also use Emacs to help with data cleaning before running machine learning processes. Emacs helps with navigating CSV and JSON files, it also is a really good batch file renamer.

Scheme

I have recently started using Guile Scheme to do some personal projects. I went with Guile over the myriad other Scheme dialects because it is the implementation used for the Guix package manager and operating system.

• Also, there is Goblins, which is a distributed object-capability programming system is officially supported on the Guile platform, and I have been really wanting to write applications using this programming style ever since I first learned about it.

• Also, there is the G-Golf foreign interface layer allows Guile to automatically use an C library that implements the GObject Introspection interface. So through Guile, like with Python, you can use any C code library used to create of all native apps in the Gnome, MATE, Cinnamon, or (my personal favorite) the Xfce desktop environments. This potentially makes Guile a viable alternative to Python scripting across all of those Linux desktop environments.

Scheme is my favorite

Of all the Lisp dialects, Scheme is my favorite, for a few reasons:

• It is absolutely tiny. Guile is relatively large (not as big as Common Lisp), but other implementations are unbelievably small. for example the Chez Scheme petite interpreter is fully compliant with the R5RS standard, and the executable is like 308 kilobytes on a 64-bit Linux computer system.

• Hygienic macros with syntax-case

• Recursive functions over using the loop macro of Common Lisp. When writing algorithms, I personally find it easier to reason about recursive functions than loops. Scheme also provides me the ease-of-mind that comes with knowing the optimizing Scheme compiler will ensure recursive loops will never overflow the stack.

• Pattern matching is well supported by most Scheme implementation.

• It is a "Lisp-1" system, meaning there is only one namespaces for variables and functions, as opposed to Common Lisp (a "Lisp-2 system") which allows a name to be either a variable, a function, or both. I personally find it easier to reason about higher-order functions in Lisp-1 systems.

• Support for Delimited Continuations, which is a fairly recent discovery of computer language theory (first being discussed back in the 1990s), but is available across a few Scheme implementations.

The problem with other languages

The modern software industry, especially in the realm of big data and machine learning, has mostly settled on a pattern of using C++ for creating performance critical libraries, and creating Python binding to the C++ libraries for scripting. I was hoping languages like Haskell and/or Rust might come along and change all this, but it will take decades (if ever) for the software industry to turn in that direction.

The problem with languages like Python, in my experience (and I believe many other software engineers would agree) is that it does not scale well to larger applications at all, whereas languages like Common Lisp and Schem do. This is for various reasons, but mostly due to how Lisp does strong dynamic typing, and also the CLOS implementation of the meta-object protocol. Yet too many companies waste time writing large applications in Python — applications that are much larger than the scripting use cases that Python was originally intended to be used. I believe this is time and money better spent on other things.

If you want a truly general purpose high-level language that can be adapted to many different tasks, Common Lisp or Scheme is considerably better. The interpreters and compilers for these languages actually provide carefully designed mechanisms, like macro expansion and pattern matching, specifically for adapting the language to different tasks. This is possible because the syntax of the language is so simple and minimal that it is very easy to devise embedded domain specific languages (EDSLs) with very little effort, and without requiring people to expend the effort of learning whole new languages.

Python's syntax is relatively complex compared to Lisp, and its APIs for modifying the compiler and interpreter are not at all well-designed for adapting the language to various tasks compared to those of Common Lisp or Scheme. Creating EDSLs is for Python is not idiomatic coding style and discouraged, but people try to adapt it to every possible task anyways, and it becomes a horrible mess.

So let me say, adressing the software industry as a whole, that if you wanted to learn just one high-level language to solve every problem for you, it ought to have been Scheme or Common Lisp. Relatively speaking, Python is so incredibly limited in what it can do compared to those languages, Python was objectively the wrong choice for this lets adapt it to all purposes way of thinking. I strongly believe that the software industry is truly in a horrible mess as a result.

Common Lisp

That said, I am also starting experimenting with Embedded Common Lisp (ECL) because it is a lightweight standards compliant Common Lisp implementation that compile your program into C++ code, and this is useful to my professional work.

So I see Common Lisp, and the ECL compiler, as a potentially viable alternative to the sub-optimal status quo of Python as a scripting layer around C++ code libraries, at least perhaps for my day job, if not being more generally true industry-wide. Mostly, ECL would allow me to write a program in Common Lisp instead of Python, but deliver to my clients the C++ code that ECL generates to be used in their machine learning projects. (I have not actually done this yet, I am still investigating whether this would be a viable solution to any of my projects).

ECL makes it easy to use C++ libraries through Lisp instead of Python. And there are so many good C++ libraries out there: Qt, OpenCV, Tensorflow, PyTorch, OpenSceneGraph, FreeCAD, Godot game engine, Blender. And it compiles easily on Linux/Unix (GCC), Windows (MSVC), and MacOS (via Clang++), so good for cross-platform development.

UPDATE (2023-12-18):

My experiments with ECL unfortunately did not work out due to the fact that ECL does not generate very human readable C++ code, it does not even generate structured code with C++ classes. A Scheme implementation such as Gambit is probably a more reliable option for doing transpilation to languages like C and JavaScript, and has good very good support for MSVC on Windows, GCC, on Linux, and Clang on Mac OS. There are also experimental Lisp-like languages such as The Wax language which are specifically designed as meta languages with type checking which allow you to output more human-redable code.

Conclusions

So in spite of Lisp being such an old family of languages (its earliest incarnations dating all the way back to 1958), and being superseded in popularity and widespread use by languages like Python and JavaScript across the software industry, Lisp is still a modern, relevant, evolving, and very useful family of programming languages. At the same time, a Lisp such as Scheme or Common Lisp would even be a better choice of programming language in many applications where Python is currently used.

I just hope I eventually find the time to try out all of these Common Lisp and Scheme related ideas I have. I especially hope ECL turns out to be a profitable technological choice for the professional work that I do. But only time will tell.