Challenges in writing portable Scheme code

TL;DR:For almost two years now I have been trying to write a large-scale software application in the Scheme programming language in a way that makes the application portable across various Scheme implementations. With this experience, I have learned some of the limitations of the Scheme programming language standard, which collectively refers to the R7RS Small and Large standards, and the SRFIs (Scheme Requests for Implementation). Even after a lengthy heroic effort by the Scheme standards committee the Large standard is still incomplete, but they have already inducted a number of useful SRFIs into the standard, enough so that the incomplete R7RS Large standard is still useful. The Scheme standardization process has been a study in pure democracy, which is just as messy and frustrating as it sounds &emdash; the term "anarchy," in in it's political sense, is an accurate way describe the process, in my opinion. Consequently, Scheme as a language standard is still quite fragmented, which provides challenges for using it to do real software engineering.

Once I was convinced to learn the Scheme programming language and had committed to getting started, the first problem I had to solve was very unlike any other language I had ever learned: which Scheme implementation should I even use? Other standardized languages have a single most-popular implementation which everyone uses (e.g. Haskell, Python) or maybe two most-popular (e.g. JavaScript), so you can use the one that came built-in to your computer, or else just download the one most popular implementation from the website, and then start doing tutorials. With Scheme, this is simply not possible, there are several dozen implementations each of which implements one of two mututally exclusive Scheme (R6RS or R7RS) standards, and each one to varying degrees of compliance.

The R7RS Small Scheme standard itself, by very deliberate design, is so small that it lacks many important features you would expect from a programming language. For example it doesn't provide you with a standard hash table, sometimes called a dictionary data structure, this feature is only provided in the SRFI standards. Even pattern matching an important feature in any Lisp-like language, is not provided at all, not in the R7RS or the SRFIs (the only two pattern matching SRFIs, numbers 200 and 204, have been withdrawn). The purpose of the R7RS Small standard, at least from my point of view as an outsider looking in on the standardization process, is that Scheme is for building other programming languages, and the R7RS Large language is one of the languages which will build upon R7RS Small.

Learning the R7RS Small language standard is fun, even useful if you are interested in programming language theory, but you can forget about solving practical problems like CSV or JSON parsing, or collating files across directories to generate a static website. You must use implementation-specific libraries to do those things, and that kind-of locks you in to whatever Scheme you choose from the very beginning. This makes your choice of Scheme an important decision, because you will probably be stuck with it for a while, unless you don't mind rewriting much of your old code when you want to switch.

So before I was able to start writing code, the first question I had to answer for myself was, "what are the standards?" And since the SRFI process standardizes practical software engineering features like dictionary data structures and JSON parsing, another question to answer is, "which Scheme provides the largest number of SRFI implementations?"

Ideally I wanted to learn the latest standards, this means R7RS, and this also narrows-down the number of Scheme implementations a lot. Many Schemes are stuck on R5RS. A few are still on R6RS including some really good compilers like Chez Scheme. That is a shame, I would have liked using those. So I had now narrowed it down to about nine choices:

• Chez (R6RS only 🙁)

• Gauche

• Chibi

• Gambit

• Chicken

• Cyclone

• Stklos

• Larceny

• Guile

• MIT/GNU

I had a specific set of features in mind. My favorite language is Haskell, and so I wanted a Scheme compiler that had a set of features most similar to the Glasgow Haskell Compiler (GHC):

1. compiles to efficient binary machine code

2. does not use C as an intermediate compilation step

3. good foreign function interface (FFI) so I can use Gtk or SDL

4. has a relatively good package ecosystem

That eliminates Gauche and Chibi as they are interpreters. Chicken, Gambit, and Cyclone all compile to C as an intermediate step, so those are out. Stklos is a compiler, but it compiles to bytecode, not to machine code, so that was out too. That leaves Larceny, MIT Scheme, and Guile.

Larceny relies on the .NET runtime. So it probably works well on Windows and Linux (via the Mono runtime), which is nice. But then this is the same sort of problem with depending on C as an intermediate language, it requires you to install quite a large compiler toolchain to make it work. So Larceny is out. (Also, it seems Larceny hasn't been updated since 2017, perhaps it is abandoned?)

MIT Scheme seems to have a lot of users as the oldest Scheme implementation, and it does provide some support for an FFI, but it is nowhere near as good as Guile's FFI.

On top of that, Guile has a thriving community of users around it. It is the core of the Guix operating system, it is very widely used and deployed, you could call Guile an "industrial strength" Scheme compiler, like Chez Scheme, but implements the R7RS standard. And there are many, many external libraries which you can use with Guile and which you can easily install with the Guix package manager.

Guile is the one that best satisfies all of my conditions. But now that I have chosen Guile to implement my R7RS standard-compliant Scheme application, there is a new set of problems awaiting me which I could not have forseen a few years ago when I knew very little about Scheme.

MIT Scheme would have been an excellent choice. It is apparently based on the original Scheme implementation by Sussman and Steele, and (someone correct me if I am wrong about this) it apparently still includes some of the features that Abelson added for teaching the computer science courses at MIT, the video tapes of which have been digitized and are available online. I found out much later on that MIT/GNU also implements more of the modern SRFIs, and follows the R7RS standard much more closely than Guile does.

But MIT Scheme also has a few idiosyncrasies that initially made me not like it as much as Guile. Firstly, there no clear way to invoke the compiler from the CLI outside of the Scheme REPL, you must invoke the REPL, then use the cf command to compile a file. MIT Scheme philosophy seems to me to be that Scheme is first and foremost for educational purposes (again, please correct me if I am wrong), and it should teach the principles of other Lisp-like programming languages, which means that you should use it from within an interactive programming environment like Emacs or Edwin. So the maintainers seemingly do not consider the use case that MIT Scheme could be used in a larger application build process, for example, to be invoked by GNU Make, whereas Guile does consider that use case.

Also the MIT Scheme REPL is a little confusing in that you can't use the import statement in the REPL, and this is very much unlike any other REPL I have ever used. The import statement can only be used in program files, whereas in the REPL you are supposed to use the load command to import library code. Also, switching environments, or running debugging commands, requires you to enter full commands (in parentheses), whereas Guile lets you enter REPL directives with the comma prefix. I presume this is again because MIT Scheme is intended to be used in an interactive programming environment like Emacs or Edwin, it is as if they are saying, "you shouldn't really be entering commands into the REPL by hand anyways, let the editor do all that for you." Guile, on the other hand, seems to consider both the interactive programming and REPL-only use cases.

A few years later, I have new-found respect for the MIT Scheme compiler, and I might even recommend it to people who are interested in learning Scheme, especially if you are an Emacs user. Probably it's greatest strength is that most of the code you write for MIT Scheme can relatively easily be ported to other Scheme implementations — at least, that has been my personal experience with it.

Let me just get the most annoying problem out of the way first: the hash-colon #:keyword syntax is not standard, yet this syntax is used throughout all of Guile's libraries and even a few of it's standards-compliant APIs like the SRFI-69 implementation of make-hash-table. If you want your Scheme code to be portable, do not even think of using #:keywords. Some Scheme implementations, like Gauche or Chez will happily read #:keywords if they are in quote forms, although they treat these as ordinary interned symbols rather than a separate uninterned data type. Other Scheme implementations, like MIT Scheme or Chibi, raise a parser exception as soon as one is seen.

Could one simply modify the Scheme read command to allow keywords? Sure, probably, for many Scheme implementations this would be possible. But that still doesn't make it portable, after all you would have to update read procedure everywhere you want to run your code. It is easier to just not use keywords.

In my experience as a software engineer, hash tables are one of the most important data structures in modern software engineering. Python and JavaScript both, by necessity, provide some of the world's most efficient implementations of this data structure. Programmers nowadays think of this data structure as a given, it would be crazy for a language to not have it. R7RS Small does not provide hash tables, but the R7RS Large standard does. These are specified in a few documents, including:

SRFI-125:

the latest Scheme standard, which partially depends on the use of "Comparator" objects, defined in SRFI-128 which was drafted at the same time as SRFI-125.

SRFI-126:

not part of R7RS Large, and the API is quite different from SRFI-125, but it provides the R6RS standard implementation of hash tables for backward compatibility in R7RS.

SRFI-69:

the old Scheme standard hash table, not part of R7RS Large, but SRFI-125 was designed to be mostly backward compatible with SRFI-69.

SRFI-114:

the old Scheme standard for Comparator objects, superseded by SRFI-128.

SRFI-128:

the latest Scheme standard for Comparator objects, which provide an abstract interface to methods for equality comparison, ordering, and hashing that can be used by the hash table algorithm to store and lookup keys in the table.

SRFI-162:

an extension to SRFI-128 that provides Comparator objects for built-in data structures like strings, characters, case-insensitive strings and characters, lists, and real numbers.

At the time of this writing, Guile only provides SRFI-69, and SRFI-126 listed under (rnrs hashtables), so Guile provides none of the R7RS Large standard hash tables (and no comparators either). And this problem stands out in my mind as the largest source of consternation in writing my portable Scheme application using Guile, especially since my application depends so heavily on this data structure.

If I had know this was going to be an issue, I would have used SRFI-126 from the very beginning, since it backward compatible in other R7RS Scheme implementations, and is built-in to Guile. It would require me to go through the trouble of copying the SRFI-126 reference implementation code into my source code repository in order to ensure it is available wherever I want to compile my program (e.g. MIT Scheme which does not provide SRFI-126), but this extra effort would indeed make my program more portable.

Another annoyance with Guile, though this has less to do with portability, is that the Guile SRFI-69 implementation is a wrapper around Guile's own built-in hash table implementation. Ordinarily, this would be totally fine, except that Guile exports the built-in hash table data structure by default, not the SRFI-69 hash table, not even when you launch Guile with the "--r7rs" option enabled. The built-in hash table is a separate data type from the SRFI-69 data type that wraps around it. So if you are trying to work with SRFI-69 hash tables only, and you forget to import (srfi 69)>, your program starts failing with weird errors about the wrong type of hash table being used as an argument to a hash table procedure. This is apparently because Guile has historically provided it's own hash tables since even before SRFI-69 was standardized, and so Guile implements hash tables in this way to keep Guile backward compatible with itself.

Fewer recent SRFI implementations

The last problem I will mention is that the Guile have not had the time to port many of the more recent SRFI implementations. MIT Scheme is very conservative in which SRFIs it implements, but at least provides you with comparators and modern hash tables. By my reckoning, Stklos provides more SRFI implementations than any other Scheme, and it would be nice if Guile could do the same. It is one thing to have a thriving package ecosystem, which Guile does have, but it is less than ideal to need to install SRFIs into one's own project as a package dependency when it could be provided to you by the compiler, especially the SRFIs that are already ratified in the R7RS Large standard, and especially for such an important data structure.

Conclusions

The most important thing I learned is that if portability is a concern for you, and I believe it should be a concern for any code base you intend to release publicly, it is a good idea to install more than one Scheme implementation and try to get your code to run on as many of them as you can. Personally, although I develop my code primarily using Guile, I have started trying to run some of my test code on both MIT Scheme and Gauche. Each of these implementations provides a slightly different set of checks on the code, so this turns out to be a pretty good way to do linting on my code base.

I will certainly continue using Guile for my personal projects. Even with it's rough edges, it still seems to me to be the best choice for practical software engineering in Scheme. But I will recommend to anyone learning Scheme to consider starting with MIT Scheme, and then porting your code to Guile and other implementations if you ever feel like you are ready to publish the code you wrote. I think this might help beginners develop better programming habits as they won't inadvertently be relying on a Guile-specific features that may not exist at all in other Scheme implementations, not even as a 3rd-party package.

The fact that there are so many Scheme implementations does lead to fragmentation and frustration, but it is also one of the greatest strengths of the Scheme language. Most languages nowadays are developed by major corporations with dozens, maybe hundreds of engineers working on them, and there is no chance any one person who is not directly employed in the development of the programming language can have a complete picture of what the language is doing with your code when it is run. From the programmer's perspective, it may as well run on magic.

Scheme does not run on magic, there is a mechanism to it which anyone can understand, and that mechanism has been decided democratically by experts in the field of programming languages and written into the document of the R7RS standard. The language standard is small and accessible such that anyone can program their own Scheme implementation themselves, and this is important in giving individuals control over their own computers and software. In spite of the fragmentation and frustration, I believe this makes the Scheme standard important to learn and to use, because it gives anyone the ability to control their computer more directly, and to understand it's inner workings.