How to patch the Guix OS package repo

Home About Codeberg

Andrew Tropin (@abcdw@fostodon.org, @abcdw@diode.zone) very generously reached out to me recently about doing some pair-programming to teach me more about how to use the Guix package manager and Guix OS. I recommended we do a relatively simple task of defining a new package for inclusion into the Guix OS distribution.

For this project, I choose the STklos Scheme compiler because it had not been ported to Guix OS yet, and I had recently been playing around with it. It built very easily on my Ubuntu computer, and I thought it would be fairly easy to build it in Guix OS as well. And except for one minor issue, which took us only a few hours working together to resolve, it turned out to be as easy as we had hoped.

In this article, I would like to explain the steps we (or anyone) would take to try define a new package for Guix OS, and then explain these steps in more detail. These steps include:

A quick overview of how Guix works

I wrote more about Guix in a separate article, please read it if you would like to know more.

Without going deeply into the theory of how Guix works, basically every package defined with package is just a Scheme procedure that you can run by calling it, and this executes the steps to build and install a software package. The build occurs in a temporary directory. The steps to build include inheriting environment variables from other packages that make up the operating system, downloading the necessary code, and then running the steps in a shell that you would ordinarily do to build it, with ./configure, make, and so on.

When execution completes, the files built in the temporary directory can be stored into the local package cache in /gnu/store. Then, by linking these built objects to "profiles" (which involves literally symbolic linking files into "profile" directory trees placed where they need to be used) they become usable as software by the rest of the system. Installing things into the store involves hashing every object to create a unique identifier for that particular build which can be addressed as a dependency of other software builds.

I do not use actually Guix OS on my computer, I use Ubuntu. However for software developement I use Guix as a package manager. And in order to use the Guix Packages, you must install the core of the Guix OS operating system because all packages depend (directly or indirectly) on the GCC compiler runtime used to build the Linux kernel for Guix OS. You just don't need to install the Guix OS Linux kernel unless you want to run the OS on your computer.

The process of developing a patch

Clone the Guix sources from Savannah

Of course, start by Git-cloning the code that you actually want to change:

Run the bootstrap script, the configure and make

WARNING: The make process took about 3 hours

guix shell -D guix -- ./bootstrap;
guix shell -D guix -- ./configure --localstatedir='/var';
guix shell -D guix -- make -k -j4; # It failed to build Info docs, safe to ignore

This will bytecode compile all packages in the standard Guix Packages channel. According to Andrew, one reason for this is that compiling the entire program allows for indexing used by guix package --search.

The make -k -j4 command took about 3 hours to run on my 1.80GHz Intel Core i7-8550U CPU laptop with 20GB of DDR4 RAM. This is just to build the package definitions in the main Guix channel, not any of the actual software packages. But this step is necessary if you want to contribute to the main Guix Packages channel.

Write the package definition

You can use the Scheme REPL to make these build-and-fix-errors cycles go faster:

guix shell -D guix -- ./pre-inst-env guix repl

Once your terminal is using the Guile REPL, tell it to use the module containing the package your are creating. For our STklos package, we decided to place it in the (gnu packages scheme) module, along side other similar packages.

,m (gnu packages scheme)

Here, the ,module or ,m directive tells the REPL to set (gnu packages scheme) as the current module in which REPL expressions are evaluated. This gives you access to all public symbols in that module, which in the Guix packages, are all package definitions.

Now you can try to build within the REPL. To test out whether your Guix package definition actually builds, simply run it by invoking its name as a procedure call:

(stklos)

Running the (stklos) procedure actually executes the function that builds the package we are defining, which is the package to build the STklos scheme compiler in Guix OS. It is almost the same as running a shell script, except that our CLI interpreter is Guile instead of Bash.

If you would prefer, you can build from a Bash REPL instead of a Guile REPL. As long as Guix can find the '(gnu packages scheme)' is in the Guile %load-path, then you can run the below command to build instead, you can build your changes to the package definitions like so:

guix build -e '(@ (gnu packages scheme) stklos)'

The -e flag is used to run Scheme code from the CLI. The (@ ...) syntax is Guile's way of evaluating code in a module, here the (gnu packages scheme) is the module, and in the REPL we can simply step into it with the ,module directive (or ,m for short). Once inside of the module, you can use all the publicly defined ("exported") symbols defined within it. In the above example, I use (stklos), which is a package, and therefore a procedure that I can execute to run package installation.

Submitting the patch

This might surprise some people nowadays, but:

The process is very old-school: just email the patch to the maintainers, they will review it and commit it for you.

  1. After taking care of the issues raised by the linter, I used Git to generate a patch file, I commit the changes to my local Git:

    git add 'gnu/packages/scheme.scm';
    git commit -m 'Add stklos.';
    
  2. Generate the patch file:

    git format-patch 'master~..master' --

    …that generates the patch file 0001-gnu-Add-stklos.patch.

    Emacs users using Magit, you can run this command from the magit-status buffer by typing W c c, and then when prompted, selecting the local master that contain the commits you want to submit.

  3. E-mail the patch file to guix-patches@gnu.org.

    If you have your git config correctly setup with your E-mail address, the patch will contain the correct E-mail headers and everything. In the patch file buffer, just run M-x mail-send and it will run your preferred sendmail client.

    But if you don't want to use Emacs to send E-mail, you can just copy the "Subject:" line from the .patch file and use it to compse an E-mail in Gmail or whatever app you use for that:

    Subject: [PATCH] gnu: Add stklos.

    Notice that the subject heading is generated from the commit message.

    Then attach the ".patch file to the E-mail message, and write more detailed description of the patch if necessary. A few kind words for the package maintainers wouldn't hurt either.

  4. Send the E-mail, you will get an automated response with an hour or two. The package maintainers may get back to you later on requesting additional changes to your patch.

Details about this particular package build

Find a similar project and copy/paste it (we used "scheme48")

For defining the STKlos package, we started with scheme48 package as our template:

(define-public scheme48
  (package
    (name "scheme48")
    (version "1.9.2")
    (source (origin
             (method url-fetch)
             (uri (string-append "https://s48.org/" version
                                 "/scheme48-" version ".tgz"))
             (sha256
              (base32
               "1x4xfm3lyz2piqcw1h01vbs1iq89zq7wrsfjgh3fxnlm1slj2jcw"))
             (patches (search-patches "scheme48-tests.patch"))))
    (build-system gnu-build-system)
    (home-page "https://s48.org/")
    (synopsis "Scheme implementation using a bytecode interpreter")
    (description
     "Scheme 48 is an implementation of Scheme based on a byte-code
interpreter and is designed to be used as a testbed for experiments in
implementation techniques and as an expository tool.")

    ;; Most files are BSD-3; see COPYING for the few exceptions.
    (license bsd-3)))

The changes we made to scheme48 to define the stklos package

@@ -1,22 +1,43 @@
-    (define-public scheme48
+    (define-public stklos
       (package
-        (name "scheme48")
-        (version "1.9.2")
+        (name "stklos")
+        (version "1.70")
         (source (origin
-                 (method url-fetch)
-                 (uri (string-append "https://s48.org/" version
-                                     "/scheme48-" version ".tgz"))
-                 (sha256
-                  (base32
-                   "1x4xfm3lyz2piqcw1h01vbs1iq89zq7wrsfjgh3fxnlm1slj2jcw"))
-                 (patches (search-patches "scheme48-tests.patch"))))
+                  (method url-fetch)
+                  (uri (string-append "https://stklos.net/download/stklos-"
+                                      version ".tar.gz"))
+                  (sha256
+                   (base32
+                    "1iw3pgycjz3kz3jd1855v2ngf8ib2almpf8v058n1mkj1qd2b88m"))))
         (build-system gnu-build-system)
-        (home-page "https://s48.org/")
-        (synopsis "Scheme implementation using a bytecode interpreter")
+        (arguments
+         (list
+          #:modules `((ice-9 ftw)
+                      ,@%gnu-build-system-modules)
+          #:phases
+          #~(modify-phases %standard-phases
+              (add-before 'configure 'patch-sh-paths
+                (lambda* (#:key inputs #:allow-other-keys)
+                  (let ((bash-bin (search-input-file inputs "/bin/bash")))
+                    (substitute* "configure"
+                      (("/bin/sh") bash-bin)))))
+              (add-after 'configure 'patch-rm-paths
+                (lambda* (#:key inputs #:allow-other-keys)
+                  (let ((rm-bin (search-input-file inputs "/bin/rm")))
+                    (ftw "."
+                         (lambda (filename stat-info f)
+                           (when (and
+                                  (equal? f 'regular)
+                                  (string=? (basename filename) "Makefile"))
+                             (substitute* filename
+                               (("/bin/rm") rm-bin)))
+                           #t))))))))
+        (home-page "https://stklos.net")
+        (synopsis "R7RS Scheme with CLOS-like object system")
         (description
-         "Scheme 48 is an implementation of Scheme based on a byte-code
-    interpreter and is designed to be used as a testbed for experiments in
-    implementation techniques and as an expository tool.")
-
-        ;; Most files are BSD-3; see COPYING for the few exceptions.
-        (license bsd-3)))
+         "STklos is a free Scheme system mostly compliant with the languages
+    features defined in R7RS small.  The aim of this implementation is to be fast
+    as well as light.  The implementation is based on an ad-hoc Virtual
+    Machine.  STklos can also be compiled as a library and embedded in an
+    application.")
+        (license gpl2)))

The resulting stklos package

(define-public stklos
  (package
    (name "stklos")
    (version "1.70")
    (source (origin
              (method url-fetch)
              (uri (string-append "https://stklos.net/download/stklos-"
                                  version ".tar.gz"))
              (sha256
               (base32
                "1iw3pgycjz3kz3jd1855v2ngf8ib2almpf8v058n1mkj1qd2b88m"))))
    (build-system gnu-build-system)
    (arguments
     (list
      #:modules `((ice-9 ftw)
                  ,@%gnu-build-system-modules)
      #:phases
      #~(modify-phases %standard-phases
          (add-before 'configure 'patch-sh-paths
            (lambda* (#:key inputs #:allow-other-keys)
              (let ((bash-bin (search-input-file inputs "/bin/bash")))
                (substitute* "configure"
                  (("/bin/sh") bash-bin)))))
          (add-after 'configure 'patch-rm-paths
            (lambda* (#:key inputs #:allow-other-keys)
              (let ((rm-bin (search-input-file inputs "/bin/rm")))
                (ftw "."
                     (lambda (filename stat-info f)
                       (when (and
                              (equal? f 'regular)
                              (string=? (basename filename) "Makefile"))
                         (substitute* filename
                           (("/bin/rm") rm-bin)))
                       #t))))))))
    (home-page "https://stklos.net")
    (synopsis "R7RS Scheme with CLOS-like object system")
    (description
     "STklos is a free Scheme system mostly compliant with the languages
features defined in R7RS small.  The aim of this implementation is to be fast
as well as light.  The implementation is based on an ad-hoc Virtual
Machine.  STklos can also be compiled as a library and embedded in an
application.")
    (license gpl2)))

The issue with the STklos build system

The authors of STklos are in the habit of writing their scripts such that commands are called by their full path. For example, instead of running the cp command, they run the /bin/cp command.

There are reasons to do this. For example if you are worried that you may have accidentally changed the PATH environment variable to something it should not be, then the behavior of commands like sh, cp, or rm may not work as you expect and cause your program to fail. By writing /bin/sh or /bin/rm instead, you can better ensure these commands are behaving as you expect, which can be good when writing software that may run on a very wide variety of computer systems. PATH is an ephemeral variable, but /bin and /usr/bin are fairly static across many systems.

However Guix is very precisely controlling the PATH environment variable, and by writing /bin/sh instead of just sh, your program is calling the wrong version of the sh command; changing it to one that is not useful to Guix, since it wants to ensure the correctness of the software it builds.

And unfortunately, the STklos authors put this habit into practice throughout all of their Makefiles and even some of the pre-make scripts called by Autoconf.

So to solve this problem, we had to resort to a fairly ugly hack: run "find and replace" on all the build scripts, and replace "/bin/" or "/usr/bin/" with nothing. Fortunately, Guile has a very simple command to do this: substitute*. So we run the file-tree walk (FTW) procedure to search for all files, and substitute* to find/replace the desired text. This hack is executed just before the "configure" phase of the build, and just after the "configure" phase since "configure" outputs some Makefiles which also contain these mistakes.

#:phases
#~(modify-phases %standard-phases
    (add-before 'configure 'patch-sh-paths
      (lambda* (#:key inputs #:allow-other-keys)
        (let ((bash-bin (search-input-file inputs "/bin/bash")))
          (substitute* "configure"
            (("/bin/sh") bash-bin)))))
    (add-after 'configure 'patch-rm-paths
      (lambda* (#:key inputs #:allow-other-keys)
        (let ((rm-bin (search-input-file inputs "/bin/rm")))
          (ftw "."
               (lambda (filename stat-info f)
                 (when (and
                        (equal? f 'regular)
                        (string=? (basename filename) "Makefile"))
                   (substitute* filename
                     (("/bin/rm") rm-bin)))
                 #t))))))))

All the rest of our "stklos" package definition is bog-standard GNU build system.

Conclusion