How to patch the Guix OS package repo
Andrew Tropin (@abcdw@fostodon.org, @abcdw@diode.zone) very generously reached out to me recently about doing some pair-programming to teach me more about how to use the Guix package manager and Guix OS. I recommended we do a relatively simple task of defining a new package for inclusion into the Guix OS distribution.
For this project, I choose the STklos Scheme compiler because it had not been ported to Guix OS yet, and I had recently been playing around with it. It built very easily on my Ubuntu computer, and I thought it would be fairly easy to build it in Guix OS as well. And except for one minor issue, which took us only a few hours working together to resolve, it turned out to be as easy as we had hoped.
In this article, I would like to explain the steps we (or anyone) would take to try define a new package for Guix OS, and then explain these steps in more detail. These steps include:
Cloning the head of the main branch of main Guix Packages channel locally and building it.
Finding a similar package to use as a template and morphing it into one that builds the new package.
Working out bugs in the build process that almost certain to occur when porting a software package to a system that the authors never intended to support.
Writing these build steps as actual Scheme code in the package declaration, building it as part of the main Guix Packages channel, and linting it.
Submitting a patch to the Guix Packages maintainers
A quick overview of how Guix works
I wrote more about Guix in a separate article, please read it if you would like to know more.
Without going deeply into the theory of how Guix works, basically every package defined with package
is just a Scheme procedure that you can run by calling it, and this executes the steps to build and install a software package. The build occurs in a temporary directory. The steps to build include inheriting environment variables from other packages that make up the operating system, downloading the necessary code, and then running the steps in a shell that you would ordinarily do to build it, with ./configure
, make
, and so on.
When execution completes, the files built in the temporary directory can be stored into the local package cache in /gnu/store
. Then, by linking these built objects to "profiles" (which involves literally symbolic linking files into "profile" directory trees placed where they need to be used) they become usable as software by the rest of the system. Installing things into the store involves hashing every object to create a unique identifier for that particular build which can be addressed as a dependency of other software builds.
I do not use actually Guix OS on my computer, I use Ubuntu. However for software developement I use Guix as a package manager. And in order to use the Guix Packages, you must install the core of the Guix OS operating system because all packages depend (directly or indirectly) on the GCC compiler runtime used to build the Linux kernel for Guix OS. You just don't need to install the Guix OS Linux kernel unless you want to run the OS on your computer.
The process of developing a patch
Clone the Guix sources from Savannah
Of course, start by Git-cloning the code that you actually want to change:
- Git URL: https://git.savannah.gnu.org/git/guix.git
- You can browse this code online at the GNU Savannah website.
Run the bootstrap script, the configure and make
WARNING: The make process took about 3 hours
guix shell -D guix -- ./bootstrap;
guix shell -D guix -- ./configure --localstatedir='/var';
guix shell -D guix -- make -k -j4; # It failed to build Info docs, safe to ignore
This will bytecode compile all packages in the standard Guix Packages channel. According to Andrew, one reason for this is that compiling the entire program allows for indexing used by guix package --search
.
The make -k -j4
command took about 3 hours to run on my 1.80GHz Intel Core i7-8550U CPU laptop with 20GB of DDR4 RAM. This is just to build the package definitions in the main Guix channel, not any of the actual software packages. But this step is necessary if you want to contribute to the main Guix Packages channel.
Write the package definition
To build the package:
guix shell -D guix -- ./pre-inst-env guix build stklos
To lint the package:
guix shell -D guix -- ./pre-inst-env guix lint stklos
You can use the Scheme REPL to make these build-and-fix-errors cycles go faster:
guix shell -D guix -- ./pre-inst-env guix repl
Once your terminal is using the Guile REPL, tell it to use the module containing the package your are creating. For our STklos package, we decided to place it in the (gnu packages scheme)
module, along side other similar packages.
,m (gnu packages scheme)
Here, the ,module
or ,m
directive tells the REPL to set (gnu packages scheme)
as the current module in which REPL expressions are evaluated. This gives you access to all public symbols in that module, which in the Guix packages, are all package definitions.
Now you can try to build within the REPL. To test out whether your Guix package definition actually builds, simply run it by invoking its name as a procedure call:
(stklos)
Running the (stklos)
procedure actually executes the function that builds the package we are defining, which is the package to build the STklos scheme compiler in Guix OS. It is almost the same as running a shell script, except that our CLI interpreter is Guile instead of Bash.
If you would prefer, you can build from a Bash REPL instead of a Guile REPL. As long as Guix can find the '(gnu packages scheme)' is in the Guile %load-path
, then you can run the below command to build instead, you can build your changes to the package definitions like so:
guix build -e '(@ (gnu packages scheme) stklos)'
The -e
flag is used to run Scheme code from the CLI. The (@ ...)
syntax is Guile's way of evaluating code in a module, here the (gnu packages scheme)
is the module, and in the REPL we can simply step into it with the ,module
directive (or ,m
for short). Once inside of the module, you can use all the publicly defined ("exported") symbols defined within it. In the above example, I use (stklos)
, which is a package, and therefore a procedure that I can execute to run package installation.
Submitting the patch
This might surprise some people nowadays, but:
- you do not need commit access to any git repositories,
- you do not need to be a member of a forge site like GitHub.
The process is very old-school: just email the patch to the maintainers, they will review it and commit it for you.
After taking care of the issues raised by the linter, I used Git to generate a patch file, I commit the changes to my local Git:
git add 'gnu/packages/scheme.scm'; git commit -m 'Add stklos.';
Generate the patch file:
git format-patch 'master~..master' --
…that generates the patch file
0001-gnu-Add-stklos.patch
.Emacs users using Magit, you can run this command from the
magit-status
buffer by typingW c c
, and then when prompted, selecting the localmaster
that contain the commits you want to submit.E-mail the patch file to guix-patches@gnu.org.
If you have your
git config
correctly setup with your E-mail address, the patch will contain the correct E-mail headers and everything. In the patch file buffer, just runM-x mail-send
and it will run your preferredsendmail
client.But if you don't want to use Emacs to send E-mail, you can just copy the "
Subject:
" line from the.patch
file and use it to compse an E-mail in Gmail or whatever app you use for that:Subject: [PATCH] gnu: Add stklos.
Notice that the subject heading is generated from the commit message.
Then attach the "
.patch
file to the E-mail message, and write more detailed description of the patch if necessary. A few kind words for the package maintainers wouldn't hurt either.Send the E-mail, you will get an automated response with an hour or two. The package maintainers may get back to you later on requesting additional changes to your patch.
Details about this particular package build
Find a similar project and copy/paste it (we used "scheme48")
For defining the STKlos package, we started with scheme48
package as our template:
(define-public scheme48
(package
(name "scheme48")
(version "1.9.2")
(source (origin
(method url-fetch)
(uri (string-append "https://s48.org/" version
"/scheme48-" version ".tgz"))
(sha256
(base32
"1x4xfm3lyz2piqcw1h01vbs1iq89zq7wrsfjgh3fxnlm1slj2jcw"))
(patches (search-patches "scheme48-tests.patch"))))
(build-system gnu-build-system)
(home-page "https://s48.org/")
(synopsis "Scheme implementation using a bytecode interpreter")
(description
"Scheme 48 is an implementation of Scheme based on a byte-code
interpreter and is designed to be used as a testbed for experiments in
implementation techniques and as an expository tool.")
;; Most files are BSD-3; see COPYING for the few exceptions.
(license bsd-3)))
The changes we made to scheme48
to define the stklos
package
@@ -1,22 +1,43 @@
- (define-public scheme48
+ (define-public stklos
(package
- (name "scheme48")
- (version "1.9.2")
+ (name "stklos")
+ (version "1.70")
(source (origin
- (method url-fetch)
- (uri (string-append "https://s48.org/" version
- "/scheme48-" version ".tgz"))
- (sha256
- (base32
- "1x4xfm3lyz2piqcw1h01vbs1iq89zq7wrsfjgh3fxnlm1slj2jcw"))
- (patches (search-patches "scheme48-tests.patch"))))
+ (method url-fetch)
+ (uri (string-append "https://stklos.net/download/stklos-"
+ version ".tar.gz"))
+ (sha256
+ (base32
+ "1iw3pgycjz3kz3jd1855v2ngf8ib2almpf8v058n1mkj1qd2b88m"))))
(build-system gnu-build-system)
- (home-page "https://s48.org/")
- (synopsis "Scheme implementation using a bytecode interpreter")
+ (arguments
+ (list
+ #:modules `((ice-9 ftw)
+ ,@%gnu-build-system-modules)
+ #:phases
+ #~(modify-phases %standard-phases
+ (add-before 'configure 'patch-sh-paths
+ (lambda* (#:key inputs #:allow-other-keys)
+ (let ((bash-bin (search-input-file inputs "/bin/bash")))
+ (substitute* "configure"
+ (("/bin/sh") bash-bin)))))
+ (add-after 'configure 'patch-rm-paths
+ (lambda* (#:key inputs #:allow-other-keys)
+ (let ((rm-bin (search-input-file inputs "/bin/rm")))
+ (ftw "."
+ (lambda (filename stat-info f)
+ (when (and
+ (equal? f 'regular)
+ (string=? (basename filename) "Makefile"))
+ (substitute* filename
+ (("/bin/rm") rm-bin)))
+ #t))))))))
+ (home-page "https://stklos.net")
+ (synopsis "R7RS Scheme with CLOS-like object system")
(description
- "Scheme 48 is an implementation of Scheme based on a byte-code
- interpreter and is designed to be used as a testbed for experiments in
- implementation techniques and as an expository tool.")
-
- ;; Most files are BSD-3; see COPYING for the few exceptions.
- (license bsd-3)))
+ "STklos is a free Scheme system mostly compliant with the languages
+ features defined in R7RS small. The aim of this implementation is to be fast
+ as well as light. The implementation is based on an ad-hoc Virtual
+ Machine. STklos can also be compiled as a library and embedded in an
+ application.")
+ (license gpl2)))
The resulting stklos
package
(define-public stklos
(package
(name "stklos")
(version "1.70")
(source (origin
(method url-fetch)
(uri (string-append "https://stklos.net/download/stklos-"
version ".tar.gz"))
(sha256
(base32
"1iw3pgycjz3kz3jd1855v2ngf8ib2almpf8v058n1mkj1qd2b88m"))))
(build-system gnu-build-system)
(arguments
(list
#:modules `((ice-9 ftw)
,@%gnu-build-system-modules)
#:phases
#~(modify-phases %standard-phases
(add-before 'configure 'patch-sh-paths
(lambda* (#:key inputs #:allow-other-keys)
(let ((bash-bin (search-input-file inputs "/bin/bash")))
(substitute* "configure"
(("/bin/sh") bash-bin)))))
(add-after 'configure 'patch-rm-paths
(lambda* (#:key inputs #:allow-other-keys)
(let ((rm-bin (search-input-file inputs "/bin/rm")))
(ftw "."
(lambda (filename stat-info f)
(when (and
(equal? f 'regular)
(string=? (basename filename) "Makefile"))
(substitute* filename
(("/bin/rm") rm-bin)))
#t))))))))
(home-page "https://stklos.net")
(synopsis "R7RS Scheme with CLOS-like object system")
(description
"STklos is a free Scheme system mostly compliant with the languages
features defined in R7RS small. The aim of this implementation is to be fast
as well as light. The implementation is based on an ad-hoc Virtual
Machine. STklos can also be compiled as a library and embedded in an
application.")
(license gpl2)))
The issue with the STklos build system
The authors of STklos are in the habit of writing their scripts such that commands are called by their full path. For example, instead of running the cp
command, they run the /bin/cp
command.
There are reasons to do this. For example if you are worried that you may have accidentally changed the PATH
environment variable to something it should not be, then the behavior of commands like sh
, cp
, or rm
may not work as you expect and cause your program to fail. By writing /bin/sh
or /bin/rm
instead, you can better ensure these commands are behaving as you expect, which can be good when writing software that may run on a very wide variety of computer systems. PATH
is an ephemeral variable, but /bin
and /usr/bin
are fairly static across many systems.
However Guix is very precisely controlling the PATH
environment variable, and by writing /bin/sh
instead of just sh
, your program is calling the wrong version of the sh
command; changing it to one that is not useful to Guix, since it wants to ensure the correctness of the software it builds.
And unfortunately, the STklos authors put this habit into practice throughout all of their Makefiles and even some of the pre-make scripts called by Autoconf.
So to solve this problem, we had to resort to a fairly ugly hack: run "find and replace" on all the build scripts, and replace "/bin/
" or "/usr/bin/
" with nothing. Fortunately, Guile has a very simple command to do this: substitute*
. So we run the file-tree walk (FTW) procedure to search for all files, and substitute*
to find/replace the desired text. This hack is executed just before the "configure" phase of the build, and just after the "configure" phase since "configure" outputs some Makefiles which also contain these mistakes.
#:phases
#~(modify-phases %standard-phases
(add-before 'configure 'patch-sh-paths
(lambda* (#:key inputs #:allow-other-keys)
(let ((bash-bin (search-input-file inputs "/bin/bash")))
(substitute* "configure"
(("/bin/sh") bash-bin)))))
(add-after 'configure 'patch-rm-paths
(lambda* (#:key inputs #:allow-other-keys)
(let ((rm-bin (search-input-file inputs "/bin/rm")))
(ftw "."
(lambda (filename stat-info f)
(when (and
(equal? f 'regular)
(string=? (basename filename) "Makefile"))
(substitute* filename
(("/bin/rm") rm-bin)))
#t))))))))
All the rest of our "stklos
" package definition is bog-standard GNU build system.