https://guix.gnu.org/feeds/blog/scheme-api.atomGNU Guix — Blog — Scheme APIfeed author nameGNU Guixhttps://guix.gnu.org/static/base/img/icon.png2020-08-14T21:45:02Zhttps://guix.gnu.org/blog/2020/securing-updates/Securing updatesLudovic Courtès2020-07-01T17:40:00Z2020-07-01T17:40:00Z Software deployment tools like Guix are in a key position when it comes
to securing the “software supply chain”—taking source code fresh from
repositories and providing users with ready-to-use binaries. We have
been paying attention to several aspects of this problem in Guix:
authentication of pre-built
binaries ,
reproducible
builds ,
bootstrapping , and
security
updates . A couple of weeks ago, we addressed the elephant in the room:
authentication of Guix code itself by guix pull ,
the tool that updates Guix and its package collection. This article
looks at what we set out to address, how we achieved it,…<p>Software deployment tools like Guix are in a key position when it comes
to securing the “software supply chain”—taking source code fresh from
repositories and providing users with ready-to-use binaries. We have
been paying attention to several aspects of this problem in Guix:
<a href="https://guix.gnu.org/manual/en/html_node/Substitute-Authentication.html">authentication of pre-built
binaries</a>,
<a href="https://guix.gnu.org/blog/tags/reproducible-builds/">reproducible
builds</a>,
<a href="https://guix.gnu.org/blog/tags/bootstrapping/">bootstrapping</a>, and
<a href="https://guix.gnu.org/blog/tags/security-updates/">security
updates</a>.</p><p>A couple of weeks ago, we addressed the elephant in the room:
authentication of Guix code itself by <a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix-pull.html"><code>guix pull</code></a>,
the tool that updates Guix and its package collection. This article
looks at what we set out to address, how we achieved it, and how it
compares to existing work in this area.</p><h1>What updates should be protected against</h1><p>The problem of securing distro updates is often viewed through the lens
of binary distributions such as Debian, where the main asset to be
protected are binaries themselves. The functional deployment model that
Guix and Nix implement is very different: conceptually, Guix is a
<em>source distribution</em>, like Gentoo if you will.</p><p>Pre-built binaries are of course available and very useful, but they’re
optional; we call them
<a href="https://guix.gnu.org/manual/en/html_node/Substitutes.html"><em>substitutes</em></a>
because they’re just that: substitutes for local builds. When you do
choose to accept substitutes, they must be signed by one of the keys you
authorized (this has been the case since <a href="https://guix.gnu.org/blog/2014/gnu-guix-06-released/">version 0.6 in
2014</a>).</p><p>Guix consists of source code for the tools as well as all the <a href="https://guix.gnu.org/manual/en/html_node/Defining-Packages.html">package
definitions</a>—the
<em>distro</em>. When users run <code>guix pull</code>, what happens behind the scene is
equivalent to <code>git clone</code> or <code>git pull</code>. There are many ways this can
go wrong. An attacker can trick the user into pulling code from an
alternate repository that contains malicious code or definitions for
backdoored packages. This is made more difficult by the fact that code
is fetched over HTTPS from
<a href="https://git.savannah.gnu.org/cgit/guix.git/">Savannah</a> by default. If
Savannah is compromised (<a href="https://www.fsf.org/blogs/sysadmin/savannah-and-www.gnu.org-downtime">as happened in
2010</a>),
an attacker can push code to the Guix repository, which everyone would
pull. The change might even go unnoticed and remain in the repository
forever. An attacker with access to Savannah can also reset the main
branch to an earlier revision, leading users to install outdated
software with known vulnerabilities—a <em>downgrade attack</em>. These are
the kind of attacks we want to protect against.</p><h1>Authenticating Git checkouts</h1><p>If we take a step back, the problem we’re trying to solve is not
specific to Guix and to software deployment tools: it’s about
<em>authenticating Git checkouts</em>. By that, we mean that when <code>guix pull</code>
obtains code from Git, it should be able to tell that all the commits it
fetched were pushed by authorized developers of the project. We’re
really looking at individual commits, not tags, because users can choose
to pull arbitrary points in the commit history of Guix and third-party
<a href="https://guix.gnu.org/manual/en/html_node/Channels.html">channels</a>.</p><p>Checkout authentication requires <a href="https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work">cryptographically signed
commits</a>.
By signing a commit, a Guix developer asserts that they are the one who
made the commit; they may be its author, or they may be the person who
applied somebody else’s changes after review. It also requires a notion
of authorization: we don’t simply want commits to have a valid
signature, we want them to be signed by an authorized key. The set of
authorized keys changes over time as people join and leave the project.</p><p>To implement that, we came up with the following mechanism and rule:</p><ol><li>The repository contains a <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/.guix-authorizations"><code>.guix-authorizations</code>
file</a>
that lists the OpenPGP key fingerprints of authorized committers.</li><li>A commit is considered authentic if and only if it is signed by one
of the keys listed in the <code>.guix-authorizations</code> file of each of
its parents. This is the <em>authorization invariant</em>.</li></ol><p>(Remember that Git commits form a directed acyclic graph (DAG) where
each commit can have zero or more parents; merge commits have two parent
commits, for instance. Do not miss <a href="https://eagain.net/articles/git-for-computer-scientists/"><em>Git for Computer
Scientists</em></a>
for a pedagogical overview!)</p><p>Let’s take an example to illustrate. In the figure below, each box is a
commit, and each arrow is a parent relationship:</p><p><img src="https://guix.gnu.org/static/blog/img/commit-graph.svg" alt="Example commit graph." /></p><p>This figure shows two lines of development: the orange line may be the
main development branch, while the purple line may correspond to a
feature branch that was eventually merged in commit <em>F</em>. <em>F</em> is a merge
commit, so it has two parents: <em>D</em> and <em>E</em>.</p><p>Labels next to boxes show who’s in <code>.guix-authorizations</code>: for commit A,
only Alice is an authorized committer, and for all the other commits,
both Bob and Alice are authorized committers. For each commit, we see
that the authorization invariant holds; for example:</p><ul><li>commit <em>B</em> was made by Alice, who was the only authorized committer
in its parent, commit <em>A</em>;</li><li>commit <em>C</em> was made by Bob, who was among the authorized committers
as of commit <em>B</em>;</li><li>commit <em>F</em> was made by Alice, who was among the authorized
committers of both parents, commits <em>D</em> and <em>E</em>.</li></ul><p>The authorization invariant has the nice property that it’s simple to
state, and it’s simple to check and enforce. This is what <code>guix pull</code>
implements. If your current Guix, <a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix-describe.html">as returned by <code>guix describe</code></a>,
is at commit <em>A</em> and you want to pull to commit <em>F</em>, <code>guix pull</code>
traverses all these commits and checks the authorization invariant.</p><p>Once a commit has been authenticated, all the commits in its <a href="https://en.wikipedia.org/wiki/Transitive_closure#In_graph_theory">transitive
closure</a>
are known to be already authenticated. <code>guix pull</code> keeps a local cache
of the commits it has previously authenticated, which allows it to
traverse only new commits. For instance, if you’re at commit <em>F</em> and
later update to a descendant of <em>F</em>, authentication starts at <em>F</em>.</p><p>Since <code>.guix-authorizations</code> is a regular file under version control,
granting or revoking commit authorization does not require special
support. In the example above, commit <em>B</em> is an authorized commit by
Alice that adds Bob’s key to <code>.guix-authorizations</code>. Revocation is
similar: any authorized committer can remove entries from
<code>.guix-authorizations</code>. Key rotation can be handled similarly: a
committer can remove their former key and add their new key in a single
commit, signed by the former key.</p><p>The authorization invariant satisfies our needs for Guix. It has one
downside: it prevents pull-request-style workflows. Indeed, merging the
branch of a contributor not listed in <code>.guix-authorizations</code> would break
the authorization invariant. It’s a good tradeoff for Guix because our
workflow relies on <a href="https://lwn.net/Articles/702177/">patches carved into stone
tablets</a> (<a href="https://issues.guix.gnu.org/">patch
tracker</a>), but it’s not suitable for every
project out there.</p><h1>Bootstrapping</h1><p>The attentive reader may have noticed that something’s missing from the
explanation above: what do we do about commit <em>A</em> in the example above?
In other words, which commit do we pick as the first one where we
can start verifying the authorization invariant?</p><p>We solve this bootstrapping issue by defining <em>channel introductions</em>.
Previously, one would identify a channel simply by its URL. Now, when
introducing a channel to users, one needs to provide an additional piece
of information: the first commit where the authorization invariant
holds, and the fingerprint of the OpenPGP key used to sign that commit
(it’s not strictly necessary but provides an additional check).
Consider this commit graph:</p><p><img src="https://guix.gnu.org/static/blog/img/commit-graph-intro.svg" alt="Example commit graph with introduction." /></p><p>On this figure, <em>B</em> is the introduction commit. Its ancestors, such as
<em>A</em> are considered authentic. To authenticate, <em>C</em>, <em>D</em>, <em>E</em>, and <em>F</em>,
we check the authorization invariant.</p><p>As always when it comes to establishing trust, distributing channel
introductions is very sensitive. The introduction of the official
<code>guix</code> channel is <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/channels.scm?id=d2fde340adf197cc42bc4e0187deaf3a7bd3968d#n155">built into
Guix</a>.
Users obtain it when they install Guix the first time; hopefully they
verify the signature on the Guix tarball or ISO image, <a href="https://guix.gnu.org/manual/en/html_node/Binary-Installation.html">as noted in the
installation
instructions</a>,
which reduces chances of getting the “wrong” Guix, but it is still very much
<a href="https://en.wikipedia.org/wiki/Trust_on_first_use">trust-on-first-use</a>
(TOFU).</p><p>For signed third-party channels, users have to provide the channel’s
introduction in their <code>channels.scm</code> file, like so:</p><pre><code class="language-scheme">(channel
(name 'my-channel)
(url "https://example.org/my-channel.git")
(introduction
(make-channel-introduction
"6f0d8cc0d88abb59c324b2990bfee2876016bb86"
(openpgp-fingerprint
"CABB A931 C0FF EEC6 900D 0CFB 090B 1199 3D9A EBB5"))))</code></pre><p>The <code>guix describe</code> command now prints the introduction if there’s one.
That way, one can share their channel configuration, including
introductions, without having to be an expert.</p><p>Channel introductions also solve another problem: forks. Respecting the
authorization invariant “forever” would effectively prevent
“unauthorized” forks—forks made by someone who’s not in <code>.guix-authorizations</code>.
Someone publishing a fork simply needs to emit a new introduction
for their fork, pointing to a different starting commit.</p><p>Last, channel introductions give a <em>point of reference</em>: if an attacker
manipulates branch heads on Savannah to have them point to unrelated
commits (such as commits on an orphan branch that do not share any
history with the “official” branches), authentication will necessarily
fail as it stumbles upon the first unauthorized commit made by the
attacker. In the figure above, the red branch with commits <em>G</em> and <em>H</em>
cannot be authenticated because it starts from <em>A</em>, which lacks
<code>.guix-authorizations</code> and thus fails the authorization invariant.</p><p>That’s all for authentication! I’m glad you read this far. At this
point you can take a break or continue with the next section on how
<code>guix pull</code> prevents downgrade attacks.</p><h1>Downgrade attacks</h1><p>An important threat for software deployment tools is <em>downgrade</em> or
<em>roll-back</em> attacks. The attack consists in tricking users into
installing older, known-vulnerable software packages, which in turn may
offer new ways to break into their system. This is not strictly related
to the authentication issue we’ve been discussing, except that it’s
another important issue in this area that we took the opportunity to
address.</p><p>Guix saves provenance info for itself: <code>guix describe</code> prints that
information, essentially the Git commits of the channels used during
<code>git pull</code>:</p><pre><code>$ guix describe
Generation 149 Jun 17 2020 20:00:14 (current)
guix 8b1f7c0
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 8b1f7c03d239ca703b56f2a6e5f228c79bc1857e</code></pre><p>Thus, <code>guix pull</code>, once it has retrieved the latest commit of the
selected branch, can verify that it is doing a <em>fast-forward update</em> in
Git parlance—just like <code>git pull</code> does, but compared to the
previously-deployed Guix. A fast-forward update is when the new commit
is a descendant of the current commit. Going back to the figure above,
going from commit <em>A</em> to commit <em>F</em> is a fast-forward update, but going
from <em>F</em> to <em>A</em> or from <em>D</em> to <em>E</em> is not.</p><p>Not doing a fast-forward update would mean that the user is deploying an
older version of the Guix currently used, or deploying an unrelated
version from another branch. In both cases, the user is at risk of
ending up installing older, vulnerable software.</p><p>By default <code>guix pull</code> now errors out on non-fast-forward updates,
thereby protecting from roll-backs. Users <a href="https://issues.guix.gnu.org/41882">who understand the
risks</a> can override that by passing
<code>--allow-downgrades</code>.</p><p>Authentication and roll-back prevention allow users to safely refer to
mirrors of the Git repository. If <code>git.savannah.gnu.org</code> is down, one
can still update by fetching from a mirror, for instance with:</p><pre><code>guix pull --url=https://github.com/guix-mirror/guix</code></pre><p>If the repository at this URL is behind what the user already deployed,
or if it’s not a genuine mirror, <code>guix pull</code> will abort. In other
cases, it will proceed.</p><p>Unfortunately, there is no way to answer the general question “<em>is</em> X
<em>the latest commit of branch</em> B <em>?</em>”. Rollback detection prevents just
that, rollbacks, but there’s no mechanism in place to tell whether a
given mirror is stale. To mitigate that, channel authors can specify,
in the repository, the channel’s <em>primary URL</em>. This piece of
information lives in the <code>.guix-channel</code> file, in the repository, so
it’s authenticated. <code>guix pull</code> uses it to print a warning when the
user pulls from a mirror:</p><pre><code>$ guix pull --url=https://github.com/guix-mirror/guix
Updating channel 'guix' from Git repository at 'https://github.com/guix-mirror/guix'...
Authenticating channel 'guix', commits 9edb3f6 to 3e51f9e (44 new commits)...
guix pull: warning: pulled channel 'guix' from a mirror of https://git.savannah.gnu.org/git/guix.git, which might be stale
Building from this channel:
guix https://github.com/guix-mirror/guix 3e51f9e
…</code></pre><p>So far we talked about mechanics in a rather abstract way. That might
satisfy the graph theorist or the Git geek in you, but if you’re up for
a quick tour of the implementation, the next section is for you!</p><h1>A long process</h1><p>We’re kinda celebrating these days, but the <a href="https://issues.guix.gnu.org/22883">initial bug
report</a> was opened… in 2016. One of
the reasons was that we were hoping the general problem was solved
already and we’d “just” have to adapt what others had done. As for the
actual design: you would think it can be implemented in ten lines of
shell script invoking <code>gpgv</code> and <code>git</code>. Perhaps that’s a possibility,
but the resulting performance would be problematic—keep in mind that
users may routinely have to authenticate hundreds of commits. So we
took a long road, but the end result is worth it. Let’s recap.</p><p>Back in April 2016, committers <a href="https://issues.guix.gnu.org/22883#4">started signing
commits</a>, with a <a href="https://issues.guix.gnu.org/22883#36">server-side hook
prohibiting unsigned commits</a>. In
July 2016, we had <a href="https://issues.guix.gnu.org/22883#33">proof-of-concept libgit2
bindings</a> with the primitives
needed to verify signatures on commits, passing them to <code>gpgv</code>; later
<a href="https://gitlab.com/guile-git/guile-git/">Guile-Git</a> was born, providing
good coverage of the libgit2 interface. Then there was a two-year
hiatus during which no code was produced in that area.</p><p>Everything went faster starting from December 2019. Progress was
incremental and may have been hard to follow, even for die-hard Guix
hackers, so here are the major milestones:</p><ul><li>In December 2019, a first <a href="https://issues.guix.gnu.org/22883#48">authentication program for use by Guix
developers</a> landed; it could
be run from a checkout with the <code>make authenticate</code> command. It
would use Guile-Git but call out to <code>gpgv</code> for signature
verification, which made it rather slow. The list of authorized
keys was hard-coded.</li><li>In April 2020, we had <a href="https://issues.guix.gnu.org/22883#61">an implementation of OpenPGP for signature
verification purposes only</a>
available as <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/openpgp.scm"><code>(guix openpgp)</code></a>.
The code is based on Göran Weinholt’s pure-Scheme
<a href="https://github.com/weinholt/industria/">Industria</a> library, with
the addition of a few features and using
<a href="https://notabug.org/cwebber/guile-gcrypt">Guile-Gcrypt</a> for faster
crypto. That led to a tenfold speedup compared to invoking <code>gpgv</code>,
which is primarily due to the fact that our code <a href="https://issues.guix.gnu.org/22883#62">foregoes OpenPGP
bells and whistles</a> and
focuses on “just” signature verification. Notably, it ignores key
expiration and revocation.</li><li>In May, <a href="https://issues.guix.gnu.org/22883#64"><code>.guix-authorizations</code> support was
added</a>, superseding the
hard-coded list of authorized keys. The OpenPGP keyring could now
be loaded straight from a <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/?h=keyring">Git branch containing all the OpenPGP
keys ever
used</a>.</li><li>Early June, the authentication code was <a href="https://issues.guix.gnu.org/41653">moved to its own
module</a>, <code>(guix git-authenticate)</code>, with a test suite.</li><li>Soon after, Git authentication was <a href="https://issues.guix.gnu.org/41767">integrated in
channels</a>: <code>guix pull</code> would now
authenticate the <code>guix</code> channel, <a href="https://issues.guix.gnu.org/22883">closing the 4-year old
mega-issue</a>.</li><li>Just today, we added the final bits, <a href="https://issues.guix.gnu.org/42048">allowing channel authors to
benefit from the feature</a>.</li></ul><p>Whether you’re a channel author or a user, the feature is now <a href="https://guix.gnu.org/manual/devel/en/html_node/Channels.html">fully
documented in the
manual</a>,
and we’d love to get your feedback!</p><h1>SHA-1</h1><p>We can’t really discuss Git commit signing without mentioning
<a href="https://en.wikipedia.org/wiki/SHA-1">SHA-1</a>. The venerable
crytographic hash function is approaching end of life, as evidenced by
<a href="https://shattered.io/">recent</a>
<a href="https://sha-mbles.github.io/">breakthroughs</a>. Signing a Git commit
boils down to signing a SHA-1 hash, because all objects in
the Git store are identified by their SHA-1 hash.</p><p>Git now relies on a <a href="https://www.usenix.org/system/files/conference/usenixsecurity17/sec17-stevens.pdf">collision attack detection
library</a>
to mitigate practical attacks. Furthermore, the Git
project is planning a <a href="https://git-scm.com/docs/hash-function-transition/">hash function
transition</a> to
address the problem.</p><p>Some projects such as Bitcoin Core choose to not rely on SHA-1 at all.
Instead, for the commits they sign, they include in the commit log the
SHA512 hash of the tree, which the <a href="https://github.com/bitcoin/bitcoin/tree/master/contrib/verify-commits">verification scripts
check</a>.</p><p>Computing a tree hash <em>for each commit</em> in Guix would probably be
prohibitively costly. For now, for lack of a better solution, we rely
on Git’s collision attack detection and look forward to a hash function
transition.</p><p>As for SHA-1 in an OpenPGP context: our authentication code <a href="https://issues.guix.gnu.org/41787">rejects
SHA-1 OpenPGP signatures</a>, as
recommended.</p><h1>Related work</h1><p>A lot of work has gone into securing the software supply chain, often in
the context of binary distros, sometimes in a more general context; more
recent work also looks into Git authentication and related issues.
This section attempts to summarize how Guix relates to similar work that
we’re aware of in these two areas. More detailed discussions can be
found in the <a href="https://issues.guix.gnu.org/22883">issue tracker</a>.</p><p><a href="https://theupdateframework.io/">The Update Framework</a> (TUF) is a
reference for secure update systems, with <a href="https://github.com/theupdateframework/specification/blob/master/tuf-spec.md#the-update-framework-specification">a well-structured
spec</a>
and a number of
<a href="https://github.com/theupdateframework/specification/blob/master/tuf-spec.md#the-update-framework-specification">implementations</a>.
TUF is a great source of inspiration to think about this problem space.
Many of its goals are shared by Guix. Not all the attacks it aims to
protect against (Section 1.5.2 of the spec) are addressed by what’s
presented in this post: <em>indefinite freeze attacks</em>,
where updates never become available, are not addressed <em>per se</em> (though
easily observable), and <em>slow retrieval attacks</em> aren’t addressed
either. The notion of <em>role</em> is also something currently missing from
the Guix authentication model, where any authorized committer can touch
any files, though the model and <code>.guix-authorizations</code> format leave room
for such an extension.</p><p>However, both in its goals and system descriptions, TUF is biased
towards systems that distribute binaries as plain files with associated
meta-data. That creates a fundamental impedance mismatch. As an
example, attacks such as <em>fast-forward attacks</em> or <em>mix-and-match
attacks</em> don’t apply in the context of Guix; likewise, the <em>repository</em>
depicted in Section 3 of the spec has little in common with a Git
repository.</p><p>Developers of OPAM, the OCaml package manager, <a href="http://opam.ocaml.org/blog/Signing-the-opam-repository/">adapted TUF for use with
their Git-based package
repository</a>,
later updated to write <a href="https://github.com/hannesm/conex">Conex</a>, a
separate tool to authenticate OPAM repositories. OPAM is interesting
because like Guix it’s a source distro and its <a href="https://github.com/ocaml/opam-repository">package
repository</a> is a Git
repository containing “build recipe”. To date, it appears that <code>opam update</code> itself does not authenticate repositories though; it’s up to
users or developer to run Conex.</p><p>Another very insightful piece of work is the 2016 paper <a href="https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_torres-arias.pdf"><em>On omitting
commits and committing
omissions</em></a>.
The paper focuses on the impact of malicious modifications to Git
repository meta-data. An attacker with access to the repository can
modify, for instance, branch references, to cause a rollback attack or a
“teleport” attack, causing users to pull an older commit or an unrelated
commit. As written above, <code>guix pull</code> would detect such attacks.
However, <code>guix pull</code> would fail to detect cases where metadata
modification does not yield a rollback or teleport, yet gives users a
different view than the intended one—for instance, a user is directed to
an authentic but different branch rather than the intended one. The
“secure push” operation and the associated <em>reference state log</em> (RSL)
the authors propose would be an improvement.</p><h1>Wrap-up and outlook</h1><p>Guix now has a mechanism that allows it to authenticate updates. If
you’ve run <code>guix pull</code> recently, perhaps you’ve noticed additional
output and a progress bar as new commits are being authenticated. Apart
from that, the switch has been completely transparent. The
authentication mechanism is built around the commit graph of Git; in
fact, it’s a mechanism to <em>authenticate Git checkouts</em> and in that sense
it is not tied to Guix and its application domain. It is available not
only for the main <code>guix</code> channel, but also for third-party channels.</p><p>To bootstrap trust, we added the notion of <em>channel introductions</em>.
These are now visible in the user interface, in particular in the output
of <code>guix describe</code> and in the configuration file of <code>guix pull</code> and
<code>guix time-machine</code>. While channel configuration remains a few lines of
code that users typically paste, this extra bit of configuration might
be intimidating. It certainly gives an incentive to provide a
command-line interface to manage the user’s list of channels: <code>guix channel add</code>, etc.</p><p>The solution here is built around the assumption that Guix is
fundamentally a source-based distribution, and is thus completely
orthogonal to the <a href="https://guix.gnu.org/manual/en/html_node/Substitute-Server-Authorization.html">public key infrastructure (PKI) Guix uses for the
signature of
substitutes</a>.
Yet, the substitute PKI could probably benefit from the fact that we now
have a secure update mechanism for the Guix source code: since <code>guix pull</code> can securely retrieve a new substitute signing key, perhaps it
could somehow handle substitute signing key revocation and delegation
automatically? Related to that, channels could perhaps advertise a
substitute URL and its signing key, possibly allowing users to register
those when they first pull from the channel. All this requires more
thought, but it looks like there are new opportunities here.</p><p>Until then, if you’re a user or a channel author, we’d love to hear from
you! We’ve already gotten feedback that these new mechanisms <a href="https://issues.guix.gnu.org/41882">broke
someone’s workflow</a>; hopefully it
didn’t break yours, but either way your input is important in improving
the system. If you’re into security and think this design is terrible
or awesome, please do provide feedback.</p><p>It’s a long and article describing a long ride on a path we discovered
as we went, and it felt like an important milestone to share!</p><h1>Acknowledgments</h1><p>Thanks to everyone who provided feedback, ideas, or carried out code
review during this long process, notably (in no particular order):
Christopher Lemmer Webber, Leo Famulari, David Thompson, Mike Gerwitz,
Ricardo Wurmus, Werner Koch, Justus Winter, Vagrant Cascadian, Maxim
Cournoyer, Simon Tournier, John Soo, and Jakub Kądziołka. Thanks also
to janneke, Ricardo, Marius, and Simon for reviewing an earlier draft of
this post.</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2020/gnu-shepherd-user-services/GNU Shepherd user servicesEfraim Flashner2020-05-17T20:00:00Z2020-05-17T20:00:00Z One of the things which sets Guix apart from other GNU/Linux
distributions is that it uses GNU
Shepherd instead of the now
ubiquitous systemd. A side effect of this is that user systemd units do
not work on Guix System. Love, hate or extremely ambivalent toward
systemd, this means that users cannot rely on already written systemd
unit files for their regular user-level services. There are a couple of benefits to using GNU Shepherd, and not all of
them are due to it already being installed on Guix. Becoming comfortable
with using Shepherd and understanding how to write and edit Shepherd
service…<p>One of the things which sets Guix apart from other GNU/Linux
distributions is that it uses <a href="https://www.gnu.org/software/shepherd/">GNU
Shepherd</a> instead of the now
ubiquitous systemd. A side effect of this is that user systemd units do
not work on Guix System. Love, hate or extremely ambivalent toward
systemd, this means that users cannot rely on already written systemd
unit files for their regular user-level services.</p><p>There are a couple of benefits to using GNU Shepherd, and not all of
them are due to it already being installed on Guix. Becoming comfortable
with using Shepherd and understanding how to write and edit Shepherd
service configurations makes the transition from other GNU/Linux
distributions to Guix System easier. More complex services with their
own logic tree, using the full power of <a href="https://www.gnu.org/software/guile/">GNU
Guile</a>, are also possible. This
means you can have one service that behaves differently if it's running
on a different system or architecture without needing to call out to
shell scripts or using minimally different service definitions.</p><p>The GNU Shepherd manual
<a href="https://www.gnu.org/software/shepherd/manual/html_node/Jump-Start.html#index-Configuration-file">suggests</a>
putting all the services inside a
monolithic <code>init.scm</code> file, located by default at
<code>$XDG_CONFIG_DIR/shepherd/init.scm</code>. While this does make it easy to keep
everything in one place, it does create one glaring issue: any changes
to the file mean that all the services need to be stopped and restarted
in order for any changes to take place.</p><p>Luckily there's a nice function called <code>scandir</code> hiding in <a href="https://www.gnu.org/software/guile/manual/html_node/File-Tree-Walk.html#index-scandir"><code>ice-9 ftw</code></a>
which returns a list of all files in a specified directory (with options
for narrowing down the list or sorting it). This means that our <code>init.scm</code>
can contain a minimum of code and all actual services can be loaded from
individual files.</p><p>First the minimal <code>init.scm</code>:</p><pre><code class="language-scheme">(use-modules (shepherd service)
((ice-9 ftw) #:select (scandir)))
;; Load all the files in the directory 'init.d' with a suffix '.scm'.
(for-each
(lambda (file)
(load (string-append "init.d/" file)))
(scandir (string-append (dirname (current-filename)) "/init.d")
(lambda (file)
(string-suffix? ".scm" file))))
;; Send shepherd into the background
(action 'shepherd 'daemonize)</code></pre><p>Let's take a sample service for running syncthing, as defined in
<code>$XDG_CONFIG_DIR/shepherd/init.d/syncthing.scm</code>:</p><pre><code class="language-scheme">(define syncthing
(make <service>
#:provides '(syncthing)
#:docstring "Run `syncthing' without calling the browser"
#:start (make-forkexec-constructor
'("syncthing" "-no-browser")
#:log-file (string-append (getenv "HOME")
"/log/syncthing.log"))
#:stop (make-kill-destructor)
#:respawn? #t))
(register-services syncthing)
(start syncthing)</code></pre><p>As with any other shepherd service it is defined and registered, and in
this case it will start automatically. When the file is loaded by
shepherd after being discovered by scandir everything works exactly as
though the service definition were located directly inside the <code>init.scm</code>.</p><p>Now lets make a change. Since syncthing already has a <code>-logfile</code> flag and
it has built-in log rotation that sounds better than using shepherd's
<code>#:log-file</code> option. First we'll make our changes to the service:</p><pre><code class="language-scheme">(define syncthing
(make <service>
#:provides '(syncthing)
#:docstring "Run `syncthing' without calling the browser"
#:start (make-forkexec-constructor
'("syncthing" "-no-browser"
"-logflags=3" ; prefix with date & time
"-logfile=/home/user/log/syncthing.log"))
#:stop (make-kill-destructor)
#:respawn? #t))
(register-services syncthing)
(start syncthing)</code></pre><p>Now we stop syncthing:</p><pre><code class="language-sh">$ herd stop syncthing</code></pre><p>And we load the new service:</p><pre><code class="language-sh">$ herd load root ~/.config/shepherd/init.d/syncthing.scm</code></pre><p>This allows for quickly iterating on services without needing to stop
all the services! Let's take a look at another service:</p><pre><code class="language-scheme">(define fccache
(make <service>
#:provides '(fccache)
#:docstring "Run 'fc-cache -frv'"
#:start (make-forkexec-constructor
'("guix" "environment" "--ad-hoc" "fontconfig" "--"
"fc-cache" "-frv")
#:log-file (string-append (getenv "HOME")
"/log/fccache.log"))
#:one-shot? #t))
(register-services fccache)</code></pre><p>In this example I want to refresh my font cache but I don't want to
actually install fontconfig either system-wide or in my profile.</p><pre><code class="language-sh">$ which fc-cache
which: no fc-cache in (/home/user/.config/guix/current/bin:/home/user/.guix-profile/bin:/home/user/.guix-profile/sbin:/run/setuid-programs:/run/current-system/profile/bin:/run/current-system/profile/sbin)
$ herd start fccache
Service fccache has been started.</code></pre><p>Of course we can import other modules and leverage the code already
written there. In this case, instead of using the string "guix
environment --ad-hoc fontutils -- fc-cache -frv" let's use the <code>guix environment</code> function already available in <code>guix scripts environment</code>:</p><pre><code class="language-scheme">(use-modules (guix scripts environment))
(define fccache
(make <service>
#:provides '(fccache)
#:docstring "Run 'fc-cache -frv'"
#:start (lambda () ; Don't run immediately when registered!
(guix-environment "--ad-hoc" "fontconfig" "--" "fc-cache" "-frv"))
#:one-shot? #t))
(register-services fccache)</code></pre><pre><code class="language-sh">$ herd load root ~/.config/shepherd/init.d/fccache.scm
Loading /home/user/.config/shepherd/init.d/fccache.scm.
$ herd start fccache
/gnu/store/hbqlzgd8hcf6ndcmx7q7miqrsxb4dmkk-gs-fonts-8.11/share/fonts: caching, new cache contents: 0 fonts, 1 dirs
/gnu/store/hbqlzgd8hcf6ndcmx7q7miqrsxb4dmkk-gs-fonts-8.11/share/fonts/type1: caching, new cache contents: 0 fonts, 1 dirs
/gnu/store/hbqlzgd8hcf6ndcmx7q7miqrsxb4dmkk-gs-fonts-8.11/share/fonts/type1/ghostscript: caching, new cache contents: 35 fonts, 0 dirs
/home/user/.guix-profile/share/fonts: caching, new cache contents: 0 fonts, 7 dirs
/home/user/.guix-profile/share/fonts/opentype: caching, new cache contents: 8 fonts, 0 dirs
/home/user/.guix-profile/share/fonts/otf: caching, new cache contents: 12 fonts, 0 dirs
/home/user/.guix-profile/share/fonts/terminus: caching, new cache contents: 18 fonts, 0 dirs
/home/user/.guix-profile/share/fonts/truetype: caching, new cache contents: 58 fonts, 0 dirs
/home/user/.guix-profile/share/fonts/ttf: caching, new cache contents: 12 fonts, 0 dirs
/home/user/.guix-profile/share/fonts/type1: caching, new cache contents: 0 fonts, 1 dirs
/home/user/.guix-profile/share/fonts/type1/ghostscript: caching, new cache contents: 35 fonts, 0 dirs
/home/user/.guix-profile/share/fonts/woff: caching, new cache contents: 1 fonts, 0 dirs
/run/current-system/profile/share/fonts: skipping, no such directory
/home/user/.local/share/fonts: skipping, no such directory
/home/user/.fonts: skipping, no such directory
/gnu/store/hbqlzgd8hcf6ndcmx7q7miqrsxb4dmkk-gs-fonts-8.11/share/fonts/type1: skipping, looped directory detected
/home/user/.guix-profile/share/fonts/opentype: skipping, looped directory detected
/home/user/.guix-profile/share/fonts/otf: skipping, looped directory detected
/home/user/.guix-profile/share/fonts/terminus: skipping, looped directory detected
/home/user/.guix-profile/share/fonts/truetype: skipping, looped directory detected
/home/user/.guix-profile/share/fonts/ttf: skipping, looped directory detected
/home/user/.guix-profile/share/fonts/type1: skipping, looped directory detected
/home/user/.guix-profile/share/fonts/woff: skipping, looped directory detected
/gnu/store/hbqlzgd8hcf6ndcmx7q7miqrsxb4dmkk-gs-fonts-8.11/share/fonts/type1/ghostscript: skipping, looped directory detected
/home/user/.guix-profile/share/fonts/type1/ghostscript: skipping, looped directory detected
/var/cache/fontconfig: not cleaning unwritable cache directory
/home/user/.cache/fontconfig: cleaning cache directory
/home/user/.fontconfig: not cleaning non-existent cache directory
fc-cache: succeeded
herd: exception caught while executing 'start' on service 'fccache':
Throw to key `quit' with args `(0)'.</code></pre><p>The problem with this approach is that <code>guix-environment</code> returns the
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/environment.scm?h=v1.1.0#n771">exit
code</a>
of the programs it calls and <code>#:start</code> expects a
<a href="https://www.gnu.org/software/shepherd/manual/html_node/Slots-of-services.html#index-Starting-a-service-1"><code>constructor</code></a>
to return <code>#t</code> or <code>#f</code> so there's some work to be done here.</p><p>This was just a quick peek into what's possible with GNU Shepherd when
run as a user. Next time we'll take a look at integrating
<a href="https://www.gnu.org/software/mcron/">mcron</a> to replicate some of
systemd's timer functionality.</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2020/grafts-continued/Grafts, continuedLudovic Courtès2020-05-06T15:00:00Z2020-05-06T15:00:00Z Guix includes a mechanism called grafts that allows us to provide
users with security
updates
in a timely fashion, even for core packages deep down in the dependency
graph. Most users value the benefits of grafts, but newcomers were also
unavoidably struck by what turned out to be the undesirable side effect
of our graft implementation on user experience. This had been a
well-known problem for a while, but 1.1.0 finally addressed these
issues . This article recaps how grafts are implemented, what problems that
caused, and how we solved it. It’s a deep dive into core Guix,…<p>Guix includes a mechanism called <em>grafts</em> that allows us to provide
users with <a href="https://guix.gnu.org/manual/en/html_node/Security-Updates.html">security
updates</a>
in a timely fashion, even for core packages deep down in the dependency
graph. Most users value the benefits of grafts, but newcomers were also
unavoidably struck by what turned out to be the undesirable side effect
of our graft implementation on user experience. This had been a
well-known problem for a while, but <a href="https://guix.gnu.org/blog/2020/gnu-guix-1.1.0-released/">1.1.0 finally addressed these
issues</a>.</p><p>This article recaps how grafts are implemented, what problems that
caused, and how we solved it. It’s a deep dive into core Guix, and I
hope it’ll be insightful to all and intriguing to the functional
programming geeks among us!</p><h1>What’s this “graft” thing anyway?</h1><p>Grafts were introduced in the early days of Guix to <a href="https://guix.gnu.org/blog/2016/timely-delivery-of-security-updates/">address probably
the main practical shortcomings of functional software
deployment</a>.
In a nutshell, functional deployment as implemented by Nix and Guix
means that, when a package changes, everything that depends on it must
be rebuilt (or re-downloaded). To deploy a security fix in the C
library or in Bash, you would thus need to rebuild everything. Even
with a huge build farm, that can significantly delay the deployment of
fixes; users would find themselves either rebuilding things locally or,
at best, re-downloading binaries of everything.</p><p>To address this, Guix developers can instead specify a <em>replacement</em> in
a <a href="https://guix.gnu.org/manual/en/html_node/Defining-Packages.html">package
definition</a>.
If we have a bug-fix for, say, libc, developers would (1) define a
package for the fixed libc, and (2) add a <code>replacement</code> field in the
original libc package pointing to that fixed package. The effect is
that <em>only the bug-fix libc needs to be built</em>. When building a
package, the bug-fix libc is automatically <em>grafted onto that package</em>,
such that the resulting package refers to the bug-fix libc. <a href="https://guix.gnu.org/manual/en/html_node/Security-Updates.html">See the
manual</a>
for more.</p><p>When “lowering” a high-level <a href="https://guix.gnu.org/manual/en/html_node/Defining-Packages.html">package
definition</a>
to a low-level
<a href="https://guix.gnu.org/manual/en/html_node/Derivations.html"><em>derivation</em></a>,
Guix traverses the package dependency graph and identifies a set of
potentially applicable grafts. Why “potentially applicable”? Consider
this scenario: assume <code>perl</code> has a <code>replacement</code>; <code>coreutils</code> has a
dependency on <code>perl</code>, but it’s a build-time dependency: <code>coreutils</code> does
not depend on <code>perl</code> at run time. Thus, <code>coreutils</code> can be used as is,
there is no need to graft it.</p><p>But how do we know whether a dependency is a built-time-only dependency?
The <a href="https://guix.gnu.org/manual/en/html_node/package-Reference.html#index-package"><code>native-inputs</code>
field</a>
of a package usually lists build-time dependencies, but it’s more of a
hint. Ultimately, the set of run-time dependencies, which we call the
<em>references</em>, is the subset of the build-time dependencies that the
garbage collector (GC) in the build daemon finds <em>in the build
result</em>—Section 5.5.1 of <a href="http://nixos.org/~eelco/pubs/phd-thesis.pdf">Eelco Dolstra’s PhD
thesis</a> describes how the
GC
scans for references. In our example, we first have to actually build
<code>coreutils</code> before we can tell whether it depends on <code>perl</code> at
run time.</p><p>Guix arranges to graft only when necessary. In this example, <code>guix build coreutils</code> would return the same as <code>guix build coreutils --no-grafts</code>. Conversely, since <code>inkscape</code> has a run-time dependency on
<code>perl</code>, <code>guix build inkscape</code> returns a derivation that grafts the
<code>perl</code> replacement onto the original <code>inkscape</code> build result, the one
returned by <code>guix build inkscape --no-grafts</code>. The (simplified)
dependency graph of the derivation for the grafted <code>inkscape</code> looks like
this:</p><p><img src="https://guix.gnu.org/static/blog/img/inkscape-graft.svg" alt="Dependency graph of the graft derivation of Inkscape." /></p><p>Grafts are a form of what <a href="https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf"><em>Build Systems à la
Carte</em></a>
by Mokhov <em>et al.</em> (a good read!) refers to as <em>dynamic dependencies</em>:
grafting depends on intermediate <em>build results</em>.</p><p>Still here? With the background in place, let’s look at the problems
that arose.</p><h1>Grafts, the user interface, and performance</h1><p>Conceptually, to decide whether to graft a package, we examine the
references of the build result of the ungrafted package. However, we
usually want <code>guix install</code> & co. to first display an overall build
plan, especially when invoked with <code>--dry-run</code>:</p><pre><code>$ guix install inkscape
The following package will be installed:
inkscape 1.0
71.3 MB will be downloaded:
/gnu/store/xq64iaxx2gmlcgnipj31wjxlf1yd2g2p-gsl-2.6
/gnu/store/zrmhnj3pwchn2msphgnwzwd3q89m46rn-aspell-0.60.8
/gnu/store/5g31zf21lk8nrfd2b8qrm19nwdm9gir9-potrace-1.16
/gnu/store/qpr7387bsjs7rpl6flrwdvn2zbzh5bch-ghostscript-with-cups-9.52
/gnu/store/7y3lvk3xf4im8n44337mc6y0ccysvfia-font-dejavu-2.37
/gnu/store/95n3zvzxzq2bxvdvz292cg04757ij30x-cups-minimal-2.3.1
…</code></pre><p>To accommodate that, the pre-1.1.0 implementation of grafts did the
following: when
<a href="https://guix.gnu.org/manual/en/html_node/Substitutes.html">substitutes</a>
were enabled, it would get the list of references of ungrafted packages
from substitutes; only when substitutes for an ungrafted package are
missing would it first try to build that package. Thus, when
substitutes are available, <code>guix install</code> and similar commands would be
able to display the build plan upfront. However, when a packages had no
substitutes, you’d see Guix start building it without saying a word
about the build plan, which was <a href="https://issues.guix.gnu.org/issue/28310">arguably
confusing</a>.</p><p>But it’s worse than that. Grafting is per-package, so every time you
would lower a package to a derivation, you would need to answer the
question “does <em>this</em> specific package have substitutes, and if so,
should it be grafted?” The end result was <a href="https://issues.guix.gnu.org/issue/22990">poor resource usage and
terrible user interface
feedback</a>. For every package
that is a graft candidate, the user would see that infamous line:</p><pre><code>updating substitutes from 'https://ci.guix.gnu.org'...</code></pre><p>The problem was particularly acute when building whole systems with
<code>guix system</code> because there would typically be a large number of such
packages. Furthermore, each of these lines would correspond to
(roughly) a single HTTP GET request on a fresh TLS connection. That can
be slow… and annoying. Perhaps to some users this <code>updating substitutes</code> stuttering was the proof of the developers’ incompetence
and perhaps, truth be told, to some of us developers it was a small
price to pay for the sophistication of grafts.</p><p>For users who disable substitutes and build everything locally, the
situation wasn’t much better: all the packages candidate for grafting
would be built one by one, thereby missing parallelization opportunities
as specified by
<a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix_002ddaemon.html"><code>--max-jobs</code></a>.</p><h1>Gathering dynamic dependencies</h1><p>To address this, all these individual dynamic dependencies need to be
gathered somehow instead of being treated one by one. Conceptually, we
would like to, roughly, do a first pass lowering packages to derivations
as if grafting was disabled, build all these derivations, and then do a
second pass to determine which packages in the graph need to be grafted and
to compute the relevant grafting derivation. That would address the
performance issue: we’d now have as much parallelism as possible, so we
wouldn’t query substitutes or build packages one by one. If we reify
that second pass to the user interface code, it also addresses the user
interface issue by allowing it to display, possibly, two build plans:
the “ungrafted” one followed by the grafted one.</p><p>The problem is that our API is inherently serial: the
<code>package-derivation</code> function takes <em>one</em> package, lowers it, and
returns its derivation:</p><pre><code class="language-scheme">(use-modules (guix)
(gnu packages base)
(gnu packages inkscape))
(define s (open-connection))
(package-derivation s coreutils)
⇒ #<derivation /gnu/store/rpfdbax1py483m9ydhvp73s7dgmn6xh4-coreutils-8.31.drv => /gnu/store/jkj7wxybgcpdamkl6fz7wwbb1ak5wxvk-coreutils-8.31-debug /gnu/store/zibwkb5xavnv6z3gzknfqjsxb9b0izh0-coreutils-8.31 7f6c92e3a000>
(package-derivation s coreutils #:graft? #f)
⇒ #<derivation /gnu/store/rpfdbax1py483m9ydhvp73s7dgmn6xh4-coreutils-8.31.drv => /gnu/store/jkj7wxybgcpdamkl6fz7wwbb1ak5wxvk-coreutils-8.31-debug /gnu/store/zibwkb5xavnv6z3gzknfqjsxb9b0izh0-coreutils-8.31 7f6c92e3a000>
(package-derivation s inkscape)
⇒ #<derivation /gnu/store/jzm2zsq8m0rj8wdsmikc0p2ik0cprrcf-inkscape-0.92.4.drv => /gnu/store/clj8rjnsip8a35hyd9nf4l65w7ahn0gs-inkscape-0.92.4 7f6c9c15b730>
(package-derivation s inkscape #:graft? #f)
⇒ #<derivation /gnu/store/psd31x1fq0v2g594z217mh56xzg21dym-inkscape-0.92.4.drv => /gnu/store/zz28ckjwfxwkx3gsm8sc452kmvfiky6y-inkscape-0.92.4 7f6c90ad4f50></code></pre><p>Lowering includes dealing with grafts, and
that’s why we ended up with one-by-one inefficiencies. An option would
be to make all the API “plural”: have <code>package-derivation</code> and its
friends accept a <em>list</em> of packages instead of a single one. That would
be a huge amount of work and the end result would be unpleasant to use:
it’s easier to reason one-by-one.</p><p>The solution implemented in 1.1.0 instead starts from this observation:
the call graph of <code>package-derivation</code> mirrors the package graph. Thus,
we could gather dynamic dependencies using <a href="https://guix.gnu.org/manual/en/html_node/The-Store-Monad.html">monad
trickery</a>
or using “control effects”. We went for the latter, which didn’t have
the “contamination” problem of monads and led to simpler code.</p><p>The starting point is that, by definition, code with dynamic
dependencies necessarily calls
<a href="https://guix.gnu.org/manual/en/html_node/The-Store.html#index-build_002dderivations"><code>build-derivations</code></a>.
Taking advantage of <a href="https://www.gnu.org/software/guile/manual/html_node/Prompts.html">delimited continuations in
Guile</a>,
<code>build-derivations</code> is instrumented to <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=041b340da409078951267b6a8c43b27716e6b7ec">abort to a “build handler”
prompt</a>
when it’s called. The build handler receives the list of derivations to
build along with a continuation to invoke to resume the aborted
computation and start building things. User interface code can install
a build handler that displays what’s going to be built:</p><pre><code class="language-scheme">(with-build-handler (lambda (continue store things mode)
(show-what-to-build store things)
(continue #t))
…)</code></pre><p>To implement dry runs, simply omit the call to <code>continue</code> and nothing
will be built. (This is a slightly simplified artist view, see
<a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=07ce23e011d18460e7ff5553d4ff640f7073075b"><code>build-notifier</code></a>
for the real thing.)</p><p>Now, we need to take advantage of this mechanism to gather the
individual <code>build-derivations</code> calls so we can later emit a single
<code>build-derivations</code> call for all the gathered derivations. The goal is
to effectively gather all the calls for ungrafted packages, build them
all at once, and then resume graft computation.</p><p>To achieve that, we write a build handler that, when invoked, returns an
<code><unresolved></code> object that captures what to build and the continuation.
In addition, we provide a primitive to <em>introduce parallelism</em> such
that, if a dynamic dependency is encountered, we keep going and attempt
to compute as much as possible without resolving that dynamic
dependency. These are <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=c40bf5816cb3ffb59920a61f71bd34b53cac3637"><code>build-accumulator</code> and
<code>map/accumulate-builds</code></a>.
<code>map/accumulate-builds</code> is like <code>map</code>, except that it accumulates and
gathers <code>build-derivations</code> request.</p><p>By using <code>map/accumulate-builds</code> instead of <code>map</code> in a few
<a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=584dfdac3795541ff020aca3f488ceaf2ddd7fc3">key</a>
<a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=25af35fa32bf6c991510406a330d4a42bd5beba8">places</a>,
we obtain a good approximation of what we wanted, as illustrated in this
run:</p><pre><code>$ guix install zile-on-guile vim-full
The following packages will be installed:
zile-on-guile 2.4.14-0.fd09781
vim-full 8.2.0411
9.4 MB will be downloaded:
/gnu/store/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781
/gnu/store/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411
downloading from https://ci.guix.gnu.org/nar/lzip/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411...
vim-full-8.2.0411 8.9MiB 7.6MiB/s 00:01 [##################] 100.0%
downloading from https://ci.guix.gnu.org/nar/lzip/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781...
zile-on-guile-2.4.14-0.fd09781 140KiB 1.8MiB/s 00:00 [##################] 100.0%
The following derivation will be built:
/gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv
The following graft will be made:
/gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv
applying 8 grafts for /gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv...
building /gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv...</code></pre><p>What we see above is first a build plan that downloads binaries for the
two ungrafted packages, followed by a build plan for one grafting
derivations: we have successfully preserved parallelism.</p><p>The solution resembles the <code>suspending</code> scheduler discussed in the <em>à
la Carte</em> paper, though decomposition is not as principled as what the
paper describes. It remains an approximation and not the
optimal way to deal with dynamic dependencies. There are still
situations <a href="https://issues.guix.gnu.org/issue/40612">where that shows</a>,
but overall, it’s a significant improvement. Unlike <a href="https://issues.guix.gnu.org/issue/22990#7">other solutions
prototyped before</a>, this one
has the advantage of being orthogonal and simple: less than <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=c40bf5816cb3ffb59920a61f71bd34b53cac3637">100 new
lines of
code</a>,
and even <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=4b75a7060058bc2e959dcb4145067f6bba3e34e5">about 30 lines
removed</a>
from the graft implementation. That alone contributes a lot to the
author’s satisfaction. :-)</p><h1>Interlude: a scatter/gather pattern?</h1><p>In the end, we’re just gathering all the <code>build-derivations</code> calls,
turning them into a single call, and finally calling all the original
site continuations with the result. The same kind of issue shows up
when dealing with sequences of remote procedure calls (RPCs) and HTTP
requests, and it seems there’s a more general pattern lurking here.
Consider code like this:</p><pre><code class="language-scheme">(map (lambda (thing)
(http-get (thing->url thing)))
lst)</code></pre><p>Wouldn’t it be nice if we could somehow capture all the <code>http-get</code>
calls, turn them into a series of <a href="https://en.wikipedia.org/wiki/HTTP_pipelining">pipelined GET
requests</a>, and resume the
continuations with their result?</p><p>I haven’t found a standard functional pattern to address this and would
welcome ideas!</p><h1>Dynamic dependencies of all shapes</h1><p>We have seen how Guix deals with dynamic dependencies. Nix supports a
similar but limited form of dynamic dependencies through
the <code>import</code> primitive of the
Nix language, <a href="https://github.com/NixOS/nix/blob/master/src/libexpr/primops.cc#L74">which can take the result of a derivation
build</a>;
it does not attempt to gather the resulting <code>buildPaths</code> calls.</p><p>If we take a step back, we can see that Nix and Guix actually support
other forms of dynamic dependencies. For example, it’s possible to
write derivations whose result is a function of the reference graph of
another derivation’s build result. This is achieved in Guix by passing
the <a href="https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html#index-gexp_002d_003ederivation"><code>#:references-graphs</code> argument of
<code>gexp->derivation</code></a>,
which leads the build daemon to <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libstore/build.cc?id=298fb2907e3f432cea7dee9f58e89ab8d9dbd56f#n1763">include the reference graph in the
build
environment</a>.</p><p>Another form of dynamic dependency is <em>derivation-building derivations</em>
or <em>recursive derivations</em>, which were <a href="https://github.com/NixOS/nix/pull/3205">recently implemented in
Nix</a>. It supports another form
of dynamic dependency where the build process of a derivation can itself
create and build derivations (these are <a href="https://en.wikipedia.org/wiki/Parallel_task_scheduling_problem"><em>moldable
tasks</em></a>
in scheduling parlance). It’s a great feature because in a nutshell, it
allows Nix to be used not only to compose packages, but also at a finer
grain as part of a package build process.</p><p>Guix supports yet another form of dynamic dependencies. The newfangled
<a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix-deploy.html"><code>guix deploy</code>
tool</a>
works by <a href="https://guix.gnu.org/blog/2019/managing-servers-with-gnu-guix-a-tutorial/">evaluating g-expressions (gexps)
remotely</a>.
For example, before actually deploying an operating system, it first
runs code on the remote node to perform sanity checks: checking whether
the declared file system UUIDs or labels are valid, checking whether
additional kernel modules should be added to the initial RAM disk, and
so forth. To do that,
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/remote.scm#n109"><code>remote-eval</code></a>
first builds a derivation that produces a Scheme program, deploys it
along with all its dependencies on that target machine, runs it, and
retrieves the result. This form of dynamic dependency also benefits
from the gathering machinery discussed above.</p><h1>Conclusion</h1><p>This is a long article on what may look like a fine point of Guix design
and implementation, but there’s much to say about it! Grafts are key to
the use of functional deployment in production because they enable quick
security updates, and it’s a lot better if they don’t harm the user
experience.</p><p>The pre-1.1.0 implementation of grafts had a negative impact on the user
interface and on performance, which was due to the sequential handling
of grafts, one package at a time. In 1.1.0 we addressed it by using
delimited continuations to gather dynamic dependencies such as grafts,
perform builds in bulk, and resume each derivation computation.</p><p>As it turned out, the implementation of dynamic dependencies raises lots
of interesting design and implementation issues, and it’s probably not
the end of the road!</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2020/guile-3-and-guix/Guile 3 & GuixLudovic Courtès2020-01-24T15:00:00Z2020-01-24T15:00:00Z Version 3.0 of GNU Guile, an implementation of the Scheme programming
language , was released just last
week .
This is a major milestone for Guile, which gets compiler improvements
and just-in-time (JIT) native code generation, leading to significant
performance improvements over 2.2. It’s also great news for all the
users of Guile, and in particular for Guix! This post discusses what it means for Guix to migrate to Guile 3 and
how that migration is already taking place. Guile in Guix Most users interact with Guix through its command-line interface, and we
work hard to make it…<p>Version 3.0 of GNU Guile, an implementation of the <a href="https://schemers.org">Scheme programming
language</a>, <a href="https://www.gnu.org/software/guile/news/gnu-guile-300-released.html">was released just last
week</a>.
This is a major milestone for Guile, which gets compiler improvements
and just-in-time (JIT) native code generation, leading to significant
performance improvements over 2.2. It’s also great news for all the
users of Guile, and in particular for Guix!</p><p><img src="https://guix.gnu.org/static/blog/img/guile-3.png" alt="Guile 3 logo." /></p><p>This post discusses what it means for Guix to migrate to Guile 3 and
how that migration is already taking place.</p><h1>Guile in Guix</h1><p>Most users interact with Guix through its command-line interface, and we
work hard to make it as approachable as possible. As any user quickly
notices, Guix uses the Scheme programming language uniformly for its
configuration—from
<a href="https://guix.gnu.org/manual/devel/en/html_node/Channels.html">channels</a>
to
<a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-package.html#profile_002dmanifest">manifests</a>
and <a href="https://guix.gnu.org/manual/devel/en/html_node/Using-the-Configuration-System.html">operating
systems</a>—and
anyone who starts packaging software knows that <a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Packages.html">package
definitions</a>
are in fact Scheme code as well.</p><p>This is a significant departure from many other, and in particular from
<a href="https://nixos.org/nix/">Nix</a>. While Nix defines several
domain-specific languages (DSLs) for these aspects—the <a href="https://nixos.org/nix/manual/#chap-writing-nix-expressions">Nix
language</a>
but also specific
<a href="https://nixos.org/nix/manual/#sec-conf-file">configuration</a>
<a href="https://nixos.org/nix/manual/#chap-distributed-builds">languages</a>—Guix
chooses Scheme as the single language for all this, <a href="https://hal.inria.fr/hal-00824004/en">together with the
definition of high-level embedded domain-specific languages
(EDSLs)</a>.</p><p>It goes beyond that: in Guix System, all the things traditionally
implemented in C or as a set of Perl or shell scripts are implemented in
Scheme. That includes <a href="https://www.gnu.org/software/shepherd/">the init
system</a>, <a href="https://guix.gnu.org/manual/en/html_node/Build-Systems.html">package
builds</a>,
<a href="https://guix.gnu.org/manual/en/html_node/Initial-RAM-Disk.html">the initial RAM disk
(initrd)</a>,
<a href="https://guix.gnu.org/blog/2016/guixsd-system-tests/">system tests</a>, and
more. Because this leads to several layers of Scheme code, executed at
different points in time, Guix includes a <a href="https://hal.inria.fr/hal-01580582/en"><em>code staging</em>
mechanism</a> built upon the nice
properties of Scheme.</p><p>Why do that? The arguments, right from the start, were twofold: using a
general-purpose language allows us to benefit from its implementation
tooling, and having interfaces for “everything” in Scheme makes it easy
for users to navigate their distro or OS code and to reuse code to build
new features or applications. Guix developers benefit from the ease of
code reuse every day; demonstrative examples include the <a href="https://guix.gnu.org/blog/2017/running-system-services-in-containers/">use of Guix
container facilities in the init
system</a>,
the development of
<a href="https://guix.gnu.org/manual/devel/en/html_node/Development.html">many</a>
<a href="https://guix.gnu.org/manual/devel/en/html_node/Utilities.html">tools</a>
providing facilities around packages, the implementation of additional
<a href="https://emacs-guix.gitlab.io/website/">user</a>
<a href="https://github.com/UMCUGenetics/hpcguix-web/">interfaces</a>, and work on
applications that use Guix as a library such as the <a href="https://www.guixwl.org/">Guix Workflow
Language</a> and
<a href="https://hpc.guix.info/blog/2019/10/towards-reproducible-jupyter-notebooks/">Guix-Jupyter</a>.</p><p>As for the benefits of the host general-purpose language, these are
rather obvious: Guix developers benefit from an expressive language, an
optimizing compiler, a debugger, a powerful <a href="https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Interactively.html">read-eval-print loop
(REPL)</a>,
an <a href="https://nongnu.org/geiser/">interactive development environment</a>,
and all sorts of libraries. Moving to Guile 3 should add to that better
performance, essentially for free. To be comprehensive, Guile 3 may
well come with a set of brand new bugs too, but so far we seem to be
doing OK!</p><h1>Migrating to Guile 3</h1><p>What does it mean for Guix to migrate to Guile 3? We’ve seen above
different ways in which Guix relies on Guile. In short, we can say that
migration is threefold:</p><ol><li>Guix is a distro that ships Guile-related packages. Like any other
distro, it will have to upgrade its <code>guile</code> package to 3.0 and to
ensure packages that depend on it and updated as well.</li><li>Guix is a program written in Guile. As such, we need to make sure
that all its dependencies (half a dozen of Guile libraries) work
with Guile 3 and that Guix itself runs fine with Guile 3.</li><li>Guix ties together operating system components. In particular, the
init system (the Shepherd) and other boot-time facilities will also
migrate.</li></ol><h2>The packages</h2><p>Updating the distro is the boring part, but it’s best to get it right.
Guix makes it possible to have unrelated versions of variants of
packages in different environments or different profiles, which is very
nice. We’ll have performed a smooth transition if users and tools see
that the packages named <code>guile</code> and <code>guile-ssh</code> (say) transparently move
from Guile 2.2 to 3.0, <em>in lockstep</em>.</p><p>Put differently, most of the upgrade work upon a programming language
version bump deals with conventions, and in particular package names.
Currently, <code>guile</code> corresponds to the 2.2 stable series and all the
<code>guile-*</code> packages are built against it. In the meantime, the package
for Guile 3 is named <code>guile-next</code> and packages built against it are
called <code>guile3.0-*</code>. Over the last few weeks we created <code>guile3.0-</code>
variants for most Guile packages, something that’s <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=89a99d53f56c7c383659d821c28286b6d71e458d">easily achieved with
Guix</a>.</p><p>The big switch will consist in renaming all current <code>guile-*</code> packages
to <code>guile2.2-*</code> packages, for use with the legacy 2.2 series, and
renaming all the <code>guile3.0-*</code> packages to <code>guile-*</code>. We will switch
soon, but before getting there, we’re making sure important packages are
available for 3.0.</p><h2>Guix-the-program</h2><p>A more interesting part is “porting” Guix itself from Guile 2.2 to
Guile 3. It seems that developers have become wary of 2-to-3
transitions for programming languages. Fear not! Switching from
Guile 2 to Guile 3 turned out to be an easy task. In fact, very little
changed in the language itself; what did change—e.g., semantics on fine
points of the module system, support for structured exceptions—is either
optional or backwards-compatible.</p><p>As Guile 2.9 pre-releases trickled in, we started testing all the Guile
libraries Guix relies on against 2.9. For the vast majority of them,
all we had to do was to <a href="https://gitlab.com/gnutls/gnutls/commit/763e31d351933222281bf9c11ff0bddb89bb701d">update their <code>configure.ac</code> to allow builds
with
3.0</a>.</p><p>Guix itself was a bit more work, mostly because it’s a rather large code
base with a picky test suite. The bit that required most work has to do
with the introduction of <a href="https://www.gnu.org/software/guile/manual/html_node/Declarative-Modules.html"><em>declarative
modules</em></a>,
an optional semantic change in modules to support more compiler
optimizations. We had several <a href="https://en.wikipedia.org/wiki/White-box_testing">“white-box
tests”</a> where tests
would happily peek at private module bindings through <a href="https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Modules.html#index-_0040_0040">the magical-evil
<code>@@</code>
operator</a>.
Because we chose to enable declarative modules, we also had to adjust
our tests to no longer do that. And well, that’s about it!</p><p>At that point, we were able to <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=da7651806102d637253cb9f5677b96d6a178fc05">create a <code>guile3.0-guix</code> package
variant</a>,
primarily for testing purposes. Soon after, we told <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-pull.html"><code>guix pull</code></a>
to <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=8234fe653e61d0090138cbd4c48d877568355439">build Guix with 3.0 instead of
2.2</a>.
Thus, Guix users who upgrade will transparently find themselves running
Guix on Guile 3.0.</p><p>The main benefit is improved performance. Guile 3 is known to be <a href="https://www.gnu.org/software/guile/news/gnu-guile-300-released.html">up to
32 times faster than
Guile 2.2</a>
on some micro-benchmarks. Assessing the performance gains on a
“real-world” application like Guix is the real test. What would be a
relevant benchmark? At its core, Guix is essentially a compiler from
high-level descriptions of packages, operating systems, and the like, to
low-level build instructions
(<a href="https://guix.gnu.org/manual/devel/en/html_node/Derivations.html"><em>derivations</em></a>).
Thus, a good benchmark is a command that exercises little more than this
compilation step:</p><pre><code>guix build libreoffice ghc-pandoc guix --dry-run --derivation</code></pre><p>or:</p><pre><code>guix system build config.scm --dry-run --derivation</code></pre><p>On x86_64, the <code>guix build</code> command above on Guile 3 is 7% faster than
on Guile 2.2, and <code>guix system build</code>, which is more
computing-intensive, is 10% faster (heap usage is ~5% higher). This is
lower than the skyrocketing speedups observed on some microbenchmarks,
but it’s probably no surprise: these <code>guix</code> commands are short-lived (a
couple of seconds) and they’re rather I/O- and GC-intensive—something
JIT compilation cannot help with.</p><p>On 32-bit ARM, we temporarily disabled JIT <a href="https://issues.guix.gnu.org/issue/39208">due to a
bug</a>; there we observe a slight
<em>slowdown</em> compared to 2.2. This can be explained by the fact that
<a href="https://wingolog.org/archives/2018/01/17/instruction-explosion-in-guile">virtual machine (VM) instructions in 3.0 are lower-level than in
2.2</a>
and will hopefully be more than compensated for when JIT is re-enabled.</p><h2>Gluing it all together</h2><p>The last part of the Guile 3 migration has to do with how Guix, and in
particular Guix System, glues things together. As explained above, Guix
manipulates several stages of Scheme code that will run a different
points in time.</p><p>Firstly, the code that runs package builds, such as <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/gnu-build-system.scm">the one that runs
<code>./configure && make && make install</code></a>,
is Guile code. Currently that code runs on Guile 2.2, but on the next
major rebuild-the-world upgrade, we will switch to Guile 3.</p><p>Additionally, Guix produces Scheme code consumed by <a href="https://www.gnu.org/software/shepherd">the
Shepherd</a>, by
<a href="https://www.gnu.org/software/mcron">GNU mcron</a>, and for <a href="https://guix.gnu.org/manual/en/html_node/Guided-Graphical-Installation.html">the graphical
installer</a>.
These will soon switch to Guile 3 as well. This kind of change is made
easy by the fact that both the package definitions and the staged code
that depends on those packages live in the same repository.</p><h1>Long live Guile 3!</h1><p>Migrating Guix to Guile 3 is a bit of work because of the many ways Guix
interacts with Guile and because of the sheer size of the code base.
For a “2-to-3” transition though, it was easy. And fundamentally, it
remains a cheap transition compared to what it brings: better
performance and new features. That’s another benefit of using a
general-purpose language.</p><p>Thumbs up to everyone involved in its development, and long live
Guile 3!</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2019/spreading-the-news/Spreading the newsLudovic Courtès2019-11-12T14:00:00Z2019-11-12T14:00:00Z Like most free software projects, Guix has no shortage of communication
channels: there’s this blog , the NEWS
file for release
notes, a bunch of mailing lists , an IRC
channel, there’s also an unofficial
sub-Reddit and certainly more. Yet, as
developers, we often find it hard to communicate important changes to
our users. Starting from a few weeks ago, guix pull --news tells
users what’s new, and it already feels very helpful! This post is about
our motivations and the implementation of this new feature. Getting the word out Developers keep…<p>Like most free software projects, Guix has no shortage of communication
channels: there’s <a href="https://guix.gnu.org/blog">this blog</a>, the <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/NEWS"><code>NEWS</code>
file</a> for release
notes, a bunch of <a href="https://guix.gnu.org/contact">mailing lists</a>, an IRC
channel, there’s also an <a href="https://www.reddit.com/r/GUIX/">unofficial
sub-Reddit</a> and certainly more. Yet, as
developers, we often find it hard to communicate important changes to
our users. Starting from a few weeks ago, <code>guix pull --news</code> tells
users what’s new, and it already feels very helpful! This post is about
our motivations and the implementation of this new feature.</p><h1>Getting the word out</h1><p>Developers keep adding crazy features, fixing bugs, and generally
improving things. But how good is it if users aren’t aware of these new
things? As an example, since June, our build farm has been <a href="https://guix.gnu.org/blog/2019/substitutes-are-now-available-as-lzip/">offering
lzip-compressed
binaries</a>,
which results in better performance when installing software. But to
take advantage of that, users need to be aware of its existence, and
they need to upgrade their Guix daemon. Likewise, how do we get people
to learn about <a href="https://guix.gnu.org/blog/2019/managing-servers-with-gnu-guix-a-tutorial/">the new <code>guix deploy</code>
command</a>
that’s now available at their fingertips, about <a href="https://guix.gnu.org/blog/tags/security-advisory/">security issues that
were fixed</a>, about
important <a href="https://lists.gnu.org/archive/html/info-guix/2019-06/msg00001.html">infrastructure
changes</a>,
new options added to existing commands, and so forth?</p><p>Our (frustrating!) experience has been that release notes, blog posts,
and mailing list announcements aren’t quite enough to get the word out.
There’s always people who’ll miss important info and realize when it’s
already late, sometimes too late. Hence this simple idea: wouldn’t it
be nice if important information would reach users right in their
terminal?</p><h1><code>guix pull</code> news</h1><p>Alright, that’s not exactly a novel idea! In Debian for example,
<a href="https://manpages.debian.org/testing/apt-listchanges/apt-listchanges.1.en.html"><code>apt-listchanges</code></a>
shows news at the level of individual packages, taken from the
<code>NEWS.Debian</code> or <code>changelog.Debian</code> files that package maintainers
update. In addition, <code>apt dist-upgrade</code> and similar commands typically
display dialog boxes and menus when special actions need to be taken
when upgrading. That’s more or less what we’re looking for.</p><p>The situation in Guix is a little different: all of it lives in <a href="https://git.savannah.gnu.org/cgit/guix.git/">a
single Git repository</a> that
contains the core, the command-line interfaces, as well as package
definitions and operating system service definitions. The <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-pull.html"><code>guix pull</code></a>
command is sort-of equivalent to <code>apt update</code>, except that it updates
not only the set of available packages but also the <code>guix</code> tools
themselves and all the <a href="https://guix.gnu.org/manual/devel/en/html_node/Using-the-Configuration-System.html">operating system
interfaces</a>.
Under the hood, <code>guix pull</code> essentially does <code>git pull</code> and consequently
updates all of this. Guix very much follows a “rolling release” model.</p><p>For some time already, <code>guix pull</code> has been able to extract information
about new and upgraded packages and <a href="https://guix.gnu.org/blog/2018/multi-dimensional-transactions-and-rollbacks-oh-my/">to present it to the
user</a>.
Our goal was to complement it with high-level information about
important changes written with users in mind. Clearly, showing the Git
commit log is not an option: commit logs are meant for developers and
there’s roughly a thousand commits per month—way too much information.
We needed high-level news entries, explicitly written for users.</p><p>The end result is this: <code>guix pull</code> now displays, in addition to a
summary of the new and upgraded packages, the headlines of applicable
news entries contributed by developers. Users can view the details by
running <code>guix pull --news</code>:</p><p><img src="https://guix.gnu.org/static/blog/img/guix-pull-news.gif" alt="'guix pull' displaying news." /></p><p>Users can no longer miss the news, for the benefit of both users and
developers!</p><h1>Under the hood</h1><p>How does this all work? There were several goals and constraints.
First, like commit logs, our high-level news entries should be anchored
in the Git history. Second, unlike commit logs, it should be possible
to amend them—to fix typos, provide additional info, and so on. Third,
the project has been paying a lot of attention to internationalization,
with translations available for <a href="https://translationproject.org/domain/guix.html">user interface
messages</a>, for <a href="https://translationproject.org/domain/guix-packages.html">package
descriptions</a>,
and for the <a href="https://guix.gnu.org/manual/">user manual</a>—it’s one of
these things that helps free software reach out to more people; thus, we
naturally wanted news to be internationalized. Last, since Guix
supports third-party
<a href="https://guix.gnu.org/manual/devel/en/html_node/Channels.html">“channels”</a>,
which are extensions of the official <code>guix</code> channel, why not provide
channel authors access to that news feature?</p><p>With all these things in mind, we designed a simple <a href="https://guix.gnu.org/manual/devel/en/html_node/Channels.html#Writing-Channel-News">news
format</a>.
In essence, channel authors, including Guix developers, can provide a
news file that looks <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/etc/news.scm">like
this</a>:</p><pre><code class="language-scheme">(channel-news
(version 0)
(entry (commit "3e962e59d849e4300e447d94487684102d9d412e")
(title (en "@command{guix graph} now supports package
transformations")
(de "@command{guix graph} unterstützt nun Paketumwandlungen"))
(body
(en "The @command{guix graph} command now supports the common package
transformation options (see @command{info \"(guix) Package Transformation
Options\"}). This is useful in particular to see the effect of the
@option{--with-input} dependency graph rewriting option.")
(de "Der Befehl @command{guix graph} unterstützt nun die mit anderen
Befehlen gemeinsamen Umwandlungsoptionen (siehe @command{info \"(guix.de)
Paketumwandlungsoptionen\"}). Sie helfen insbesondere dabei, die Wirkung der
Befehlszeilenoption @option{--with-input} zum Umschreiben des
Abhängigkeitsgraphen zu sehen.")))
(entry (commit "49af34cfac89d384c46269bfd9388b2c73b1220a")
(title (en "@command{guix pull} now honors
@file{/etc/guix/channels.scm}")
(es "Ahora @command{guix pull} tiene en cuenta
@file{/etc/guix/channels.scm}"))
(body
(en "The @command{guix pull} command will now read the
@file{/etc/guix/channels.scm} file if it exists and if the per-user
@file{~/.config/guix/channels.scm} is not present. This allows administrators
of multi-user systems to define site-wide defaults.")
(es "Ahora la orden @command{guix pull} lee el fichero
@file{/etc/guix/channels.scm} si existe y el fichero personalizable
@file{~/.config/guix/channels.scm} no está presente. Esto permite a quienes
administran sistemas con múltiples usuarias definir valores predeterminados
en el sistema."))))</code></pre><p>Each news entry refers to a commit, the commit that introduced the
change it documents, and it has a title and body. Those can use Texinfo
markup for rich formatting, and translations can be provided directly
within the news file.</p><p>When <code>guix pull --news</code> runs, it determines which news entries are
applicable given the user’s previous Guix instance. The <code>(guix channels)</code> module provides a simple programming interface for that:</p><pre><code class="language-scheme">(use-modules (guix channels) (srfi srfi-1))
(channel-news-for-commit (first %default-channels)
"66b707a7d2325daadeed5ca913637eea3a2628e7"
"ac19950507e941b6263f62f4ee4e8934c1b1598e")
⇒ (#<<channel-news-entry> commit: "3e962e59d849e4300e447d94487684102d9d412e" tag: #f title: (("en" . "@command{guix graph} now supports package\ntransformations") …) body: …> #<<channel-news-entry> commit: "49af34cfac89d384c46269bfd9388b2c73b1220a" tag: #f title: (("en" . "@command{guix pull} now honors\n@file{/etc/guix/channels.scm}") …) body: …>)</code></pre><p>One thing is quite unusual (one might say “weird” :-)) about this news
format: it refers to commit IDs “in-band”. In other words, unlike Git
commit logs, which are “out-of-band”, the news file is contained
<em>inside</em> the repository that it refers to. Technically, it means that
before pushing a news entry, one must make sure it refers to the right
commit ID (the news format allows you to refer to tags as well, but one
may not want to create tags for every “news-worthy” change). Likewise,
rebasing might invalidate commit IDs that appear in the news file. So
this whole “in-band” log has drawbacks, but the big win is that it
allows us to amend news entries to fix typos, add translations, and so
on.</p><h1>In other news…</h1><p>Since it was applied <a href="https://issues.guix.gnu.org/issue/37413">a bit more than a month
ago</a>, we’ve already put the
news mechanism to good use on quite a few occasions: giving users
instructions on how to deal with locales after the last glibc upgrade,
giving them upgrade info for
<a href="https://guix.gnu.org/blog/2019/insecure-permissions-on-profile-directory-cve-2019-18192/">CVE-2019-18192</a>,
telling them about new command-line options, and more.</p><p>In parallel, given that reading the mailing lists is akin to “drinking
from a fire hose” as they say, Christopher Baines has been thinking
about how to provide regular development updates to interested users and
developers. Chris <a href="https://lists.gnu.org/archive/html/guix-devel/2019-11/msg00016.html">announced last
week</a>
a prototype of a “Guix Weekly News” web site that would aggregate
information about package updates automatically extracted from the <a href="http://data.guix.gnu.org/">Guix
Data Service</a>, along with manually written
updates. It would seem that this service could readily grab info from
channel news as well.</p><p>What about you, what do you expect in terms of news distribution? Join
us <a href="https://guix.gnu.org/contact">on the mailing list and on IRC</a> and
let us know!</p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2019/managing-servers-with-gnu-guix-a-tutorial/Managing Servers with GNU Guix: A TutorialJakob L. Kreuze2019-11-04T15:00:00Z2019-11-04T15:00:00Z The outcome of this year's
GSoC
is a Guile-based programming interface named guix deploy for
automatically creating, upgrading, and changing the configurations of
machines running the Guix System. The tool is comparable to
Ansible or
NixOps , but makes use of the system
configuration facilities provided by Guix. A post from earlier this
summer
described an early version of the programming interface, but we're
already a few months into autumn, so it's time for guide on how you
can use guix deploy in production. Simple case: managing a home server If the machine you…<p>The outcome of this year's
<a href="https://summerofcode.withgoogle.com/projects/#5232565294727168">GSoC</a>
is a Guile-based programming interface named <code>guix deploy</code> for
automatically creating, upgrading, and changing the configurations of
machines running the Guix System. The tool is comparable to
<a href="https://www.ansible.com/">Ansible</a> or
<a href="https://nixos.org/nixops/">NixOps</a>, but makes use of the system
configuration facilities provided by Guix. A <a href="http://guix.gnu.org/blog/2019/towards-guix-for-devops/">post from earlier this
summer</a>
described an early version of the programming interface, but we're
already a few months into autumn, so it's time for guide on how you
can use <code>guix deploy</code> in production.</p><h4>Simple case: managing a home server</h4><p>If the machine you need to manage is already running the Guix System,
it shouldn't be too hard to incorporate <code>guix deploy</code> into your
workflow. All that's needed is the <code><operating-system></code> declaration
you've been passing to <code>guix system reconfigure</code> and some information
about the machine (specifically its IP address and architecture). The
<code>guix deploy</code> command is invoked with the filename of a "deployment
specification" as an argument, whose contents should look something
like this:</p><pre><code class="language-scheme">;; Module imports
(use-modules (gnu) (guix))
(use-service-modules networking ssh)
(use-package-modules bootloaders)
;; Operating system description
(define os
(operating-system
(locale "en_US.utf8")
(timezone "America/New_York")
(keyboard-layout (keyboard-layout "us" "altgr-intl"))
(bootloader (bootloader-configuration
(bootloader grub-bootloader)
(target "/dev/sda")
(keyboard-layout keyboard-layout)))
(file-systems (cons* (file-system
(mount-point "/")
(device "/dev/sda1")
(type "ext4"))
%base-file-systems))
(host-name "alyssas-home-server")
(users (cons* (user-account
(name "alyssa")
(comment "Alyssa")
(group "users")
(home-directory "/home/alyssa")
(supplementary-groups
'("wheel" "netdev" "audio" "video")))
%base-user-accounts))
(sudoers-file (plain-file "sudoers" "\
root ALL=(ALL) ALL
%wheel ALL=NOPASSWD: ALL\n"))
(services (append
(list (service openssh-service-type
(openssh-configuration
(permit-root-login #t)))
(service dhcp-client-service-type))
%base-services))))
;; List of machines to deploy
(list (machine
(operating-system os)
(environment managed-host-environment-type)
(configuration (machine-ssh-configuration
(host-name "alyssa-p-hacker.tld")
(system "i686-linux")
(identity "/path/to/ssh-key")))))</code></pre><p>Even if Scheme isn't your forté, parts of this should look familiar if
you've used Guix before. The "operating system description" section
in particular is something you might use with <code>guix system reconfigure</code>. What's new is the last part: We construct a <code>list</code>
containing one <code>machine</code> of the <code>managed-host-environment-type</code>, for
which we've specified that <code>os</code> is the <code>operating-system</code>
declaration that we want to install on it, and that we can connect to
it using the parameters specified by the <code>machine-ssh-configuration</code>.</p><p>Let's take a step back for a moment and explain what a <code>machine</code> is.
<code>guix deploy</code> aims to support a number of different use-cases, which
we abstract as "environment types". We'll see other environment types
later in this article, but the general idea is that these environments
specify how resources should be "provisioned" or created. For
example, an environment type designed for working with a Virtual
Private Server (VPS) provider might make calls the provider's API to
request a virtual machine before installing the <code>machine</code>'s
<code>operating-system</code> declaration on it.</p><p>The environment type used in this example,
<code>managed-host-environment-type</code>, is intended for machines that are
already running Guix System and are accessible over SSH. It expects
that the <code>configuration</code> field of the <code>machine</code> be an instance of
<code>machine-ssh-configuration</code>, whose available fields are described in
the
<a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-deploy.html">manual</a>.
This gives <code>guix deploy</code> the information it needs to connect to the
machine's SSH daemon.</p><p>Running <code>guix deploy</code> with this file would build the "operating system
closure" of <code>os</code> -- a bundle of the packages, configuration
files, and other dependencies necessary to realize that configuration
-- for the architecture specified by <code>system</code> (in this case
<code>i686-linux</code>), send it over SSH to <code>alyssa-p-hacker.tld</code>, and then
remotely "activate" the configuration by creating a new system
generation and upgrading running services. Sweet! Upgrading our
single server setup has been reduced to an endeavour involving just
over a dozen keystrokes.</p><h4>More advanced case: managing a virtual private server deployment</h4><p>One server not cutting it for you? <code>guix deploy</code> can still help.
Suppose we run a web service that we'd like to split up across
multiple machines for performance reasons.</p><pre><code class="language-scheme">(define %forum-server-count 4)
(define (forum-server n)
(operating-system
(host-name (format #f "forum-server-~a" n))
...
(services (append (list (service httpd-service-type
(httpd-configuration
...)))
%base-services))))
(map (lambda (n)
(machine
(system (forum-server n))
(environment digital-ocean-environment-type)
(configuration (digital-ocean-configuration
(region "nyc3")
(size "s-1vcpu-1gb")
(enable-ipv6 #t)))))
(iota %forum-server-count))</code></pre><p>This example isn't as concrete as the first one; I'm intentionally
omitting parts of the configuration to make the example clearer.
Here, we automate the creation of <code>%forum-server-count</code> Digital Ocean
"droplets" in their <code>NYC3</code> region by creating a list of 4 machines.</p><p>Assuming that the environment variable <code>GUIX_DIGITAL_OCEAN_TOKEN</code> is
properly set, running <code>guix deploy</code> with this file will do much of the
same as the previous example. The difference is that four virtual
machines will be automatically created on Digital Ocean.</p><p>One important thing to note about the <code>digital-ocean-environment-type</code>
is that, currently, it <em>does not</em> automatically clean up unused
virtual machines. If you change something in the deployment
specification and run <code>guix deploy</code> again, the virtual machines from
the previous deployment will remain until you destroy them yourself.</p><h4>A quick peek into the internals of <code>digital-ocean-environment-type</code></h4><p>It would be an overstatement to say that the process of implementing a
new environment type is easy, but a fair amount of the work has
already been done for you. We'll use the definition of
<code>digital-ocean-environment-type</code> as an example.</p><pre><code class="language-scheme">(define digital-ocean-environment-type
(environment-type
(machine-remote-eval digital-ocean-remote-eval)
(deploy-machine deploy-digital-ocean)
(roll-back-machine roll-back-digital-ocean)
(name 'digital-ocean-environment-type)
(description "Provisioning of \"droplets\": virtual machines
provided by the Digital Ocean virtual private server (VPS) service.")))</code></pre><p>The <code>environment-type</code> record specifies a small amount of metadata
(<code>name</code> and <code>description</code>), as well as the names of three procedures:
one for remotely evaluating a
<a href="http://guix.gnu.org/manual/en/html_node/G_002dExpressions.html#G_002dExpressions">G-Expression</a>
on the host (<code>machine-remote-eval</code>), one for deploying an
<code>operating-system</code> declaration to the host, and one for rolling the
host back one generation.</p><p>This might sound like a lot, but the pattern for these high-level
environment types is to somehow obtain a machine running Guix System,
set up an SSH daemon, and then delegate to
<code>managed-host-environment-type</code>. <code>digital-ocean-remote-eval</code> is a
pretty good example of this:</p><pre><code class="language-scheme">(define (digital-ocean-remote-eval target exp)
"Internal implementation of 'machine-remote-eval' for MACHINE instances with
an environment type of 'digital-ocean-environment-type'."
(mlet* %store-monad ((name (droplet-name target))
(network -> (droplet-public-ipv4-network name))
(address -> (hash-ref network "ip_address"))
(ssh-key -> (digital-ocean-configuration-ssh-key
(machine-configuration target)))
(delegate -> (machine
(inherit target)
(environment managed-host-environment-type)
(configuration
(machine-ssh-configuration
(host-name address)
(identity ssh-key)
(system "x86_64-linux"))))))
(machine-remote-eval delegate exp)))</code></pre><p>As you can see, you could reasonably go about implementing an
environment type without ever having to learn what a G-Expression is.
Here, <code>droplet-name</code> derives the name of the droplet from the
machine's <code>operating-system</code> declaration, the information necessary to
connect to the droplet is found using <code>droplet-public-ipv4-network</code>,
and that's used to create machine of <code>managed-host-environment-type</code>.</p><h4>In conclusion</h4><p>I sincerely hope that <code>guix deploy</code> proves to be a useful to anyone
dealing with system administration or software development.
Transactional upgrades should provide peace of mind to those managing
servers (it's worth noting that few existing tools are capable of
recovering from failed deployments), and I believe that
procedurally-generated deployment configurations could very well be
the future of distribution for software such as web services: Imagine
if setting up a <a href="https://joinmastodon.org">Mastodon</a> instance were as
easy as downloading a Scheme file and handing it off to <code>guix deploy</code>.
The ease of writing code that generates code isn't the only benefit of
using Guile for something like this. Guile is a general-purpose
programming language, so more advanced tooling can reasonably be built
atop <code>guix deploy</code>. A GTK or Emacs DevOps interface, perhaps? (If
that idea sounds outlandish, consider that the latter has <a href="https://emacs-guix.gitlab.io/website/">already
happened</a> for the package
management interface.)</p><p>It's been a great summer working alongside everyone in the Guix
community. <code>guix deploy</code> is brand new (and a little unstable!), but
we've had enthusiastic adoption by several on the mailing lists who
were quick to report any issues they found. I'd like to thank
everyone on the <code>#guix</code> IRC channel and the mailing lists who got me
up to speed with the code, answered my questions, gave feedback when I
submitted my patches, and put <code>guix deploy</code> under the pressure of use
in production. And of course, I want to thank my mentors Christopher
Lemmer Webber and David Thompson. I had to make some hard design
decisions, but this was made easier thanks to the guidance of two
experienced Guix veterans.</p><p>Oh, and this isn't a goodbye. I really feel I've found my place as a
Guix contributor, and I can't wait to see what the future will bring
for <code>guix deploy</code>. Catch ya on the mailing lists!</p><h4>Editor's note</h4><p>Thank you for all of your hard work, Jakob!</p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2019/towards-guix-for-devops/Towards Guix for DevOpsJakob L. Kreuze2019-07-12T19:00:00Z2019-07-12T19:00:00Z Hey, there! I'm Jakob, a Google Summer of Code intern and new contributor to
Guix. Since May, I've been working on a DevOps automation tool for the Guix
System, which we've been calling guix deploy . The idea for a Guix DevOps tool has been making rounds on the mailing lists for
some time now. Years, in fact; Dave Thompson and Chris Webber put together a
proof-of-concept for it way back in 2015. Thus, we've had plenty of time to gaze
upon the existing tools for this sort of thing -- Ansible , NixOps -- and
fantasize about a…<p>Hey, there! I'm Jakob, a Google Summer of Code intern and new contributor to
Guix. Since May, I've been working on a DevOps automation tool for the Guix
System, which we've been calling <code>guix deploy</code>.</p><p>The idea for a Guix DevOps tool has been making rounds on the mailing lists for
some time now. Years, in fact; Dave Thompson and Chris Webber put together a
proof-of-concept for it way back in 2015. Thus, we've had plenty of time to gaze
upon the existing tools for this sort of thing -- <a href="https://www.ansible.com/">Ansible</a>, <a href="https://nixos.org/nixops/">NixOps</a> -- and
fantasize about a similar tool, albeit with the expressive power of Guile scheme
and the wonderful system configuration facilities of Guix. And now, those
fantasies are becoming a reality.</p><p>"DevOps" is a term that might be unfamiliar to a fair number of Guix users. I'll
spare you the detour to Wikipedia and give a brief explanation of what <code>guix deploy</code> does.</p><p>Imagine that you've spent the afternoon playing around with Guile's <code>(web)</code>
module, developing software for a web forum. Awesome! But a web forum with no
users is pretty boring, so you decide to shell out a couple bucks for a virtual
private server to run your web forum. You feel that Wildebeest admirers on the
internet deserve a platform of their own for discussion, and decide to dedicate
the forum to that.</p><p>As it turns out, <em>C. gnou</em> is a more popular topic than you ever would have
imagined. Your web forum soon grows in size -- attracting hundreds of thousands
of simultaneous users. Despite Guile's impressive performance characteristics,
one lowly virtual machine is too feeble to support such a large population of
Wildebeest fanatics. So you decide to use Apache as a load-balancer, and shell
out a couple more bucks for a couple more virtual private servers. Now you've
got a problem on your hands; you're the proud owner of five or so virtual
machines, and you need to make sure they're all running the most recent version
of either your web forum software or Apache.</p><p>This is where <code>guix deploy</code> comes into play. Just as you'd use an
<code>operating-system</code> declaration to configure services and user accounts on a
computer running the Guix System, you can now use that same <code>operating-system</code>
declaration to remotely manage any number of machines. A "deployment" managing
your Wildebeest fan site setup might look something like this:</p><pre><code class="language-scheme">...
;; Service for our hypothetical guile web forum application.
(define guile-forum-service-type
(service-type (name 'guile-forum)
(extensions
(list (service-extension shepherd-root-service-type
guile-forum-shepherd-service)
(service-extension account-service-type
(const %guile-forum-accounts))))
(default-value (guile-forum-configuration))
(description "A web forum written in GNU Guile.")))
...
(define %forum-server-count 4)
(define (forum-server n)
(operating-system
(host-name (format #f "forum-server-~a" n))
...
(services
(append (list (service guile-forum-service-type
(guile-forum-configuration
"GNU Fan Forum!")))
%base-services))))
(define load-balancer-server
(operating-system
(host-name "load-balancer-server"
...
(services
(append (list (service httpd-service-type
(httpd-configuration
...)))
%base-services)))))
;; One machine running our load balancer.
(cons (machine
(system load-balancer-server)
(environment manged-host-environment-type)
(configuration (machine-ssh-configuration
...)))
;; And a couple running our forum software!
(let loop ((n 1)
(servers '()))
(if (> n %forum-server-count)
servers
(loop (1+ n)
(cons (machine
(system (forum-server n))
(environment manged-host-environment-type)
(configuration (machine-ssh-configuration
...)))
servers)))))</code></pre><p>The take-away from that example is that there's a new <code>machine</code> type atop the
good ol' <code>operating-system</code> type, specifying how the machine should be
<em>provisioned</em>. The version of <code>guix deploy</code> that's currently on the master
branch only supports <code>managed-host-environment-type</code>, which is used for machines
that are already up and running the Guix System. Provisioning, in that sense,
only really involves opening an SSH connection to the host. But I'm sure you can
imagine a <code>linode-environment-type</code> which automatically sets up a virtual
private server through Linode, or a <code>libvirt-environment-type</code> that spins up a
virtual machine for running your services. Those types are what I'll be working
on in the coming months, in addition to cleaning up the code that's there now.</p><p>And yes, you did read that right. <code>guix deploy</code> is on the Guix master branch
right now! In fact, we've already done a successful deployment right here on
<a href="http://ci.guix.gnu.org/">ci.guix.gnu.org</a>. So, if this sounds as though it'd be up your alley, run <code>guix pull</code>, crack open the <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-deploy.html">manual</a>, and let us know how it goes!</p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2019/substitutes-are-now-available-as-lzip/Substitutes are now available as lzipLudovic Courtès2019-06-17T14:30:00Z2019-06-17T14:30:00Z For a long time, our build farm at ci.guix.gnu.org has been delivering
substitutes
(pre-built binaries) compressed with gzip. Gzip was never the best
choice in terms of compression ratio, but it was a reasonable and
convenient choice: it’s rock-solid, and zlib made it easy for us to have
Guile
bindings
to perform in-process compression in our multi-threaded guix publish
server. With the exception of building software from source, downloads take the
most time of Guix package upgrades. If users can download less,
upgrades become faster, and happiness ensues. Time has come to improve
on this, and starting…<p>For a long time, our build farm at ci.guix.gnu.org has been delivering
<a href="https://www.gnu.org/software/guix/manual/en/html_node/Substitutes.html">substitutes</a>
(pre-built binaries) compressed with gzip. Gzip was never the best
choice in terms of compression ratio, but it was a reasonable and
convenient choice: it’s rock-solid, and zlib made it easy for us to have
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/zlib.scm">Guile
bindings</a>
to perform in-process compression in our multi-threaded <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-publish.html"><code>guix publish</code></a>
server.</p><p>With the exception of building software from source, downloads take the
most time of Guix package upgrades. If users can download less,
upgrades become faster, and happiness ensues. Time has come to improve
on this, and starting from early June, Guix can publish and fetch
<a href="https://nongnu.org/lzip/">lzip</a>-compressed substitutes, in addition to
gzip.</p><h1>Lzip</h1><p><a href="https://nongnu.org/lzip/">Lzip</a> is a relatively little-known
compression format, initially developed by Antonio Diaz Diaz ca. 2013.
It has several C and C++ implementations with surprisingly few lines of
code, which is always reassuring. One of its distinguishing features is
a very good compression ratio with reasonable CPU and memory
requirements, <a href="https://nongnu.org/lzip/lzip_benchmark.html">according to benchmarks published by the
authors</a>.</p><p><a href="https://nongnu.org/lzip/lzlib.html">Lzlib</a> provides a well-documented C
interface and Pierre Neidhardt set out to write bindings for that
library, which eventually landed as the <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/lzlib.scm"><code>(guix lzlib)</code>
module</a>.</p><p>With this in place we were ready to start migrating our tools, and then
our build farm, to lzip compression, so we can all enjoy smaller
downloads. Well, easier said than done!</p><h1>Migrating</h1><p>The compression format used for substitutes is not a core component like
it can be in “traditional” binary package formats <a href="https://lwn.net/Articles/789449/">such as
<code>.deb</code></a> since Guix is conceptually a
“source-based” distro. However, deployed Guix installations did not
support lzip, so we couldn’t just switch our build farm to lzip
overnight; we needed to devise a transition strategy.</p><p>Guix asks for the availability of substitutes over HTTP. For example, a
question such as:</p><blockquote><p>“Dear server, do you happen to have a binary of
<code>/gnu/store/6yc4ngrsig781bpayax2cg6pncyhkjpq-emacs-26.2</code> that I could download?”</p></blockquote><p>translates into prose to an HTTP GET of
<a href="https://ci.guix.gnu.org/6yc4ngrsig781bpayax2cg6pncyhkjpq.narinfo">https://ci.guix.gnu.org/6yc4ngrsig781bpayax2cg6pncyhkjpq.narinfo</a>,
which returns something like:</p><pre><code>StorePath: /gnu/store/6yc4ngrsig781bpayax2cg6pncyhkjpq-emacs-26.2
URL: nar/gzip/6yc4ngrsig781bpayax2cg6pncyhkjpq-emacs-26.2
Compression: gzip
NarHash: sha256:0h2ibqpqyi3z0h16pf7ii6l4v7i2wmvbrxj4ilig0v9m469f6pm9
NarSize: 134407424
References: 2dk55i5wdhcbh2z8hhn3r55x4873iyp1-libxext-1.3.3 …
FileSize: 48501141
System: x86_64-linux
Deriver: 6xqibvc4v8cfppa28pgxh0acw9j8xzhz-emacs-26.2.drv
Signature: 1;berlin.guixsd.org;KHNpZ25hdHV…</code></pre><p>(This narinfo format is inherited from <a href="https://nixos.org/nix/">Nix</a> and
implemented
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/substitute.scm?id=121d9d1a7a2406a9b1cbe22c34343775f5955b34#n283">here</a>
and
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/publish.scm?id=121d9d1a7a2406a9b1cbe22c34343775f5955b34#n265">here</a>.)
This tells us we can download the actual binary from
<code>/nar/gzip/…-emacs-26.2</code>, and that it will be about 46 MiB (the
<code>FileSize</code> field.) This is what <code>guix publish</code> serves.</p><p>The trick we came up with was to allow <code>guix publish</code> to advertise
several URLs, one per compression format. Thus, for recently-built
substitutes, we get something <a href="https://ci.guix.gnu.org/mvhaar2iflscidl0a66x5009r44fss15.narinfo">like
this</a>:</p><pre><code>StorePath: /gnu/store/mvhaar2iflscidl0a66x5009r44fss15-gimp-2.10.12
URL: nar/gzip/mvhaar2iflscidl0a66x5009r44fss15-gimp-2.10.12
Compression: gzip
FileSize: 30872887
URL: nar/lzip/mvhaar2iflscidl0a66x5009r44fss15-gimp-2.10.12
Compression: lzip
FileSize: 18829088
NarHash: sha256:10n3nv3clxr00c9cnpv6x7y2c66034y45c788syjl8m6ga0hbkwy
NarSize: 94372664
References: 05zlxc7ckwflz56i6hmlngr86pmccam2-pcre-8.42 …
System: x86_64-linux
Deriver: vi2jkpm9fd043hm0839ibbb42qrv5xyr-gimp-2.10.12.drv
Signature: 1;berlin.guixsd.org;KHNpZ25hdHV…</code></pre><p>Notice that there are two occurrences of the <code>URL</code>, <code>Compression</code>, and
<code>FileSize</code> fields: one for gzip, and one for lzip. Old Guix instances
will just pick the first one, gzip; newer Guix will pick whichever
supported method provides the smallest <code>FileSize</code>, usually lzip. This
will make migration trivial in the future, should we add support for
other compression methods.</p><p>Users need to upgrade their Guix daemon to benefit from lzip. On a
“foreign distro”, simply run <code>guix pull</code> as root. On standalone Guix
systems, run <code>guix pull && sudo guix system reconfigure /etc/config.scm</code>. In both cases, the daemon has to be restarted, be it
with <code>systemctl restart guix-daemon.service</code> or with <code>herd restart guix-daemon</code>.</p><h1>First impressions</h1><p>This new gzip+lzip scheme has been deployed on ci.guix.gnu.org for a
week. Specifically, we run <code>guix publish -C gzip:9 -C lzip:9</code>, meaning
that we use the highest compression ratio for both compression methods.</p><p>Currently, only a small subset of the package substitutes are available
as both lzip and gzip; those that were already available as gzip have
not been recompressed. The following Guile program that taps into the
API of <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-weather.html"><code>guix weather</code></a>
allows us to get some insight:</p><pre><code class="language-scheme">(use-modules (gnu) (guix)
(guix monads)
(guix scripts substitute)
(srfi srfi-1)
(ice-9 match))
(define all-packages
(@@ (guix scripts weather) all-packages))
(define package-outputs
(@@ (guix scripts weather) package-outputs))
(define (fetch-lzip-narinfos)
(mlet %store-monad ((items (package-outputs (all-packages))))
(return
(filter (lambda (narinfo)
(member "lzip" (narinfo-compressions narinfo)))
(lookup-narinfos "https://ci.guix.gnu.org" items)))))
(define (lzip/gzip-ratio narinfo)
(match (narinfo-file-sizes narinfo)
((gzip lzip)
(/ lzip gzip))))
(define (average lst)
(/ (reduce + 0 lst)
(length lst) 1.))</code></pre><p>Let’s explore this at the
<a href="https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop">REPL</a>:</p><pre><code class="language-scheme">scheme@(guile-user)> (define lst
(with-store s
(run-with-store s (fetch-lzip-narinfos))))
computing 9,897 package derivations for x86_64-linux...
updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
scheme@(guile-user)> (length lst)
$4 = 2275
scheme@(guile-user)> (average (map lzip/gzip-ratio lst))
$5 = 0.7398994395478715</code></pre><p>As of this writing, around 20% of the package substitutes are
available as lzip, so take the following stats with a grain of salt.
Among those, the lzip-compressed substitute is on average 26% smaller
than the gzip-compressed one. What if we consider only packages bigger
than 5 MiB uncompressed?</p><pre><code class="language-scheme">scheme@(guile-user)> (define biggest
(filter (lambda (narinfo)
(> (narinfo-size narinfo)
(* 5 (expt 2 20))))
lst))
scheme@(guile-user)> (average (map lzip/gzip-ratio biggest))
$6 = 0.5974238562384483
scheme@(guile-user)> (length biggest)
$7 = 440</code></pre><p>For those packages, lzip yields substitutes that are 40% smaller on
average. Pretty nice! Lzip decompression is slightly more
CPU-intensive than gzip decompression, but downloads are
bandwidth-bound, so the benefits clearly outweigh the costs.</p><h1>Going forward</h1><p>The switch from gzip to lzip has the potential to make upgrades “feel”
faster, and that is great in itself.</p><p>Fundamentally though, we’ve always been looking in this project at
peer-to-peer solutions with envy. Of course, the main motivation is to
have a community-supported and resilient infrastructure, rather than a
centralized one, and that vision goes <a href="https://www.gnu.org/software/guix/blog/2017/reproducible-builds-a-status-update/">hand-in-hand with reproducible
builds</a>.</p><p>We started working on <a href="https://issues.guix.gnu.org/issue/33899">an extension to publish and fetch
substitutes</a> over
<a href="https://ipfs.io/">IPFS</a>. Thanks to its content-addressed nature, IPFS
has the potential to further reduce the amount of data that needs to be
downloaded on an upgrade.</p><p>The good news is that IPFS developers are also <a href="https://github.com/ipfs/package-managers">interested in working
with package manager
developers</a>, and I bet
there’ll be interesting discussions at <a href="https://camp.ipfs.io/">IPFS
Camp</a> in just a few days. We’re eager to pursue
our IPFS integration work, and if you’d like to join us and hack the
good hack, <a href="https://www.gnu.org/software/guix/contact/">let’s get in
touch!</a></p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2019/connecting-reproducible-deployment-to-a-long-term-source-code-archive/Connecting reproducible deployment to a long-term source code archiveLudovic Courtès2019-03-29T15:50:00Z2019-03-29T15:50:00Z GNU Guix can be used as a “package manager” to install and upgrade
software packages as is familiar to GNU/Linux users, or as an
environment manager, but it can also provision containers or virtual
machines, and manage the operating system running on your machine. One foundation that sets it apart from other tools in these areas is
reproducibility . From a high-level view, Guix allows users to
declare complete software environments and instantiate them. They can
share those environments with others, who can replicate them or adapt
them to their needs. This aspect is key to reproducible computational
experiments: scientists…<p>GNU Guix can be used as a “package manager” to install and upgrade
software packages as is familiar to GNU/Linux users, or as an
environment manager, but it can also provision containers or virtual
machines, and manage the operating system running on your machine.</p><p>One foundation that sets it apart from other tools in these areas is
<em>reproducibility</em>. From a high-level view, Guix allows users to
<em>declare</em> complete software environments and instantiate them. They can
share those environments with others, who can replicate them or adapt
them to their needs. This aspect is key to reproducible computational
experiments: scientists need to reproduce software environments before
they can reproduce experimental results, and this is one of the things
we are focusing on in the context of the
<a href="https://guix-hpc.bordeaux.inria.fr">Guix-HPC</a> effort. At a lower
level, the project, along with others in the <a href="https://reproducible-builds.org">Reproducible
Builds</a> community, is working to ensure
that software build outputs are <a href="https://reproducible-builds.org/docs/definition/">reproducible,
bit for bit</a>.</p><p>Work on reproducibility at all levels has been making great progress.
Guix, for instance, allows you to <a href="https://www.gnu.org/software/guix/blog/2018/multi-dimensional-transactions-and-rollbacks-oh-my/">travel back in
time</a>.
That Guix can travel back in time <em>and</em> build software reproducibly is a
great step forward. But there’s still an important piece that’s missing
to make this viable: a stable source code archive. This is where
<a href="https://www.softwareheritage.org">Software Heritage</a> (SWH for short)
comes in.</p><h1>When source code vanishes</h1><p>Guix contains thousands of package definitions. Each <a href="https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html">package
definition</a>
specifies the package’s source code URL and hash, the package’s
dependencies, and its build procedure. Most of the time, the package’s
source code is an archive (a “tarball”) fetched from a web site, but
more and more frequently the source code is a specific revision checked
out directly from a version control system.</p><p>The obvious question, then, is: what happens if the source code URL
becomes unreachable? The whole reproducibility endeavor collapses when
source code disappears. And source code <em>does</em> disappear, or, even
worse, it can be modified in place. At GNU we’re doing a good job of
having stable hosting that keeps releases around “forever”, unchanged
(modulo rare exceptions). But a lot of free software out there is
hosted on personal web pages with a short lifetime and on commercial
hosting services that come and go.</p><p>By default Guix would look up source code by hash in the caches of our
build farms. This comes for free: the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Substitutes.html">“substitute”
mechanism</a>
extends to all “build artifacts”, including downloads. However, with
limited capacity, our build farms do not keep all the source code of all
the packages for a long time. Thus, one could very well find oneself
unable to rebuild a package months or years later, simply because its
source code disappeared or moved to a different location.</p><h1>Connecting to the archive</h1><p>It quickly became clear that reproducible builds had “reproducible
source code downloads”, so to speak, as a prerequisite. The Software
Heritage archive is the missing piece that would finally allow us to
reproduce software environments years later in spite of the volatility
of code hosting sites. Software Heritage’s mission is to archive
essentially “all” the source code ever published, including version
control history. Its archive already periodically ingests release
tarballs from the GNU servers, repositories from GitHub, packages from
PyPI, and much more.</p><p><img src="https://www.gnu.org/software/guix/static/blog/img/software-heritage-logo-title.svg" alt="Software Heritage logo" /></p><p>We quickly settled on a scheme where Guix would fall back to the
Software Heritage archive whenever it fails to download source code from
its original location. That way, package definitions don’t need to be
modified: they still refer to the original source code URL, but the
downloading machinery transparently goes to Software Heritage when
needed.</p><p>There are two types of source code downloads in Guix: tarball downloads,
and version control checkouts. In the former case, resorting to
Software Heritage is easy: Guix knows the SHA256 hash of the tarball so
it can look it up by hash using <a href="https://archive.softwareheritage.org/api/1/content/raw/">the <code>/content</code> endpoint of the
archive’s
interface</a>.</p><p>Fetching version control checkouts is more involved. In this case, the
downloader would first resolve the commit identifier to obtain a
<a href="https://archive.softwareheritage.org/api/1/revision/">Software Heritage
revision</a>. The
actual code for that revision is then fetched through the
<a href="https://docs.softwareheritage.org/devel/swh-vault/api.html"><em>vault</em></a>.</p><p>The vault conveniently allows users to fetch the tarball corresponding
to a revision. However, not all revisions are readily available as
tarballs (understandably), so the vault has an interface that allows you
to request the “<em>cooking</em>” of a specific revision. Cooking is
asynchronous and can take some time. Currently, if a revision hasn’t
been cooked yet, the Guix download machinery will request it and wait
until it is available. The process can take some time but will
eventually succeed.</p><p>Success! At this point, we have essentially bridged the gap between the
stable archive that Software Heritage provides and the reproducible
software deployment pipeline of Guix. This code <a href="https://issues.guix.info/issue/33432">was
integrated</a> in November 2018,
making it the first free software distribution backed by a stable
archive.</p><h1>The challenges ahead</h1><p>This milestone was encouraging: we had seemingly achieved our goal, but
we also knew of several shortcomings. First, even though the software
we package is still primarily distributed as tarballs, Software Heritage
keeps relatively few of these tarballs. Software Heritage does ingest
tarballs, notably those found on <a href="https://ftp.gnu.org/gnu/">the GNU
servers</a>, but the primary focus is on
preserving complete version control repositories rather than release
tarballs.</p><p>It is not yet clear to us what to do with plain old tarballs. On one
hand, they are here and cannot be ignored. Furthermore, some provide
artifacts that are not in version control, such as <code>configure</code> scripts,
and often enough they are accompanied by a cryptographic signature from
the developers that allows recipients to <em>authenticate</em> the code—an
important piece of information that’s often missing from version control
history. On the other hand, version control tags are increasingly
becoming the mechanism of choice to distribute software releases. It
may be that tags will become the primary mechanism for software release
distribution soon enough.</p><p>Version control tags turn out not to be ideal either, because they’re
mutable and per-repository. Conversely, Git commit identifiers are
unambiguous and repository-independent because they’re essentially
content-addressed, but our package definitions often refer to tags, not
commits, because that makes it clearer that we’re providing an actual
release and not an arbitrary revision (this is another illustration of
<a href="https://en.wikipedia.org/wiki/Zooko's_triangle">Zooko’s triangle</a>).</p><p>This leads to another limitation that stems from the mismatch between
the way Guix and Software Heritage compute hashes over version control
checkouts: both compute a hash over a serialized representation of the
directory, but they serialize the directory in a different way (SWH
serializes directories as Git trees, while Guix uses “normalized
archives” or Nars, the format the build daemon manipulates, which is
inherited from Nix.) That prevents Guix from looking up revisions by
content hash. The solution will probably involve changing Guix to
support the same method as Software Heritage, and/or adding Guix’s method
to Software Heritage.</p><p>Having to wait for “cooking” completion can also be problematic. The
Software Heritage team is investigating the possibility to
<a href="https://forge.softwareheritage.org/T1350">automatically cook all version control
tags</a>. That way, relevant
revisions would almost always be readily available through the vault.</p><p>Similarly, we have no guarantee that software provided by Guix is
available in the archive. Our plan is to <a href="https://forge.softwareheritage.org/T1352">extend Software
Heritage</a> such that it would
periodically archive the source code of software packaged by Guix.</p><h1>Going further</h1><p>In the process of adding support for Software Heritage, Guix <a href="https://issues.guix.info/issue/33432">gained
Guile bindings to the Software Heritage HTTP
interface</a>. Here’s a couple of
things we can do:</p><pre><code class="language-scheme">(use-modules (guix swh))
;; Check whether SWH has ever crawled our repository.
(define o (lookup-origin "https://git.savannah.gnu.org/git/guix.git"))
⇒ #<<origin> id: 86312956 …>
;; It did! When was its last visit?
(define last-visit
(first (origin-visits o)))
(date->string (visit-date last-visit))
⇒ "Fri Mar 29 10:07:45Z 2019"
;; Does it have our “v0.15.0” Git tag?
(lookup-origin-revision "https://git.savannah.gnu.org/git/guix.git" "v0.15.0")
⇒ #<<revision> id: "359fdda40f754bbf1b5dc261e7427b75463b59be" …></code></pre><p>Guix itself is a Guile library so when we combine the two, there are
interesting things we can do:</p><pre><code class="language-scheme">(use-modules (guix) (guix swh)
(gnu packages base)
(gnu packages golang))
;; This is our GNU Coreutils package.
coreutils
⇒ #<package coreutils@8.30 gnu/packages/base.scm:342 1c67b40>
;; Does SWH have its tarball?
(lookup-content (origin-sha256 (package-source coreutils))
"sha256")
⇒ #<<content> checksums: (("sha1" …)) data-url: …>
;; Our package for HashiCorp’s Configuration Language (HCL) is
;; built from a Git commit.
(define commit
(git-reference-commit
(origin-uri (package-source go-github-com-hashicorp-hcl))))
;; Is this particular commit available in the archive?
(lookup-revision commit)
⇒ #<<revision> id: "23c074d0eceb2b8a5bfdbb271ab780cde70f05a8" …></code></pre><p>We’re currently using a subset of this interface, but there’s certainly
more we could do. For example, we can compute archive coverage of the
Guix packages; we can also request the archival of each package’s source
code <em>via</em> the <a href="https://archive.softwareheritage.org/api/1/origin/save/">“save code”
interface</a>—though
all this is subject to <a href="https://archive.softwareheritage.org/api/#rate-limiting">rate
limiting</a>.</p><h1>Wrap-up</h1><p>Software Heritage support in Guix creates a bridge from the stable
source code archive to reproducible software deployment with complete
provenance tracking. For the first time it gives us a software package
distribution that can be rebuilt months or years later. This is
particularly beneficial in the context of reproducible science: finally
we can describe reproducible software environments, a prerequisite for
reproducible computational experiments.</p><p>Going further, we can provide a complete software supply tool chain with
provenance tracking that links revisions in the archive to
bit-reproducible build artifacts produced by Guix. Oh and Guix itself
<a href="https://archive.softwareheritage.org/api/1/origin/git/url/https://git.savannah.gnu.org/git/guix.git/">is
archived</a>,
so we have this meta-level where we could link Guix revisions to the
revisions of packages it provides… There are still technical challenges
to overcome, but that vision is shaping up.</p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2018/a-packaging-tutorial-for-guix/A packaging tutorial for GuixPierre Neidhardt2018-10-10T16:00:00Z2018-10-10T16:00:00Z Introduction GNU Guix stands out as the hackable package manager, mostly because it uses
GNU Guile , a powerful high-level programming language, one of the Scheme
dialects from the Lisp family . Package definitions are also written in Scheme, which empowers Guix in some very
unique ways, unlike most other package managers that use shell scripts or
simple languages. Use functions, structures, macros and all of Scheme expressiveness for your
package definitions. Inheritance makes it easy to customize a package by inheriting from it and
modifying only what is needed.…<h1>Introduction</h1><p>GNU Guix stands out as the <em>hackable</em> package manager, mostly because it uses
<a href="https://www.gnu.org/software/guile/">GNU Guile</a>, a powerful high-level programming language, one of the <a href="https://en.wikipedia.org/wiki/Scheme_(programming_language)">Scheme</a>
dialects from the <a href="https://en.wikipedia.org/wiki/Lisp_(programming_language)">Lisp family</a>.</p><p>Package definitions are also written in Scheme, which empowers Guix in some very
unique ways, unlike most other package managers that use shell scripts or
simple languages.</p><ul><li><p>Use functions, structures, macros and all of Scheme expressiveness for your
package definitions.</p></li><li><p>Inheritance makes it easy to customize a package by inheriting from it and
modifying only what is needed.</p></li><li><p>Batch processing: the whole package collection can be parsed, filtered and
processed. Building a headless server with all graphical interfaces stripped
out? It's possible. Want to rebuild everything from source using specific
compiler optimization flags? Pass the <code>#:make-flags "..."</code> argument to the
list of packages. It wouldn't be a stretch to think <a href="https://wiki.gentoo.org/wiki/USE_flag">Gentoo USE flags</a> here,
but this goes even further: the changes don't have to be thought out
beforehand by the packager, they can be <em>programmed</em> by the user!</p></li></ul><p>The following tutorial covers all the basics around package creation with Guix.
It does not assume much knowledge of the Guix system nor of the Lisp language.
The reader is only expected to be familiar with the command line and to have some
basic programming knowledge.</p><h1>A "Hello World" package</h1><p>The <a href="https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html">“Defining Packages” section of the manual</a> introduces the basics of Guix
packaging. In the following section, we will partly go over those basics again.</p><p><code>GNU hello</code> is a dummy project that serves as an idiomatic example for
packaging. It uses the GNU build system (<code>./configure && make && make install</code>).
Guix already provides a package definition which is a perfect example to start
with. You can look up its declaration with <code>guix edit hello</code> from the
command line. Let's see how it looks:</p><pre><code class="language-scheme">(define-public hello
(package
(name "hello")
(version "2.10")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnu/hello/hello-" version
".tar.gz"))
(sha256
(base32
"0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
(build-system gnu-build-system)
(synopsis "Hello, GNU world: An example GNU package")
(description
"GNU Hello prints the message \"Hello, world!\" and then exits. It
serves as an example of standard GNU coding practices. As such, it supports
command-line arguments, multiple languages, and so on.")
(home-page "https://www.gnu.org/software/hello/")
(license gpl3+)))</code></pre><p>As you can see, most of it is rather straightforward. But let's review the
fields together:</p><ul><li><strong>name:</strong> The project name. Using Scheme conventions, we prefer to keep it
lower case, without underscore and using dash-separated words.</li><li><p><strong>source:</strong> This field contains a description of the source code origin. The
<code>origin</code> record contains these fields:</p><ol><li>The method, here <code>url-fetch</code> to download via HTTP/FTP, but other methods
exist, such as <code>git-fetch</code> for Git repositories.</li><li>The URI, which is typically some <code>https://</code> location for <code>url-fetch</code>. Here
the special <code>mirror://gnu</code> refers to a set of well known locations, all of
which can be used by Guix to fetch the source, should some of them fail.</li><li>The <code>sha256</code> checksum of the requested file. This is essential to ensure
the source is not corrupted. Note that Guix works with base32 strings,
hence the call to the <code>base32</code> function.</li></ol></li><li><strong>build-system:</strong> This is where the power of abstraction provided by the Scheme
language really shines: in this case, the <code>gnu-build-system</code>
abstracts away the famous <code>./configure && make && make install</code> shell invocations. Other build systems include the
<code>trivial-build-system</code> which does not do anything and requires
from the packager to program all the build steps, the
<code>python-build-system</code>, the <code>emacs-build-system</code>, <a href="https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html">and many
more</a>.</li><li><strong>synopsis:</strong> It should be a concise summary of what the package
does. For many packages a tagline from the
project's home page can be used as the synopsis.</li><li><strong>description:</strong> Same as for the synopsis, it's fine to re-use the project
description from the homepage. Note that Guix uses Texinfo
syntax.</li><li><strong>home-page:</strong> Use HTTPS if available.</li><li><strong>license:</strong> See <code>guix/licenses.scm</code> in the project source for a full list.</li></ul><p>Time to build our first package! Nothing fancy here for now: we will stick to a
dummy "my-hello", a copy of the above declaration.</p><p>As with the ritualistic "Hello World" taught with most programming languages,
this will possibly be the most "manual" approach. We will work out an ideal
setup later; for now we will go the simplest route.</p><p>Save the following to a file <code>my-hello.scm</code>.</p><pre><code class="language-scheme">(use-modules (guix packages)
(guix download)
(guix build-system gnu)
(guix licenses))
(package
(name "my-hello")
(version "2.10")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnu/hello/hello-" version
".tar.gz"))
(sha256
(base32
"0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
(build-system gnu-build-system)
(synopsis "Hello, Guix world: An example custom Guix package")
(description
"GNU Hello prints the message \"Hello, world!\" and then exits. It
serves as an example of standard GNU coding practices. As such, it supports
command-line arguments, multiple languages, and so on.")
(home-page "https://www.gnu.org/software/hello/")
(license gpl3+))</code></pre><p>We will explain the extra code in a moment.</p><p>Feel free to play with the different values of the various fields. If you
change the source, you'll need to update the checksum. Indeed, Guix refuses to
build anything if the given checksum does not match the computed checksum of the
source code. To obtain the correct checksum of the package declaration, we
need to download the source, compute the sha256 checksum and convert it to
base32.</p><p>Thankfully, Guix can automate this task for us; all we need is to provide the
URI:</p><pre><code class="language-sh">$ guix download mirror://gnu/hello/hello-2.10.tar.gz
Starting download of /tmp/guix-file.JLYgL7
From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz...
following redirection to `https://mirror.ibcp.fr/pub/gnu/hello/hello-2.10.tar.gz'...
…10.tar.gz 709KiB 2.5MiB/s 00:00 [##################] 100.0%
/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i</code></pre><p>In this specific case the output tells us which mirror was chosen.
If the result of the above command is not the same as in the above snippet,
update your <code>my-hello</code> declaration accordingly.</p><p>Note that GNU package tarballs come with an OpenPGP signature, so you
should definitely check the signature of this tarball with <code>gpg</code> to
authenticate it before going further:</p><pre><code class="language-sh">$ guix download mirror://gnu/hello/hello-2.10.tar.gz.sig
Starting download of /tmp/guix-file.03tFfb
From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz.sig...
following redirection to `https://ftp.igh.cnrs.fr/pub/gnu/hello/hello-2.10.tar.gz.sig'...
….tar.gz.sig 819B 1.2MiB/s 00:00 [##################] 100.0%
/gnu/store/rzs8wba9ka7grrmgcpfyxvs58mly0sx6-hello-2.10.tar.gz.sig
0q0v86n3y38z17rl146gdakw9xc4mcscpk8dscs412j22glrv9jf
$ gpg --verify /gnu/store/rzs8wba9ka7grrmgcpfyxvs58mly0sx6-hello-2.10.tar.gz.sig /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
gpg: Signature made Sun 16 Nov 2014 01:08:37 PM CET
gpg: using RSA key A9553245FDE9B739
gpg: Good signature from "Sami Kerola <kerolasa@iki.fi>" [unknown]
gpg: aka "Sami Kerola (http://www.iki.fi/kerolasa/) <kerolasa@iki.fi>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 8ED3 96E3 7E38 D471 A005 30D3 A955 3245 FDE9 B739</code></pre><p>You can then happily run</p><pre><code class="language-sh">$ guix package --install-from-file=my-hello.scm</code></pre><p>You should now have <code>my-hello</code> in your profile!</p><pre><code class="language-sh">$ guix package --list-installed=my-hello
my-hello 2.10 out
/gnu/store/f1db2mfm8syb8qvc357c53slbvf1g9m9-my-hello-2.10</code></pre><p>We've gone as far as we could without any knowledge of Scheme. Now is the right
time to introduce the minimum we need from the language before we can proceed.</p><h1>A Scheme crash-course</h1><p>As we've seen above, basic packages don't require much Scheme knowledge, if
at all. But as you progress and your desire to write more and more complex
packages grows, it will become both necessary and empowering to hone your Lisper
skills.</p><p>Since an extensive Lisp course is very much out of the scope of this tutorial,
we will only cover some basics here.</p><p>Guix uses the Guile implementation of Scheme. To start playing with the
language, install it with <code>guix package --install guile</code> and start a <a href="https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop">REPL</a> by
running <code>guile</code> from the command line.</p><p>Alternatively you can also run <code>guix environment --ad-hoc guile -- guile</code> if
you'd rather not have Guile installed in your user profile.</p><p>In the following examples we use the <code>></code> symbol to denote the REPL prompt, that
is, the line reserved for user input. See <a href="https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Interactively.html">the Guile manual</a> for more details on
the REPL.</p><ul><li><p>Scheme syntax boils down to a tree of expressions (or <em>s-expression</em> in Lisp
lingo). An expression can be a literal such as numbers and strings, or a
compound which is a parenthesized list of compounds and literals. <code>#t</code> and
<code>#f</code> stand for the booleans "true" and "false", respectively.</p><p>Examples of valid expressions:</p><pre><code class="language-scheme">> "Hello World!"
"Hello World!"
> 17
17
> (display (string-append "Hello " "Guix" "\n"))
"Hello Guix!"</code></pre></li><li><p>This last example is a function call embedded in another function call. When
a parenthesized expression is evaluated, the first term is the function and
the rest are the arguments passed to the function. Every function returns the
last evaluated expression as value.</p></li><li><p>Anonymous functions are declared with the <code>lambda</code> term:</p><pre><code class="language-scheme">> (lambda (x) (* x x))
#<procedure 120e348 at <unknown port>:24:0 (x)></code></pre><p>The above lambda returns the square of its argument. Since everything is an
expression, the <code>lambda</code> expression returns an anonymous function, which can
in turn be applied to an argument:</p><pre><code class="language-scheme">> ((lambda (x) (* x x)) 3)
9</code></pre></li><li><p>Anything can be assigned a global name with <code>define</code>:</p><pre><code class="language-scheme">> (define a 3)
> (define square (lambda (x) (* x x)))
> (square a)
9</code></pre></li><li><p>Procedures can be defined more concisely with the following syntax:</p><pre><code class="language-scheme">(define (square x) (* x x))</code></pre></li><li><p>A list structure can be created with the <code>list</code> procedure:</p><pre><code class="language-scheme">> (list 2 a 5 7)
(2 3 5 7)</code></pre></li><li><p>The <em>quote</em> disables evaluation of a parenthesized expression: the first term
is not called over the other terms. Thus it effectively returns a list of
terms.</p><pre><code class="language-scheme">> '(display (string-append "Hello " "Guix" "\n"))
(display (string-append "Hello " "Guix" "\n"))
> '(2 a 5 7)
(2 a 5 7)</code></pre></li><li><p>The <em>quasiquote</em> disables evaluation of a parenthesized expression until a
comma re-enables it. Thus it provides us with fine-grained control over what
is evaluated and what is not.</p><pre><code class="language-scheme">> `(2 a 5 7 (2 ,a 5 ,(+ a 4)))
(2 a 5 7 (2 3 5 7))</code></pre><p>Note that the above result is a list of mixed elements: numbers, symbols (here
<code>a</code>) and the last element is a list itself.</p></li><li><p>Multiple variables can be named locally with <code>let</code>:</p><pre><code class="language-scheme">> (define x 10)
> (let ((x 2)
(y 3))
(list x y))
(2 3)
> x
10
> y
ERROR: In procedure module-lookup: Unbound variable: y</code></pre><p>Use <code>let*</code> to allow later variable declarations to refer to earlier
definitions.</p><pre><code class="language-scheme">> (let* ((x 2)
(y (* x 3)))
(list x y))
(2 6)</code></pre></li><li><p>The keyword syntax is <code>#:</code>, it is used to create unique identifiers. See also
the <a href="https://www.gnu.org/software/guile/manual/html_node/Keywords.html">Keywords section in the Guile manual</a>.</p></li><li><p>The percentage <code>%</code> is typically used for read-only global variables in the
build stage. Note that it is merely a convention, like <code>_</code> in C. Scheme Lisp
treats <code>%</code> exactly the same as any other letter.</p></li><li><p>Modules are created with <code>define-module</code>. For instance</p><pre><code class="language-scheme">(define-module (guix build-system ruby)
#:use-module (guix store)
#:export (ruby-build
ruby-build-system))</code></pre><p>defines the module <code>ruby</code> which must be located in
<code>guix/build-system/ruby.scm</code> somewhere in <code>GUILE_LOAD_PATH</code>. It depends on
the <code>(guix store)</code> module and it exports two symbols, <code>ruby-build</code> and
<code>ruby-build-system</code>.</p></li></ul><p>For a more detailed introduction, check out <a href="http://www.troubleshooters.com/codecorn/scheme_guile/hello.htm">Scheme at a Glance</a>, by Steve Litt.</p><p>One of the reference Scheme books is the seminal <em>Structure and Interpretation
of Computer Programs</em>, by Harold Abelson and Gerald Jay Sussman, with Julie
Sussman. You'll find a free copy <a href="https://mitpress.mit.edu/sites/default/files/sicp/index.html">online</a>, together with <a href="https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-001-structure-and-interpretation-of-computer-programs-spring-2005/video-lectures/">videos of the lectures
by the authors</a>. The book is available in Texinfo format as the <code>sicp</code> Guix
package. Go ahead, run <code>guix package --install sicp</code> and start reading with
<code>info sicp</code> (or with the Emacs Info reader). An unofficial ebook <a href="https://sarabander.github.io/sicp/">is also
available</a>.</p><p>You'll find more books, tutorials and other resources at <a href="https://schemers.org/">https://schemers.org/</a>.</p><h1>Setup</h1><p>Now that we know some Scheme basics we can detail the different possible setups
for working on Guix packages.</p><p>There are several ways to set up a Guix packaging environment.</p><p>We recommend you work directly on the Guix source checkout since it makes it
easier for everyone to contribute to the project.</p><p>But first, let's look at other possibilities.</p><h3>Local file</h3><p>This is what we previously did with <code>my-hello</code>. With the Scheme basics we've
covered, we are now able to explain the leading chunks. As stated in <code>guix package --help</code>:</p><pre><code class="language-sh">-f, --install-from-file=FILE
install the package that the code within FILE
evaluates to</code></pre><p>Thus the last expression <em>must</em> return a package, which is the case in our
earlier example.</p><p>The <code>use-modules</code> expression tells which of the modules we need in the file.
Modules are a collection of values and procedures. They are commonly called
"libraries" or "packages" in other programming languages.</p><h3>GUIX_PACKAGE_PATH</h3><p><em>Note: Starting from Guix 0.16, the more flexible Guix "channels" are the
preferred way and supersede <code>GUIX_PACKAGE_PATH</code>. See below.</em></p><p>It can be tedious to specify the file from the command line instead of simply
calling <code>guix package --install my-hello</code> as you would do with the official
packages.</p><p>Guix makes it possible to streamline the process by adding as many "package
declaration paths" as you want.</p><p>Create a directory, say <code>~./guix-packages</code> and add it to the <code>GUIX_PACKAGE_PATH</code>
environment variable:</p><pre><code class="language-sh">$ mkdir ~/guix-packages
$ export GUIX_PACKAGE_PATH=~/guix-packages</code></pre><p>To add several directories, separate them with a colon (<code>:</code>).</p><p>Our previous <code>my-hello</code> needs some adjustments though:</p><pre><code class="language-scheme">(define-module (my-hello)
#:use-module (guix licenses)
#:use-module (guix packages)
#:use-module (guix build-system gnu)
#:use-module (guix download))
(define-public my-hello
(package
(name "my-hello")
(version "2.10")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnu/hello/hello-" version
".tar.gz"))
(sha256
(base32
"0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
(build-system gnu-build-system)
(synopsis "Hello, Guix world: An example custom Guix package")
(description
"GNU Hello prints the message \"Hello, world!\" and then exits. It
serves as an example of standard GNU coding practices. As such, it supports
command-line arguments, multiple languages, and so on.")
(home-page "https://www.gnu.org/software/hello/")
(license gpl3+)))</code></pre><p>Note that we have assigned the package value to an exported variable name with
<code>define-public</code>. This is effectively assigning the package to the <code>my-hello</code>
variable so that it can be referenced, among other as dependency of other
packages.</p><p>If you use <code>guix package --install-from-file=my-hello.scm</code> on the above file, it
will fail because the last expression, <code>define-public</code>, does not return a
package. If you want to use <code>define-public</code> in this use-case nonetheless, make
sure the file ends with an evaluation of <code>my-hello</code>:</p><pre><code class="language-scheme">; ...
(define-public my-hello
; ...
)
my-hello</code></pre><p>This last example is not very typical.</p><p>Now <code>my-hello</code> should be part of the package collection like all other official
packages. You can verify this with:</p><pre><code class="language-sh">$ guix package --show=my-hello</code></pre><h3>Guix channels</h3><p>Guix 0.16 features channels, which is very similar to <code>GUIX_PACKAGE_PATH</code> but
provides better integration and provenance tracking. Channels are not
necessarily local, they can be maintained as a public Git repository for
instance. Of course, several channels can be used at the same time.</p><p>See the <a href="http://guix.info/manual/en/Channels.html">“Channels” section in the manual</a> for setup details.</p><h3>Direct checkout hacking</h3><p>Working directly on the Guix project is recommended: it reduces the friction
when the time comes to submit your changes upstream to let the community benefit
from your hard work!</p><p>Unlike most software distributions, the Guix repository holds in one place both
the tooling (including the package manager) and the package definitions. This
choice was made so that it would give developers the flexibility to modify the
API without breakage by updating all packages at the same time. This reduces
development inertia.</p><p>Check out the official <a href="https://git-scm.com/">Git</a> repository:</p><pre><code class="language-sh">$ git clone https://git.savannah.gnu.org/git/guix.git</code></pre><p>In the rest of this article, we use <code>$GUIX_CHECKOUT</code> to refer to the location of
the checkout.</p><p>Follow the instruction from the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Contributing.html">"Contributing" chapter</a> in the manual to set up the
repository environment.</p><p>Once ready, you should be able to use the package definitions from the
repository environment.</p><p>Feel free to edit package definitions found in <code>$GUIX_CHECKOUT/gnu/packages</code>.</p><p>The <code>$GUIX_CHECKOUT/pre-inst-env</code> script lets you use <code>guix</code> over the package
collection of the repository.</p><ul><li><p>Search packages, such as Ruby:</p><pre><code class="language-sh">$ cd $GUIX_CHECKOUT
$ ./pre-inst-env guix package --list-available=ruby
ruby 1.8.7-p374 out gnu/packages/ruby.scm:119:2
ruby 2.1.6 out gnu/packages/ruby.scm:91:2
ruby 2.2.2 out gnu/packages/ruby.scm:39:2</code></pre></li><li><p>Build a package, here Ruby version 2.1:</p><pre><code class="language-sh">$ ./pre-inst-env guix build --keep-failed ruby@2.1
/gnu/store/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6</code></pre></li><li><p>Install it to your user profile:</p><pre><code class="language-sh">$ ./pre-inst-env guix package --install ruby@2.1</code></pre></li><li><p>Check for common mistakes:</p><pre><code class="language-sh">$ ./pre-inst-env guix lint ruby@2.1</code></pre></li></ul><p>Guix strives at maintaining a high packaging standard; when contributing to the
Guix project, remember to</p><ul><li>follow the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Coding-Style.html">coding style</a>,</li><li>and review the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Submitting-Patches.html">check list</a> from the manual.</li></ul><p>Once you are happy with the result, you are welcome to send your contribution to
make it part of Guix. This process is also detailed in the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Contributing.html">manual</a>.</p><p>It's a community effort so the more join in, the better Guix becomes!</p><h1>Extended example</h1><p>The above "Hello World" example is as simple as it goes. Packages can be more
complex than that and Guix can handle more advanced scenarios. Let's look at
another, more sophisticated package (slightly modified from the source):</p><pre><code class="language-scheme">(define-module (gnu packages version-control)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix utils)
#:use-module (guix packages)
#:use-module (guix git-download)
#:use-module (guix build-system cmake)
#:use-module (gnu packages ssh)
#:use-module (gnu packages web)
#:use-module (gnu packages pkg-config)
#:use-module (gnu packages python)
#:use-module (gnu packages compression)
#:use-module (gnu packages tls))
(define-public my-libgit2
(let ((commit "e98d0a37c93574d2c6107bf7f31140b548c6a7bf")
(revision "1"))
(package
(name "my-libgit2")
(version (git-version "0.26.6" revision commit))
(source (origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/libgit2/libgit2/")
(commit commit)))
(file-name (git-file-name name version))
(sha256
(base32
"17pjvprmdrx4h6bb1hhc98w9qi6ki7yl57f090n9kbhswxqfs7s3"))
(patches (search-patches "libgit2-mtime-0.patch"))
(modules '((guix build utils)))
(snippet '(begin
;; Remove bundled software.
(delete-file-recursively "deps")
#t))))
(build-system cmake-build-system)
(outputs '("out" "debug"))
(arguments
`(#:tests? #t ; Run the test suite (this is the default)
#:configure-flags '("-DUSE_SHA1DC=ON") ; SHA-1 collision detection
#:phases
(modify-phases %standard-phases
(add-after 'unpack 'fix-hardcoded-paths
(lambda _
(substitute* "tests/repo/init.c"
(("#!/bin/sh") (string-append "#!" (which "sh"))))
(substitute* "tests/clar/fs.h"
(("/bin/cp") (which "cp"))
(("/bin/rm") (which "rm")))
#t))
;; Run checks more verbosely.
(replace 'check
(lambda _ (invoke "./libgit2_clar" "-v" "-Q")))
(add-after 'unpack 'make-files-writable-for-tests
(lambda _ (for-each make-file-writable (find-files "." ".*")))))))
(inputs
`(("libssh2" ,libssh2)
("http-parser" ,http-parser)
("python" ,python-wrapper)))
(native-inputs
`(("pkg-config" ,pkg-config)))
(propagated-inputs
;; These two libraries are in 'Requires.private' in libgit2.pc.
`(("openssl" ,openssl)
("zlib" ,zlib)))
(home-page "https://libgit2.github.com/")
(synopsis "Library providing Git core methods")
(description
"Libgit2 is a portable, pure C implementation of the Git core methods
provided as a re-entrant linkable library with a solid API, allowing you to
write native speed custom Git applications in any language with bindings.")
;; GPLv2 with linking exception
(license license:gpl2))))</code></pre><p>(In those cases were you only want to tweak a few fields from a package
definition, you should rely on inheritance instead of copy-pasting everything.
See below.)</p><p>Let's discuss those fields in depth.</p><h3><code>git-fetch</code> method</h3><p>Unlike the <code>url-fetch</code> method, <code>git-fetch</code> expects a <code>git-reference</code> which takes
a Git repository and a commit. The commit can be any Git reference such as
tags, so if the <code>version</code> is tagged, then it can be used directly. Sometimes
the tag is prefixed with a <code>v</code>, in which case you'd use <code>(commit (string-append "v" version))</code>.</p><p>To ensure that the source code from the Git repository is stored in a unique
directory with a readable name we use <code>(file-name (git-file-name name version))</code>.</p><p>Note that there is also a <code>git-version</code> procedure that can be used to derive the
version when packaging programs for a specific commit.</p><h3>Snippets</h3><p>Snippets are quoted (i.e. non-evaluated) Scheme code that are a means of patching
the source. They are a Guix-y alternative to the traditional <code>.patch</code> files.
Because of the quote, the code in only evaluated when passed to the Guix daemon
for building.</p><p>There can be as many snippet as needed.</p><p>Snippets might need additional Guile modules which can be imported from the
<code>modules</code> field.</p><h3>Inputs</h3><p>First, a syntactic comment: See the quasi-quote / comma syntax?</p><pre><code class="language-scheme">(native-inputs
`(("pkg-config" ,pkg-config)))</code></pre><p>is equivalent to</p><pre><code class="language-scheme">(native-inputs
(list (list "pkg-config" pkg-config)))</code></pre><p>You'll mostly see the former because it's shorter.</p><p>There are 3 different input types. In short:</p><ul><li><strong>native-inputs:</strong> Required for building but not runtime – installing a package
through a substitute won't install these inputs.</li><li><strong>inputs:</strong> Installed in the store but not in the profile, as well as being
present at build time.</li><li><strong>propagated-inputs:</strong> Installed in the store and in the profile, as well as
being present at build time.</li></ul><p>See <a href="https://www.gnu.org/software/guix/manual/en/html_node/package-Reference.html">the package reference in the manual</a> for more details.</p><p>The distinction between the various inputs is important: if a dependency can be
handled as an <em>input</em> instead of a <em>propagated input</em>, it should be done so, or
else it "pollutes" the user profile for no good reason.</p><p>For instance, a user installing a graphical program that depends on a
command line tool might only be interested in the graphical part, so there is no
need to force the command line tool into the user profile. The dependency is a
concern to the package, not to the user. <em>Inputs</em> make it possible to handle
dependencies without bugging the user by adding undesired executable files (or
libraries) to their profile.</p><p>Same goes for <em>native-inputs</em>: once the program is installed, build-time
dependencies can be safely garbage-collected.
It also matters when a substitute is available, in which case only the <em>inputs</em>
and <em>propagated inputs</em> will be fetched: the <em>native inputs</em> are not required to
install a package from a substitute.</p><h3>Outputs</h3><p>Just like how a package can have multiple inputs, it can also produce multiple
outputs.</p><p>Each output corresponds to a separate directory in the store.</p><p>The user can choose which output to install; this is useful to save space or
to avoid polluting the user profile with unwanted executables or libraries.</p><p>Output separation is optional. When the <code>outputs</code> field is left out, the
default and only output (the complete package) is referred to as <code>"out"</code>.</p><p>Typical separate output names include <code>debug</code> and <code>doc</code>.</p><p>It's advised to separate outputs only when you've shown it's worth it: if the
output size is significant (compare with <code>guix size</code>) or in case the package is
modular.</p><h3>Build system arguments</h3><p>The <code>arguments</code> is a keyword-value list used to configure the build process.</p><p>The simplest argument <code>#:tests?</code> can be used to disable the test suite when
building the package. This is mostly useful when the package does not feature
any test suite. It's strongly recommended to keep the test suite on if there is
one.</p><p>Another common argument is <code>:make-flags</code>, which specifies a list of flags to
append when running make, as you would from the command line. For instance, the
following flags</p><pre><code class="language-scheme">#:make-flags (list (string-append "prefix=" (assoc-ref %outputs "out"))
"CC=gcc")</code></pre><p>translate into</p><pre><code class="language-sh">$ make CC=gcc prefix=/gnu/store/...-<out></code></pre><p>This sets the C compiler to <code>gcc</code> and the <code>prefix</code> variable (the installation
directory in Make parlance) to <code>(assoc-ref %outputs "out")</code>, which is a build-stage
global variable pointing to the destination directory in the store (something like
<code>/gnu/store/...-my-libgit2-20180408</code>).</p><p>Similarly, it's possible to set the "configure" flags.</p><pre><code class="language-scheme">#:configure-flags '("-DUSE_SHA1DC=ON")</code></pre><p>The <code>%build-inputs</code> variable is also generated in scope. It's an association
table that maps the input names to their store directories.</p><p>The <code>phases</code> keyword lists the sequential steps of the build system. Typically
phases include <code>unpack</code>, <code>configure</code>, <code>build</code>, <code>install</code> and <code>check</code>. To know
more about those phases, you need to work out the appropriate build system
definition in <code>$GUIX_CHECKOUT/guix/build/gnu-build-system.scm</code>:</p><pre><code class="language-scheme">(define %standard-phases
;; Standard build phases, as a list of symbol/procedure pairs.
(let-syntax ((phases (syntax-rules ()
((_ p ...) `((p . ,p) ...)))))
(phases set-SOURCE-DATE-EPOCH set-paths install-locale unpack
bootstrap
patch-usr-bin-file
patch-source-shebangs configure patch-generated-file-shebangs
build check install
patch-shebangs strip
validate-runpath
validate-documentation-location
delete-info-dir-file
patch-dot-desktop-files
install-license-files
reset-gzip-timestamps
compress-documentation)))</code></pre><p>Or from the REPL:</p><pre><code class="language-scheme">> (add-to-load-path "/path/to/guix/checkout")
> ,module (guix build gnu-build-system)
> (map first %standard-phases)
(set-SOURCE-DATE-EPOCH set-paths install-locale unpack bootstrap
patch-usr-bin-file patch-source-shebangs configure
patch-generated-file-shebangs build check install patch-shebangs strip
validate-runpath validate-documentation-location delete-info-dir-file
patch-dot-desktop-files install-license-files reset-gzip-timestamps
compress-documentation)</code></pre><p>If you want to know more about what happens during those phases, consult the
associated procedures.</p><p>For instance, as of this writing the definition of <code>unpack</code> for the GNU build
system is</p><pre><code class="language-scheme">(define* (unpack #:key source #:allow-other-keys)
"Unpack SOURCE in the working directory, and change directory within the
source. When SOURCE is a directory, copy it in a sub-directory of the current
working directory."
(if (file-is-directory? source)
(begin
(mkdir "source")
(chdir "source")
;; Preserve timestamps (set to the Epoch) on the copied tree so that
;; things work deterministically.
(copy-recursively source "."
#:keep-mtime? #t))
(begin
(if (string-suffix? ".zip" source)
(invoke "unzip" source)
(invoke "tar" "xvf" source))
(chdir (first-subdirectory "."))))
#t)</code></pre><p>Note the <code>chdir</code> call: it changes the working directory to where the source was
unpacked.
Thus every phase following the <code>unpack</code> will use the source as a working
directory, which is why we can directly work on the source files.
That is to say, unless a later phase changes the working directory to something
else.</p><p>We modify the list of <code>%standard-phases</code> of the build system with the
<code>modify-phases</code> macro as per the list of specified modifications, which may have
the following forms:</p><ul><li><code>(add-before PHASE NEW-PHASE PROCEDURE)</code>: Run <code>PROCEDURE</code> named <code>NEW-PHASE</code> before <code>PHASE</code>.</li><li><code>(add-after PHASE NEW-PHASE PROCEDURE)</code>: Same, but afterwards.</li><li><code>(replace PHASE PROCEDURE)</code>.</li><li><code>(delete PHASE)</code>.</li></ul><p>The <code>PROCEDURE</code> supports the keyword arguments <code>inputs</code> and <code>outputs</code>. Each
input (whether <em>native</em>, <em>propagated</em> or not) and output directory is referenced
by their name in those variables. Thus <code>(assoc-ref outputs "out")</code> is the store
directory of the main output of the package. A phase procedure may look like
this:</p><pre><code class="language-scheme">(lambda* (#:key inputs outputs #:allow-other-keys)
(let (((bash-directory (assoc-ref inputs "bash"))
(output-directory (assoc-ref outputs "out"))
(doc-directory (assoc-ref outputs "doc"))
; ...
#t)</code></pre><p>The procedure must return <code>#t</code> on success. It's brittle to rely on the return
value of the last expression used to tweak the phase because there is no
guarantee it would be a <code>#t</code>. Hence the trailing <code>#t</code> to ensure the right value
is returned on success.</p><h3>Code staging</h3><p>The astute reader may have noticed the quasi-quote and comma syntax in the
argument field. Indeed, the build code in the package declaration should not be
evaluated on the client side, but only when passed to the Guix daemon. This
mechanism of passing code around two running processes is called <a href="https://arxiv.org/abs/1709.00833">code staging</a>.</p><h3>"Utils" functions</h3><p>When customizing <code>phases</code>, we often need to write code that mimics the
equivalent system invocations (<code>make</code>, <code>mkdir</code>, <code>cp</code>, etc.) commonly used during
regular "Unix-style" installations.</p><p>Some like <code>chmod</code> are native to Guile. See the <a href="https://www.gnu.org/software/guile/manual/html_node">Guile reference manual</a> for a
complete list.</p><p>Guix provides additional helper functions which prove especially handy in the
context of package management.</p><p>Some of those functions can be found in
<code>$GUIX_CHECKOUT/guix/guix/build/utils.scm</code>. Most of them mirror the behaviour
of the traditional Unix system commands:</p><ul><li><strong>which:</strong> Like the <code>which</code> system command.</li><li><strong>find-files:</strong> Akin to the <code>find</code> system command.</li><li><strong>mkdir-p:</strong> Like <code>mkdir -p</code>, which creates all parents as needed.</li><li><strong>install-file:</strong> Similar to <code>install</code> when installing a file to a (possibly
non-existing) directory. Guile has <code>copy-file</code> which works
like <code>cp</code>.</li><li><strong>copy-recursively:</strong> Like <code>cp -r</code>.</li><li><strong>delete-file-recursively:</strong> Like <code>rm -rf</code>.</li><li><strong>invoke:</strong> Run an executable. This should be used instead of <code>system*</code>.</li><li><strong>with-directory-excursion:</strong> Run the body in a different working directory,
then restore the previous working directory.</li><li><strong>substitute*:</strong> A "sed-like" function.</li></ul><h3>Module prefix</h3><p>The license in our last example needs a prefix: this is because of how the
<code>license</code> module was imported in the package, as <code>#:use-module ((guix licenses) #:prefix license:)</code>. The <a href="https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Modules.html">Guile module import mechanism</a> gives the user full
control over namespacing: this is needed to avoid clashes between, say, the
<code>zlib</code> variable from <code>licenses.scm</code> (a <em>license</em> value) and the <code>zlib</code> variable
from <code>compression.scm</code> (a <em>package</em> value).</p><h1>Other build systems</h1><p>What we've seen so far covers the majority of packages using a build system
other than the <code>trivial-build-system</code>. The latter does not automate anything
and leaves you to build everything manually. This can be more demanding and we
won't cover it here for now, but thankfully it is rarely necessary to fall back
on this system.</p><p>For the other build systems, such as ASDF, Emacs, Perl, Ruby and many more, the
process is very similar to the GNU build system except for a few specialized
arguments.</p><p>Learn more about build systems in</p><ul><li><a href="https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html#Build-Systems">the manual, section 4.2 Build systems</a>,</li><li>the source code in the <code>$GUIX_CHECKOUT/guix/build</code> and
<code>$GUIX_CHECKOUT/guix/build-system</code> directories.</li></ul><h1>Programmable and automated package definition</h1><p>We can't repeat it enough: having a full-fledged programming language at hand
empowers us in ways that reach far beyond traditional package management.</p><p>Let's illustrate this with some awesome features of Guix!</p><h3>Recursive importers</h3><p>You might find some build systems good enough that there is little to do at all
to write a package, to the point that it becomes repetitive and tedious after a
while. A <em>raison d'être</em> of computers is to replace human beings at those
boring tasks. So let's tell Guix to do this for us and create the package
definition of an R package from CRAN (the output is trimmed for conciseness):</p><pre><code class="language-sh">$ guix import cran --recursive walrus
(define-public r-mc2d
; ...
(license gpl2+)))
(define-public r-jmvcore
; ...
(license gpl2+)))
(define-public r-wrs2
; ...
(license gpl3)))
(define-public r-walrus
(package
(name "r-walrus")
(version "1.0.3")
(source
(origin
(method url-fetch)
(uri (cran-uri "walrus" version))
(sha256
(base32
"1nk2glcvy4hyksl5ipq2mz8jy4fss90hx6cq98m3w96kzjni6jjj"))))
(build-system r-build-system)
(propagated-inputs
`(("r-ggplot2" ,r-ggplot2)
("r-jmvcore" ,r-jmvcore)
("r-r6" ,r-r6)
("r-wrs2" ,r-wrs2)))
(home-page "https://github.com/jamovi/walrus")
(synopsis "Robust Statistical Methods")
(description
"This package provides a toolbox of common robust statistical tests, including robust descriptives, robust t-tests, and robust ANOVA. It is also available as a module for 'jamovi' (see <https://www.jamovi.org> for more information). Walrus is based on the WRS2 package by Patrick Mair, which is in turn based on the scripts and work of Rand Wilcox. These analyses are described in depth in the book 'Introduction to Robust Estimation & Hypothesis Testing'.")
(license gpl3)))</code></pre><p>The recursive importer won't import packages for which Guix already has package
definitions, except for the very first.</p><p>Not all applications can be packaged this way, only those relying on a select
number of supported systems. Read about the full list of importers in the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-import.html">guix
import section</a> of the manual.</p><h3>Automatic update</h3><p>Guix can be smart enough to check for updates on systems it knows. It can
report outdated package definitions with</p><pre><code class="language-sh">$ guix refresh hello</code></pre><p>In most cases, updating a package to a newer version requires little more than
changing the version number and the checksum. Guix can do that automatically as
well:</p><pre><code class="language-sh">$ guix refresh hello --update</code></pre><h3>Inheritance</h3><p>If you've started browsing the existing package definitions, you might have
noticed that a significant number of them have a <code>inherit</code> field:</p><pre><code class="language-scheme">(define-public adwaita-icon-theme
(package (inherit gnome-icon-theme)
(name "adwaita-icon-theme")
(version "3.26.1")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnome/sources/" name "/"
(version-major+minor version) "/"
name "-" version ".tar.xz"))
(sha256
(base32
"17fpahgh5dyckgz7rwqvzgnhx53cx9kr2xw0szprc6bnqy977fi8"))))
(native-inputs
`(("gtk-encode-symbolic-svg" ,gtk+ "bin")))))</code></pre><p>All unspecified fields are inherited from the parent package. This is very
convenient to create alternative packages, for instance with different source,
version or compilation options.</p><h1>Getting help</h1><p>Sadly, some applications can be tough to package. Sometimes they need a patch to
work with the non-standard filesystem hierarchy enforced by the store.
Sometimes the tests won't run properly. (They can be skipped but this is not
recommended.) Other times the resulting package won't be reproducible.</p><p>Should you be stuck, unable to figure out how to fix any sort of packaging
issue, don't hesitate to ask the community for help.</p><p>See the <a href="https://www.gnu.org/software/guix/contact/">Guix homepage</a> for information on the mailing lists, IRC, etc.</p><h1>Conclusion</h1><p>This tutorial was a showcase of the sophisticated package management that Guix
boasts. At this point we have mostly restricted this introduction to the
<code>gnu-build-system</code> which is a core abstraction layer on which more advanced
abstractions are based.</p><p>Where do we go from here? Next we ought to dissect the innards of the build
system by removing all abstractions, using the <code>trivial-build-system</code>: this
should give us a thorough understanding of the process before investigating some
more advanced packaging techniques and edge cases.</p><p>Other features worth exploring are the interactive editing and debugging
capabilities of Guix provided by the Guile REPL.</p><p>Those fancy features are completely optional and can wait; now is a good time
to take a well-deserved break. With what we've introduced here you should be
well armed to package lots of programs. You can get started right away and
hopefully we will see your contributions soon!</p><h1>References</h1><ul><li><p>The <a href="https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html">package reference in the manual</a></p></li><li><p><a href="https://gitlab.com/pjotrp/guix-notes/blob/master/HACKING.org">Pjotr’s hacking guide to GNU Guix</a></p></li><li><p><a href="https://www.gnu.org/software/guix/guix-ghm-andreas-20130823.pdf">"GNU Guix: Package without a
scheme!"</a>,
by Andreas Enge</p></li></ul><h1>Notices</h1><p><a href="http://creativecommons.org/licenses/by-sa/4.0/"><img src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" alt="CC-BY-SA" title="CC-BY-SA
4.0" /></a></p><p>This work is licensed under a Creative Commons Attribution-ShareAlike 4.0
International License.</p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager for the GNU system. The Guix System Distribution or GuixSD is
an advanced distribution of the GNU system that relies on GNU Guix and
<a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects the user's
freedom</a>.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. Guix uses low-level
mechanisms from the Nix package manager, except that packages are
defined as native <a href="https://www.gnu.org/software/guile">Guile</a> modules,
using extensions to the <a href="http://schemers.org">Scheme</a> language. GuixSD
offers a declarative approach to operating system configuration
management, and is highly customizable and hackable.</p><p>GuixSD can be used on an i686, x86_64, ARMv7, and AArch64 machines. It
is also possible to use Guix on top of an already installed GNU/Linux
system, including on mips64el and aarch64.</p>https://guix.gnu.org/blog/2018/multi-dimensional-transactions-and-rollbacks-oh-my/Multi-dimensional transactions and rollbacks, oh my!Ludovic Courtès2018-07-24T14:30:00Z2018-07-24T14:30:00Z One of the highlights of version
0.15.0
was the overhaul of guix pull ,
the command that updates Guix and its package collection. In Debian
terms, you can think of guix pull as: apt-get update && apt-get install apt Let’s be frank, guix pull does not yet run as quickly as this
apt-get command—in the “best case”, when pre-built binaries are
available, it currently runs in about 1m30s on a recent laptop. More
about the performance story in a future post… One of the key features of…<p>One of the <a href="https://www.gnu.org/software/guix/blog/2018/gnu-guix-and-guixsd-0.15.0-released/">highlights of version
0.15.0</a>
was the overhaul of <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-pull.html"><code>guix pull</code></a>,
the command that updates Guix and its package collection. In Debian
terms, you can think of <code>guix pull</code> as:</p><pre><code>apt-get update && apt-get install apt</code></pre><p>Let’s be frank, <code>guix pull</code> does not yet run as quickly as this
<code>apt-get</code> command—in the “best case”, when pre-built binaries are
available, it currently runs in about 1m30s on a recent laptop. More
about the performance story in a future post…</p><p>One of the key features of the new <code>guix pull</code> is the ability to <em>roll
back</em> to previous versions of Guix. That’s a distinguishing feature
that opens up new possibilities.</p><h1>“Profile generations”</h1><p>Transactional upgrades and rollbacks have been a distinguishing feature
of Guix since Day 1. They come for free as a consequence of the
functional package management model inherited from the Nix package
manager. To many users, this alone is enough to justify using a
functional package manager: if an upgrade goes wrong, you can always
roll back. Let’s recap how this all works.</p><p>As a user, you install packages in your own <em>profile</em>, which defaults to
<code>~/.guix-profile</code>. Then from time to time you update Guix and its
package collection:</p><pre><code>$ guix pull</code></pre><p>This updates <code>~/.config/guix/current</code>, giving you an updated <code>guix</code>
executable along with an updated set of packages. You can now upgrade
the packages that are in your profile:</p><pre><code>$ guix package -u
The following packages will be upgraded:
diffoscope 93 → 96 /gnu/store/…-diffoscope-96
emacs 25.3 → 26.1 /gnu/store/…-emacs-26.1
gimp 2.8.22 → 2.10.4 /gnu/store/…-gimp-2.10.4
gnupg 2.2.7 → 2.2.9 /gnu/store/…-gnupg-2.2.9</code></pre><p>The upgrade creates a new <em>generation</em> of your profile—the previous
generation of your profile, with diffoscope 93, emacs 25.3, and so on is
still around. You can list profile generations:</p><pre><code>$ guix package --list-generations
Generation 1 Jun 08 2018 20:06:21
diffoscope 93 out /gnu/store/…-diffoscope-93
emacs 25.3 out /gnu/store/…-emacs-25.3
gimp 2.8.22 out /gnu/store/…-gimp-2.8.22
gnupg 2.2.7 out /gnu/store/…-gnupg-2.2.7
python 3.6.5 out /gnu/store/…-python-3.6.5
Generation 2 Jul 12 2018 12:42:08 (current)
- diffoscope 93 out /gnu/store/…-diffoscope-93
- emacs 25.3 out /gnu/store/…-emacs-25.3
- gimp 2.8.22 out /gnu/store/…-gimp-2.8.22
- gnupg 2.2.7 out /gnu/store/…-gnupg-2.2.7
+ diffoscope 96 out /gnu/store/…-diffoscope-96
+ emacs 26.1 out /gnu/store/…-emacs-26.1
+ gimp 2.10.4 out /gnu/store/…-gimp-2.10.4
+ gnupg 2.2.9 out /gnu/store/…-gnupg-2.2.9</code></pre><p>That shows our two generations with the diff between Generation 1 and
Generation 2. We can at any time run <code>guix package --roll-back</code> and get
our previous versions of gimp, emacs, and so on. Each generation is
just a bunch of symlinks to those packages, so what we have looks like
this:</p><p><img src="https://www.gnu.org/software/guix/static/blog/img/guix-pull1.png" alt="Image of the profile generations." /></p><p>Notice that python was not updated, so it’s shared between both
generations. And of course, all the dependencies that didn’t change in
between—e.g., the C library—are shared among all packages.</p><h1><code>guix pull</code> generations</h1><p>Like I wrote above, <code>guix pull</code> brings the latest set of package
definitions from Git <code>master</code>. The Guix package collection usually
contains only the latest version of each package; for example, current
<code>master</code> only has version 26.1 of Emacs and version 2.10.4 of the GIMP
(there are notable exceptions such as GCC or Python.) Thus, <code>guix package -i gimp</code>, from today’s master, can only install gimp 2.10.4.
Often, that’s not a problem: you can keep old profile generations
around, so if you really need that older version of Emacs, you can run
it from your previous generation.</p><p>Still, having <code>guix pull</code> keep track of the changes to Guix and its
package collection is useful. Starting from 0.15.0, <code>guix pull</code> creates
a new generation, just like <code>guix package</code> does. After you’ve run <code>guix pull</code>, you can now list Guix generations as well:</p><pre><code>$ guix pull -l
Generation 10 Jul 14 2018 00:02:03
guix 27f7cbc
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: origin/master
commit: 27f7cbc91d1963118e44b14d04fcc669c9618176
Generation 11 Jul 20 2018 10:44:46
guix 82549f2
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: origin/master
commit: 82549f2328c59525584b92565846217c288d8e85
14 new packages: bsdiff, electron-cash, emacs-adoc-mode,
emacs-markup-faces, emacs-rust-mode, inchi, luakit, monero-gui,
nethack, openbabel, qhull, r-txtplot, stb-image, stb-image-write
52 packages upgraded: angband@4.1.2, aspell-dict-en@2018.04.16-0,
assimp@4.1.0, bitcoin-core@0.16.1, botan@2.7.0, busybox@1.29.1,
…
Generation 12 Jul 23 2018 15:22:52 (current)
guix fef7bab
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: origin/master
commit: fef7baba786a96b7a3100c9c7adf8b45782ced37
20 new packages: ccrypt, demlo, emacs-dired-du,
emacs-helm-org-contacts, emacs-ztree, ffmpegthumbnailer,
go-github-com-aarzilli-golua, go-github-com-kr-text,
go-github-com-mattn-go-colorable, go-github-com-mattn-go-isatty,
go-github-com-mgutz-ansi, go-github-com-michiwend-golang-pretty,
go-github-com-michiwend-gomusicbrainz, go-github-com-stevedonovan-luar,
go-github-com-wtolson-go-taglib, go-github-com-yookoala-realpath,
go-gitlab-com-ambrevar-damerau, go-gitlab-com-ambrevar-golua-unicode,
guile-pfds, u-boot-cubietruck
27 packages upgraded: c-toxcore@0.2.4, calibre@3.28.0,
emacs-evil-collection@20180721-2.5d739f5,
…</code></pre><p>The nice thing here is that <code>guix pull</code> provides high-level information
about the differences between two subsequent generations of Guix.</p><p>In the end, Generation 1 of our profile was presumably built with Guix
Generation 11, while Generation 2 of our profile was built with Guix
Generation 12. We have a clear mapping between Guix generations as
created by <code>guix pull</code> and profile generations as created with <code>guix package</code>:</p><p><img src="https://www.gnu.org/software/guix/static/blog/img/guix-pull3.png" alt="Image of the Guix generations." /></p><p>Each generation created by <code>guix pull</code> corresponds to one commit in the
Guix repo. Thus, if I go to another machine and run:</p><pre><code>$ guix pull --commit=fef7bab</code></pre><p>then I know that I get the exact same Guix instance as my Generation 12
above. From there I can install diffoscope, emacs, etc. and I know I’ll
get the exact same binaries as those I have above, thanks to
<a href="https://reproducible-builds.org/docs/definition/">reproducible builds</a>.</p><p>These are very strong guarantees in terms of reproducibility and
provenance tracking—properties that are
<a href="https://github.com/canonical-websites/snapcraft.io/issues/651">typically</a>
<a href="https://lwn.net/Articles/752982/">missing</a> from “applications bundles”
à la Docker.</p><p>In addition, you can easily run an older Guix. For instance, this is
how you would install the version of gimp that was current as of
Generation 10:</p><pre><code>$ ~/.config/guix/current-10-link/bin/guix package -i gimp</code></pre><p>At this point your profile contains gimp coming from an old Guix along
with packages installed from the latest Guix. Past and present coexist
in the same profile. The historical dimension of the profile no longer
matches exactly the history of Guix itself.</p><h1>Composing Guix revisions</h1><p>Some people have expressed interest in being able to compose packages
coming from different revisions of Guix—say to create a profile
containing old versions of Python and NumPy, but also the latest and
greatest GCC. It may seem far-fetched but it has very real
applications: there are large collections of scientific packages and in
particular bioinformatics packages that don’t move as fast as our
beloved flagship free software packages, and users may require ancient
versions of some of the tools.</p><p>We could keep old versions of many packages but maintainability costs
would grow exponentially. Instead, Guix users can take advantage of the
version control history of Guix itself to mix and match packages coming
from different revisions of Guix. As shown above, it’s already possible
to achieve this by running the <code>guix</code> program off the generation of
interest. It does the job, but can we do better?</p><p>In the process of enhancing <code>guix pull</code> we developed a high-level API
that allows an instance of Guix to “talk” to a different instance of
Guix—<a href="https://debbugs.gnu.org/cgi/bugreport.cgi?bug=32115">an
“inferior”</a>. It’s
what allows <code>guix pull</code> to display the list of packages that were added
or upgraded between two revisions. The next logical step will be to
provide seamless integration of packages coming from an inferior. That
way, users would be able to refer to “past” package graphs right from <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#index-profile-manifest">a
profile
manifest</a>
or from the command-line. Future work!</p><h1>On coupling</h1><p>The time traveler in you might be wondering: Why are package definitions
coupled with the package manager, doesn’t it make it harder to compose
packages coming from different revisions? Good point!</p><p>Tight coupling certainly complicates this kind of composition: we can’t
just have any revision of Guix load package definitions from any other
revision; this could fail altogether, or it could provide a different
build result. Another potential issue is that <code>guix pull</code>ing an older
revision not only gives you an older set of packages, it also gives you
older tools, bug-for-bug.</p><p>The reason for this coupling is that a package definition <a href="https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html">like this
one</a>
doesn’t exist in a vacuum. Its meaning is defined by the implementation
of <a href="https://www.gnu.org/software/guix/manual/en/html_node/package-Reference.html">package
objects</a>,
by
<a href="https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html"><code>gnu-build-system</code></a>,
by a number of <a href="https://arxiv.org/abs/1305.4584">lower-level</a>
<a href="https://hal.inria.fr/hal-01580582/en">abstractions</a> that are all
defined as extensions of the Scheme language in Guix itself, and
ultimately by <a href="https://www.gnu.org/software/guile/">Guile</a>, which
implements the language Guix is written in. Each instance created by
<code>guix pull</code> brings all these components. Because Guix is implemented as
a set of programming language extensions and libraries, that package
definitions depend on all these parts becomes manifest. Instead of
being frozen, the APIs and package definitions evolve together, which
gives us developers a lot of freedom on the changes we can make.</p><p><a href="https://nixos.org/nix/">Nix</a> results from a different design choice.
Nix-the-package-manager implements the Nix language, which acts as a
“frozen” interface. Package definitions in Nixpkgs are written in that
language, and a given version of Nix can <em>possibly</em> interpret both
current and past package definitions without further ado. The Nix
language does evolve though, so at one point an old Nix inevitably
<a href="https://github.com/NixOS/nixpkgs/blob/master/lib/minver.nix">becomes unable to evaluate a new
Nixpkgs</a>,
and <em>vice versa</em>.</p><p>These two approaches make different tradeoffs. Nix’ loose coupling
simplifies the implementation and makes it easy to compose old and new
package definitions, to some extent; Guix’ tight coupling makes such
composition more difficult to implement, but it leaves developers more
freedom and, we hope, may support “time travels” over longer period of
times. Time will tell!</p><h1>It’s like driving a DeLorean</h1><p><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/1/15/TeamTimeCar.com-BTTF_DeLorean_Time_Machine-OtoGodfrey.com-JMortonPhoto.com-05.jpg/800px-TeamTimeCar.com-BTTF_DeLorean_Time_Machine-OtoGodfrey.com-JMortonPhoto.com-05.jpg" alt="Inside the cabin of the DeLorean time machine in “Back to the Future.”" /></p><p><em><a href="https://commons.wikimedia.org/wiki/File:TeamTimeCar.com-BTTF_DeLorean_Time_Machine-OtoGodfrey.com-JMortonPhoto.com-05.jpg">Picture</a>
of a DeLorean cabin by Oto Godfrey and Justin Morton, under <a href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">CC-BY-SA
4.0</a>.</em></p><p>That profile generations are kept around already gave users a time
machine of sorts—you can always roll back to a previous state of your
software environment. With the addition of roll-back support for <code>guix pull</code>, this adds another dimension to the time machine: you can
roll-back to a previous state of Guix itself and from there create
alternative futures or even mix bits from the past with bits from the
present. We hope you’ll enjoy it!</p>https://guix.gnu.org/blog/2018/customize-guixsd-use-stock-ssh-agent-everywhere/Customize GuixSD: Use Stock SSH Agent Everywhere!Chris Marusich2018-05-26T17:00:00Z2018-05-26T17:00:00Z I frequently use SSH. Since I don't like typing my password all the
time, I use an SSH agent. Originally I used the GNOME Keyring as my
SSH agent, but recently I've switched to using the ssh-agent from
OpenSSH. I accomplished this by doing the following two things: Replace the default GNOME Keyring with a custom-built version that
disables the SSH agent feature. Start my desktop session with OpenSSH's ssh-agent so that it's
always available to any applications in my desktop session. Below, I'll show you in…<p>I frequently use SSH. Since I don't like typing my password all the
time, I use an SSH agent. Originally I used the GNOME Keyring as my
SSH agent, but recently I've switched to using the <code>ssh-agent</code> from
OpenSSH. I accomplished this by doing the following two things:</p><ul><li><p>Replace the default GNOME Keyring with a custom-built version that
disables the SSH agent feature.</p></li><li><p>Start my desktop session with OpenSSH's <code>ssh-agent</code> so that it's
always available to any applications in my desktop session.</p></li></ul><p>Below, I'll show you in detail how I did this. In addition to being
useful for anyone who wants to use OpenSSH's <code>ssh-agent</code> in GuixSD, I
hope this example will help to illustrate how GuixSD enables you to
customize your entire system to be just the way you want it!</p><h1>The Problem: GNOME Keyring Can't Handle My SSH Keys</h1><p>On GuixSD, I like to use the <a href="https://www.gnome.org">GNOME desktop
environment</a>. GNOME is just one of <a href="https://www.gnu.org/software/guix/manual/en/html_node/Desktop-Services.html">the
various desktop environments that GuixSD
supports</a>.
By default, the GNOME desktop environment on GuixSD comes with a lot
of goodies, including the <a href="https://wiki.gnome.org/Projects/GnomeKeyring">GNOME
Keyring</a>, which is
GNOME's integrated solution for securely storing secrets, passwords,
keys, and certificates.</p><p>The GNOME Keyring has many useful features. One of those is <a href="https://wiki.gnome.org/Projects/GnomeKeyring/Ssh">its SSH
Agent feature</a>.
This feature allows you to use the GNOME Keyring as an SSH agent.
This means that when you invoke a command like <code>ssh-add</code>, it will add
the private key identities to the GNOME Keyring. Usually this is
quite convenient, since it means that GNOME users basically get an SSH
agent for free!</p><p>Unfortunately, up until <a href="https://www.gnome.org/news/2018/03/gnome-3-28-released/">GNOME 3.28 (the current
release)</a>,
the GNOME Keyring's SSH agent implementation was not as complete as
the stock SSH agent from OpenSSH. As a result, <a href="https://bugzilla.gnome.org/show_bug.cgi?id=775981">earlier versions of
GNOME Keyring did not support many use
cases</a>. This was a
problem for me, since GNOME Keyring couldn't read my modern SSH keys.
To make matters worse, by design the SSH agent for GNOME Keyring and
OpenSSH both use the same environment variables (e.g.,
<code>SSH_AUTH_SOCK</code>). This makes it difficult to use OpenSSH's
<code>ssh-agent</code> everywhere within my GNOME desktop environment.</p><p>Happily, starting with GNOME 3.28, <a href="https://bugzilla.gnome.org/show_bug.cgi?id=775981">GNOME Keyring delegates all SSH
agent functionality to the stock SSH agent from
OpenSSH</a>. They
have removed their custom implementation entirely. This means that
today, I could solve my problem simply by using the most recent
version of GNOME Keyring. I'll probably do just that when the new
release gets included in Guix. However, when I first encountered this
problem, GNOME 3.28 hadn't been released yet, so the only option
available to me was to customize GNOME Keyring or remove it entirely.</p><p>In any case, I'm going to show you how I solved this problem by
modifying the default GNOME Keyring from the Guix package collection.
The same ideas can be used to customize any package, so hopefully it
will be a useful example. And what if you don't use GNOME, but you do
want to use OpenSSH's <code>ssh-agent</code>? In that case, you may still need
to customize your GuixSD system a little bit. Let me show you how!</p><h1>The Solution: <code>~/.xsession</code> and a Custom GNOME Keyring</h1><p>The goal is to make OpenSSH's <code>ssh-agent</code> available everywhere when we
log into our GNOME desktop session. First, we must arrange for
<code>ssh-agent</code> to be running whenever we're logged in.</p><p>There are many ways to accomplish this. For example, I've seen people
implement shell code in their shell's start-up files which basically
manages their own <code>ssh-agent</code> process. However, I prefer to just
start <code>ssh-agent</code> once and not clutter up my shell's start-up files
with unnecessary code. So that's what we're going to do!</p><h1>Launch OpenSSH's <code>ssh-agent</code> in Your <code>~/.xsession</code></h1><p>By default, GuixSD uses the <a href="https://sourceforge.net/projects/slim.berlios">SLiM desktop
manager</a>. When you log
in, SLiM presents you with a menu of so-called "desktop sessions",
which correspond to the desktop environments you've declared in your
<a href="https://www.gnu.org/software/guix/manual/en/html_node/operating_002dsystem-Reference.html">operating system
declaration</a>.
For example, if you've added the
<a href="https://www.gnu.org/software/guix/manual/en/html_node/Desktop-Services.html">gnome-desktop-service</a>
to your operating system declaration, then you'll see an option for
GNOME at the SLiM login screen.</p><p>You can further customize your desktop session with the <code>~/.xsession</code>
file. The contract for this file in GuixSD is the same as it is for
many GNU/Linux distributions: <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/xorg.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n392">if it exists, then it will be
executed</a>.
The arguments passed to it will be the command line invocation that
would normally be executed to start the desktop session that you
selected from the SLiM login screen. Your <code>~/.xsession</code> is expected
to do whatever is necessary to customize and then start the specified
desktop environment. For example, when you select GNOME from the SLiM
login screen, your <code>~/.xsession</code> file will basically be executed like
this (for the exact execution mechanism, please refer to the source
code linked above):</p><pre><code class="language-shell">$ ~/.xsession gnome-session</code></pre><p>The upshot of all this is that the <code>~/.xsession</code> is an <em>ideal</em> place
to set up your SSH agent! If you start an SSH agent in your
<code>~/.xsession</code> file, you can have the SSH agent available everywhere,
automatically! Check it out: Put this into your <code>~/.xsession</code> file,
and make the file executable:</p><pre><code class="language-shell">#!/run/current-system/profile/bin/bash
exec ssh-agent "$@"</code></pre><p>When you invoke <code>ssh-agent</code> in this way, it executes the specified
program in an environment where commands like <code>ssh-add</code> just work. It
does this by setting environment variables such as <code>SSH_AUTH_SOCK</code>,
which programs like <code>ssh-add</code> find and use automatically. Because
GuixSD allows you to customize your desktop session like this, you can
use any SSH agent you want in any desktop environments that you want,
automatically!</p><p>Of course, if you're using GNOME Keyring version 3.27 or earlier (like
I was), then this isn't quite enough. In that case, the SSH agent
feature of GNOME Keyring will override the environment variables set
by OpenSSH's <code>ssh-agent</code>, so commands like <code>ssh-add</code> will wind up
communicating with the GNOME Keyring instead of the <code>ssh-agent</code> you
launched in your <code>~/.xsession</code>. This is bad because, as previously
mentioned, GNOME Keyring version 3.27 or earlier doesn't support as
many uses cases as OpenSSH's <code>ssh-agent</code>.</p><p>How can we work around this problem?</p><h1>Customize the GNOME Keyring</h1><p>One heavy-handed solution would be to remove GNOME Keyring entirely.
That would work, but then you would lose out on all the other great
features that it has to offer. Surely we can do better!</p><p>The GNOME Keyring documentation
<a href="https://wiki.gnome.org/Projects/GnomeKeyring/Ssh">explains</a> that one
way to disable the SSH agent feature is to include the
<code>--disable-ssh-agent</code> configure flag when building it. Thankfully,
Guix provides some ways to customize software in <em>exactly</em> this way!</p><p>Conceptually, we "just" have to do the following two things:</p><ul><li><p>Customize the existing <code>gnome-keyring</code> package.</p></li><li><p>Make the <code>gnome-desktop-service</code> use our custom <code>gnome-keyring</code>
package.</p></li></ul><h1>Create a Custom GNOME Keyring Package</h1><p>Let's begin by defining a custom <code>gnome-keyring</code> package, which we'll
call <code>gnome-keyring-sans-ssh-agent</code>. With Guix, we can do this in
less than ten lines of code:</p><pre><code class="language-scheme">(define-public gnome-keyring-sans-ssh-agent
(package
(inherit gnome-keyring)
(name "gnome-keyring-sans-ssh-agent")
(arguments
(substitute-keyword-arguments
(package-arguments gnome-keyring)
((#:configure-flags flags)
`(cons "--disable-ssh-agent" ,flags))))))</code></pre><p>Don't worry if some of that code is unclear at first. I'll clarify it
now!</p><p>In Guix, a <code><package></code> record like the one above is defined by a macro
called <code>define-record-type*</code> (<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/records.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n178">defined in the file guix/records.scm in
the Guix
source</a>).
It's similar to an <a href="https://www.gnu.org/software/guile/manual/en/html_node/SRFI_002d9-Records.html#SRFI_002d9-Records">SRFI-9
record</a>.
The <code>inherit</code> feature of this macro is very useful: it creates a new
copy of an existing record, overriding specific fields in the new copy
as needed.</p><p>In the above, we define <code>gnome-keyring-sans-ssh-agent</code> to be a copy of
the <code>gnome-keyring</code> package, and we use <code>inherit</code> to change the <code>name</code>
and <code>arguments</code> fields in that new copy. We also use the
<code>substitute-keyword-arguments</code> macro (<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/utils.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n345">defined in the file
guix/utils.scm in the Guix
source</a>)
to add <code>--disable-ssh-agent</code> to the list of <a href="https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html">configure
flags</a>
defined in the <code>gnome-keyring</code> package. The effect of this is to
define a new GNOME Keyring package that is built exactly the same as
the original, but in which the SSH agent is disabled.</p><p>I'll admit this code may seem a little opaque at first, but all code
does when you first learn it. Once you get the hang of things, you
can customize packages any way you can imagine. If you want to learn
more, you should read the docstrings for the <code>define-record-type*</code> and
<code>substitute-keyword-arguments</code> macros in the Guix source code. It's
also very helpful to <code>grep</code> the source code to see examples of how
these macros are used in practice. For example:</p><pre><code class="language-shell">$ # Search the currently installed Guix for the current user.
$ grep -r substitute-keyword-arguments ~/.config/guix/latest
$ # Search the Guix Git repository, assuming you've checked it out here.
$ grep -r substitute-keyword-arguments ~/guix</code></pre><h1>Use the Custom GNOME Keyring Package</h1><p>OK, we've created our own custom GNOME Keyring package. Great! Now,
how do we use it?</p><p>In GuixSD, the GNOME desktop environment is treated as a <a href="https://www.gnu.org/software/guix/manual/en/html_node/Services.html">system
service</a>. To
make GNOME use our custom GNOME Keyring package, we must somehow
customize the <code>gnome-desktop-service</code> (<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/desktop.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n795">defined in the file
gnu/services/desktop.scm</a>)
to use our custom package. How do we customize a service? Generally,
the answer depends on the service. Thankfully, many of GuixSD's
services, including the <code>gnome-desktop-service</code>, follow a similar
pattern. In this case, we "just" need to pass a custom
<code><gnome-desktop-configuration></code> record to the <code>gnome-desktop-service</code>
procedure in our operating system declaration, like this:</p><pre><code class="language-scheme">(operating-system
...
(services (cons*
(gnome-desktop-service
#:config my-gnome-desktop-configuration)
%desktop-services)))</code></pre><p>Here, the <code>cons*</code> procedure just adds the GNOME desktop service to the
<code>%desktop-services</code> list, returning the new list. For details, please
refer to <a href="https://www.gnu.org/software/guile/manual/en/html_node/List-Constructors.html#index-cons_002a">the Guile
manual</a>.</p><p>Now the question is: what should <code>my-gnome-desktop-configuration</code> be?
Well, if we examine <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/desktop.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n799">the definition of this record type in the Guix
source</a>,
we see the following:</p><pre><code class="language-scheme">(define-record-type* <gnome-desktop-configuration> gnome-desktop-configuration
make-gnome-desktop-configuration
gnome-desktop-configuration
(gnome-package gnome-package (default gnome)))</code></pre><p>The <code>gnome</code> package referenced here is a "meta" package: it exists
only to aggregate many GNOME packages together, including
<code>gnome-keyring</code>. To see <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gnome.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n5977">its
definition</a>,
we can simply invoke <code>guix edit gnome</code>, which <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-edit.html#Invoking-guix-edit">opens the file where
the package is
defined</a>:</p><pre><code class="language-scheme">(define-public gnome
(package
(name "gnome")
(version (package-version gnome-shell))
(source #f)
(build-system trivial-build-system)
(arguments '(#:builder (mkdir %output)))
(propagated-inputs
;; TODO: Add more packages according to:
;; <https://packages.debian.org/jessie/gnome-core>.
`(("adwaita-icon-theme" ,adwaita-icon-theme)
("baobab" ,baobab)
("font-cantarell" ,font-cantarell)
[... many packages omitted for brevity ...]
("gnome-keyring" ,gnome-keyring)
[... many packages omitted for brevity ...]
(synopsis "The GNU desktop environment")
(home-page "https://www.gnome.org/")
(description
"GNOME is the graphical desktop for GNU. It includes a wide variety of
applications for browsing the web, editing text and images, creating
documents and diagrams, playing media, scanning, and much more.")
(license license:gpl2+)))</code></pre><p>Apart from being a little long, this is <a href="https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html#Defining-Packages">just a normal package
definition</a>.
We can see that <code>gnome-keyring</code> is included in the list of
<code>propagated-inputs</code>. So, we need to create a replacement for the
<code>gnome</code> package that uses our <code>gnome-keyring-sans-ssh-agent</code> instead
of <code>gnome-keyring</code>. The following package definition accomplishes
that:</p><pre><code class="language-scheme">(define-public gnome-sans-ssh-agent
(package
(inherit gnome)
(name "gnome-sans-ssh-agent")
(propagated-inputs
(map (match-lambda
((name package)
(if (equal? name "gnome-keyring")
(list name gnome-keyring-sans-ssh-agent)
(list name package))))
(package-propagated-inputs gnome)))))</code></pre><p>As before, we use <code>inherit</code> to create a new copy of the <code>gnome</code>
package that overrides the original <code>name</code> and <code>propagated-inputs</code>
fields. Since Guix packages are just defined using good old scheme,
we can use existing language features like
<a href="https://www.gnu.org/software/guile/manual/en/html_node/List-Mapping.html#index-map"><code>map</code></a>
and
<a href="https://www.gnu.org/software/guile/manual/en/html_node/Pattern-Matching.html#Pattern-Matching"><code>match-lambda</code></a>
to manipulate the list of propagated inputs. The effect of the above
is to create a new package that is the same as the <code>gnome</code> package but
uses <code>gnome-keyring-sans-ssh-agent</code> instead of <code>gnome-keyring</code>.</p><p>Now that we have <code>gnome-sans-ssh-agent</code>, we can create a custom
<code><gnome-desktop-configuration></code> record and pass it to the
<code>gnome-desktop-service</code> procedure as follows:</p><pre><code class="language-scheme">(operating-system
...
(services (cons*
(gnome-desktop-service
#:config (gnome-desktop-configuration
(gnome-package gnome-sans-ssh-agent)))
%desktop-services)))</code></pre><h1>Wrapping It All Up</h1><p>Finally, you need to run the following commands as <code>root</code> to create
and boot into the new <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-system.html">system
generation</a>
(replace <code>MY-CONFIG</code> with the path to the customized operating system
configuration file):</p><pre><code class="language-shell"># guix system reconfigure MY-CONFIG
# reboot</code></pre><p>After you log into GNOME, any time you need to use SSH, the stock SSH
agent from OpenSSH that you started in your <code>~/.xsession</code> file will be
used instead of the GNOME Keyring's SSH agent. It just works! Note
that it still works even if you select a non-GNOME desktop session
(like XFCE) at the SLiM login screen, since the <code>~/.xsession</code> is not
tied to any particular desktop session,</p><p>In the unfortunate event that something went wrong and things just
aren't working when you reboot, don't worry: with GuixSD, you can
safely roll back to the previous system generation via <a href="https://www.gnu.org/software/guix/manual/en/html_node/Using-the-Configuration-System.html#index-roll_002dback_002c-of-the-operating-system">the usual
mechanisms</a>.
For example, you can run this from the command line to roll back:</p><pre><code class="language-shell"># guix system roll-back
# reboot</code></pre><p>This is one of the great benefits that comes from the fact that <a href="https://www.gnu.org/software/guix/manual/en/html_node/Introduction.html#Introduction">Guix
follows the functional software deployment
model</a>.
However, note that because the <code>~/.xsession</code> file (like many files in
your home directory) is not managed by Guix, you must manually undo
the changes that you made to it in order to roll back fully.</p><h1>Conclusion</h1><p>I hope this helps give you some ideas for how you can customize your
own GuixSD system to make it exactly what you want it to be. Not only
can you customize your desktop session via your <code>~/.xsession</code> file,
but Guix also provides tools for you to modify any of the default
packages or services to suit your specific needs.</p><p>Happy hacking!</p><h1>Notices</h1><p><a href="http://creativecommons.org/publicdomain/zero/1.0/"><img src="https://licensebuttons.net/p/zero/1.0/88x31.png" alt="CC0" title="CC0 1.0
Universal" /></a></p><p>To the extent possible under law, Chris Marusich has waived all
copyright and related or neighboring rights to this article,
"Customize GuixSD: Use Stock SSH Agent Everywhere!". This work is
published from: United States.</p><p>The views expressed in this article are those of Chris Marusich and do
not necessarily reflect the views of his past, present, or future
employers.</p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager for the GNU system. The Guix System Distribution or GuixSD is
an advanced distribution of the GNU system that relies on GNU Guix and
<a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects the user's
freedom</a>.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. Guix uses low-level
mechanisms from the Nix package manager, except that packages are
defined as native <a href="https://www.gnu.org/software/guile">Guile</a> modules,
using extensions to the <a href="http://schemers.org">Scheme</a> language. GuixSD
offers a declarative approach to operating system configuration
management, and is highly customizable and hackable.</p><p>GuixSD can be used on an i686, x86_64 and armv7 machines. It is also
possible to use Guix on top of an already installed GNU/Linux system,
including on mips64el and aarch64.</p>https://guix.gnu.org/blog/2014/guix-at-the-2014-gnu-hackers-meeting/Guix at the 2014 GNU Hackers MeetingLudovic Courtès2014-10-11T00:00:00+02002014-10-11T00:00:00+0200 The Guix talk of this summer's GNU Hackers Meeting is now available (get the slides ). DOWNLOAD VIDEO (WebM, 60 minutes) It gives an introduction to Guix from a user's viewpoint, and covers topics such as features for GNU maintainers, programming interfaces, declarative operating system configuration, status of the GNU/Hurd port, and the new Emacs and Web interfaces---with a bunch of demos. Do not miss other fine talks from the GHM . Many thanks to everyone who took care of the video recordings. …<p>The Guix talk of this summer's GNU Hackers Meeting is now available (get the <a href="https://www.gnu.org/software/guix/guix-ghm-20140815.pdf">slides</a>).</p><video src="https://audio-video.gnu.org/video/ghm2014/2014-08--courtes--were-building-the-gnu-system--ghm.webm" controls="controls"><div class="action-box centered-text"><a class="button-big" href="https://audio-video.gnu.org/video/ghm2014/2014-08--courtes--were-building-the-gnu-system--ghm.webm">DOWNLOAD VIDEO</a><p>(WebM, 60 minutes)</p></div></video><p>It gives an introduction to Guix from a user's viewpoint, and covers topics such as features for GNU maintainers, programming interfaces, declarative operating system configuration, status of the GNU/Hurd port, and the new Emacs and Web interfaces---with a bunch of demos.</p><p>Do not miss <a href="http://audio-video.gnu.org/video/ghm2014/">other fine talks from the GHM</a>. Many thanks to everyone who took care of the video recordings.</p>https://guix.gnu.org/blog/2013/guix-at-the-european-lisp-symposium/Guix at the European Lisp SymposiumLudovic Courtès2013-05-21T00:00:00+02002013-05-21T00:00:00+0200 A paper presenting the design of Guix's Scheme API and packaging language has been accepted for the 2013 European Lisp Symposium (ELS) . ELS will take place in Madrid on June 3-4. …<div><p>A <a href="http://arxiv.org/abs/1305.4584">paper</a> presenting the design of Guix's Scheme API and packaging language has been accepted for the <a href="http://www-sop.inria.fr/members/Manuel.Serrano/conferences/els13.html">2013 European Lisp Symposium (ELS)</a>. ELS will take place in Madrid on June 3-4.<br /></p></div>