An introduction to Lisp Packages
Combining multiple lisp files together into a cohesive whole
For a more general introduction to Common Lisp (but not packages) check out my 52 page guide [click here].
Writing Lisp code is an enjoyable experience for me, and I slowly improved my ability in this regard. After all, practice is the best teacher. My journey in Lisp land went through the following stages:
Packages are a central mechanism in Common Lisp for managing different sets of names and avoiding name collisons that may occur if multiple files contain variables or functions with the same name. We need to understand the package system to correctly split our code across multiple files and then load them together as one program.
Thus, this short guide will teach you the basics of the package system, and then we conclude with an instructive example of how to organise code across multiple files.
Part 1: Basics of Packages
1. What are Packages?
A package in Common Lisp is a namespace that maps the print names of symbols to the symbols themselves. This is illustrated below:
Print names allow for easy access to the symbol, for example, you can just type a symbol’s name on a keyboard and the Lisp reader will be able to access the symbol. The Lisp reader will lookup each print name it encounters within the current package (to be discussed shortly) to locate & access the relevant symbol named by the string. For example:
2. How are Packages Stored?
Package objects are themselves named by strings. Lisp has a single package namespace known as the package registry for mapping package names to package objects. Every time you refer to a package by its name, Lisp will access this registry and obtain the relevant package object.
It is important to note that we need to register a package before we can refer to it by name in our code. As we will see later, this has important implications on the order we load our packages into our program: at any point, we cannot refer to a package that has not yet been registered.
Finally, you can access all the packages currently registered with the function
list-all-packages. Lisp allows for multiple packages to co-exist with each other. A symbol name is unique within a package, but the same symbol name can exist within another package without conflict.
3. Revisiting the “Current Package”
Whilst multiple packages can and do exist, at any given time, only one package is active or current in the Lisp system. The current package is, by definition, the one that is the value of the global variable
The current package is used by the Lisp reader to translate strings into symbols. If a symbol string does not exist in the current package, an error will be returned.Whenever we define new symbols, such as via
defun, they are interned into the current package.
You can see what the current package is by typing
*PACKAGE* into your REPL.
4. Working Outside the Current Package
To access symbols in other packages (noting that these packages must be registered first before their use), a package qualifier (in the format of
package-name:) is required to preceed the symbol name.
As an example,
foo:bar when seen by the Lisp reader, refers to the symbol whose name is
bar in the package whose name is
5. Understanding Internal & External Symbols
To add an important point to the above
- This is true only if
baris an external symbol of
foo, that is, a symbol that is supposed to be visible outside of
- A reference to an internal symbol requires the intentionally clumsier syntax
External symbols are part of the package’s public interface to other packages. External symbols are chosen with care and are advertised to users of the package.
Internal symbols are for internal use only. Most symbols are created as internal symbols; they become external only if they appear explicitly in an export command for the package.
A symbol name within a package can be either external or internal in that package, but not both.
6. Revisiting Package Qualifiers
A package qualifier containing only a single colon must refer to an external symbol — one the package exports for public use.
A double-colon qualifier can refer to any symbol from the named package, although its a good idea to avoid accessing these symbols as are they typically meant for internal use only (and hence not exported).
External packages that have been exported by the package can be accessed with a single colon qualifier:CL-USER> (my-package:external-function)Internal packages can be accessed with a double colon qualifier (although this should be avoided). Note that packages do not restrict access to any functions, they are simply a mechanism to manage namespaces:CL-USER> (my-package::internal-function)
Note that all symbols in the current package can be accessed without a package qualifier; the above discussion is only relevant when we are accessing symbols in packages other than the current package.
Part 2: Taking a Break
We covered a lot of concepts in the above. Let’s take a break for a bit and perhaps you can re-read the above so that you understand the basic principles better. It took me some time to get my head around this topic.
In the next part, we will first introduce three packages that are built into the Common Lisp standard, and we will then learn how to write our own packages and switch between packages.
Part 3: Basics of Packages (contd.)
7. Built-in Packages
There are three packages built into the Common Lisp standard:
- COMMON-LISP: The COMMON-LISP package stores all the functions that are defined in the Common Lisp standard. It is a read-only package (i.e. you cannot add your own symbols to it)
- CL-USER: The CL-USER package is the default current package when you start a new Lisp session. It contains all the symbols in the COMMON-LISP package, together with any user generated symbols. We use CL-USER to add our own symbols to the base set provided by the language standard
- KEYWORD: The KEYWORD package allows us to access keyword symbols without an explicit
keyword:package qualification, such as in the below example:
Below two are equivalent:CL-USER> keyword:a
8. Defining Our Own Packages
We can define and register our own packages with the
defpackage macro. The basic syntax of this macro is recorded in the below example.
(:use :cl :other-package-1 :other-package-2)
9. Exporting Symbols with :EXPORT
:export command within
defpackage is used to define and export external symbols of our package. In our example,
symbol-3 are external symbols that have been exported, such that:
- Any package that inherits
my-packagecan access these symbols without a package qualifier
- We can refer to them in any package with a single colon package qualifier
As a reminder, all un-exported, or internal, symbols must be accessed with a double colon package qualifier.
10. Inheriting Packages with :USE
:use command within
defpackage is used to inherit packages. In our example above, we have inherited the COMMON-LISP package
:cl and two other packages,
- When we inherit packages, all of the external symbols in the inherited package will be treated as internal symbols in the package we are defining
- For example, all of the external symbols in
:clare now internal symbols in
my-packageand we can thus access all COMMON-LISP standard functions within this package without a package qualifier (i.e. we can do
(+ 1 3)instead of the longer form
(cl:+ 1 3)when writing code within
- Thus it is a good idea to always inherit
:clwithin the packages we define
USE-PACKAGE command is very similar to
DEFPACKAGE in that it inherits symbols from a given package. It takes two forms:
;; With only one argument, use-package will add my-package-2 to the use list of the current package:(use-package my-package-2);; With two arguments, use-package will add my-package-2 to the use list of my-package-1:(use-package my-package-1 my-package-2)
Finally, you can get a package’s use list by calling the function
As some general advice, it is better to avoid inheriting packages outside of
:cland perhaps some of your own standard libraries that you use very frequently (and thus want to avoid having to type the package qualifier each time).
Rather it is better to call symbols from packages explicitly with a package qualifier as:
(1) It is easier for us to understand where a symbol is located (as the package qualifier tells us this)
(2) It avoids name clashes where multiple packages use the same symbols
11. Switching Between Packages
IN-PACKAGE command is used to switch the current package. As an example, in the below, we switch the current package to
12. Interning Symbols in a Package
If you recall, whenever we define new symbols, such as via
defun, they are interned (entered) into the current package. Thus, when we want to intern symbols in a particular package, we need to first switch to that package via an
in-package form (see examples below).
Part 4: Example
There are a lot of slightly disjointed concepts in the above. Let’s now bring it all together with an example. This example will also show you how to combine and load multiple lisp files together.
We will work with the following files:
- Utilities.lisp: A file that contains some of our own user-defined functions
- Quant.lisp: A file that does some calculations and makes use of utilities.lisp
- Main.lisp: A file that references the two above
- Packages.lisp: A file that combines the above and brings everything together into a cohesive whole so that it can be run as one program
The source code for the files, with annotations, are as follows.
Below is the source code for our basic utilities.lisp. It contains one function that allows us to sum all the elements of a list.
(in-package :utilities)(defun list-sum (my-list)
(reduce #'+ my-list))
At the start of each of our lisp files, we should put one
in-package form to switch to the relevant package (in this case,
utilities). Thus, all symbols defined subsequent to this form will be interned in the
Each file should contain exactly one
IN-PACKAGE form, and it should be the first form in the file other than comments. If you violate this rule and switch packages in the middle of a file, you can confuse others who don’t notice the second
It is fine to have multiple files switch to the same package, each with an identical
IN-PACKAGE form at the start of their code. This just means all the symbols defined in these files will be interned in the package that has been switched to.
Below is the source code for our second file, quant.lisp. It loads the above utilities.lisp file and also loads the
cl-csv package from Quicklisp (the de facto library manager for Common Lisp).
This file then loads a csv file and calculates the sum of a list.
(in-package :quant)(ql:quickload 'cl-csv);; Change the below path to where you sae your utilities file:
(load "/Users/ashokkhanna/utilities.lisp");; Change the below path to where you save your csv file:
(defparameter my-filename "/Users/ashokkhanna/my-csv.csv")(defparameter csv-file (cl-csv:read-csv (pathname my-filename)))(defparameter my-sum (list-sum (list 1 2 3 4 5)))
Note the following:
- The first form
in-packageis used to switch the current package to
quant. As a reminder, all of our files should have an
in-packageform at the top
- We reference the
list-sumfunction in the
utilitiespackage without a qualifier → We will see later that we make the
quantpackage inherit the
utilitiespackage, allowing us to refer to the latter’s symbols without a package qualifier
- In contrast, we reference the
cl-csvwith a package qualifier → We will see later that we did not inherit this package and thus are required to reference its functions with a package qualifier
Below is the source code for our third file, main.lisp. It loads the above quant.lisp file, which means it also indirectly loads utilities.lisp.
(in-package :main);; Change the below path to where you save your quant file:
(load "/Users/ashokkhanna/quant.lisp")(print (quant::list-sum (list 1 2 3 4 5 6 7 8 9 10)))
Note the following:
- We will see shortly that we did not export
my-sumfrom quant.lisp and therefore need to use a package qualifier to reference both, specifically with a double colon since they are internal symbols of quant.lisp
- We did however export
csv-filefrom quant.lisp and can reference it without a package qualifier
Below is the source code for the final and most important file for today, packages.lisp.
(in-package :cl)(defpackage :utilities
(:export :list-sum))(defpackage :quant
(:use :cl :utilities)
(:export :csv-file))(defpackage :main
(:use :cl :quant))
;; Change filepath to where you save your main.lisp:
Note the following:
- We define our three packages
mainhere, and not in their own files
- Note how we define the packages in order: we need to define
utilitiesfirst as it is referenced in the
quantpackage declaration; similarly,
quantmust be defined before we define
- Each package uses (inherits) the standard CL library via
- In addition,
utilities. Hence, as we noted above, we can reference the exported or external symbols of utilities within the quant package without a package qualifier
utilitiespackage exports the symbol
quantpackage exports the
csv-filepackage but not
list-sum. Hence main.lisp can directly access
csv-filebut requires a double colon package qualifier to access
list-sum(more technically, a double colon package qualifier is required if we want to reference
quant::list-sum. However, we could also reference it directly from the
utilitiespackage, in which case we only need a single colon package qualifier as it is an exported symbol of this package, i.e.
utilities:list-sumalso will work).
Finally note that our packages.lisp file starts with
(in-package :cl). This is because
defpackage is defined in the CL package, so to make sure we can access it, we switch the current package to
5. Running as One Program
With all of our files written and referenced correctly, we can run our program by simply loading and running packages.lisp. This file will add all the package definitions to the package registry and then load main.lisp, which in turn loads quant.lisp, which in turn loads utilities.lisp.
Note how we segregated package definitions from the actual code we write. This is because we need to define our packages in order, so it is easier to do this together in one place.
This concludes our brief guide on packages and loading multiple files together. There are a lot more nuances to this subject, which you can read online in various blogs, tutorials and stack overflow posts. But at least for today we have been able to achieve the basic task of compiling and loading multiple files together. You may need to read the above a few times to fully grasp the concepts, unfortunately I find the Lisp package system a bit complex and hard to follow. I hope this guide helped in this regard.
Source code for the above example can be found here. Please star the repo if you like this guide :) I suggest experimenting with various options, try and deliberately break the code to see how things work.
If you like this post, please press the “clap” button :) You can also drop me an e-mail at email@example.com for any comments or feedback.
There is a lot more to cover in this topic, but the good news is you are well on your way to becoming a super professional programmer who can create large files that work with each other. Some recommended next readings:
- Using package-local nickname to be able to write your own shorthand names for some package qualifiers → A great resource is this: https://gist.github.com/phoe/2b63f33a2a4727a437403eceb7a6b4a3
- Understanding the difference between difference between a system (as in asdf) and a package (what we discussed above) and also the finer points of QuickLisp. This post was quite helpful and you should also google some of these concepts on your own :): https://www.reddit.com/r/lisp/comments/kt0sxn/basics_of_lisp_packages_ie_working_with_multiple/gikbd8v?utm_source=share&utm_medium=web2x&context=3
I’ll update this section as I get more useful readings to add. Thanks!