An introduction to Lisp Packages

Combining multiple lisp files together into a cohesive whole

For a more general introduction to Common Lisp (but not packages) check out my 52 page guide [click here].

Writing Lisp code is an enjoyable experience for me, and I slowly improved my ability in this regard. After all, practice is the best teacher. My journey in Lisp land went through the following stages:

Packages are a central mechanism in Common Lisp for managing different sets of names and avoiding name collisons that may occur if multiple files contain variables or functions with the same name. We need to understand the package system to correctly split our code across multiple files and then load them together as one program.

Thus, this short guide will teach you the basics of the package system, and then we conclude with an instructive example of how to organise code across multiple files.

Part 1: Basics of Packages

1. What are Packages?

A package in Common Lisp is a namespace that maps the print names of symbols to the symbols themselves. This is illustrated below:

Variables and functions are named by symbols; Symbols are named by strings (print name); Packages are mapping table of strings to symbols

Print names allow for easy access to the symbol, for example, you can just type a symbol’s name on a keyboard and the Lisp reader will be able to access the symbol. The Lisp reader will lookup each print name it encounters within the current package (to be discussed shortly) to locate & access the relevant symbol named by the string. For example:

Symbols & Packages work together to make our life easier by allowing us to access variables and functions by simply typing the string name associated with them

2. How are Packages Stored?

Package objects are themselves named by strings. Lisp has a single package namespace known as the package registry for mapping package names to package objects. Every time you refer to a package by its name, Lisp will access this registry and obtain the relevant package object.

It is important to note that we need to register a package before we can refer to it by name in our code. As we will see later, this has important implications on the order we load our packages into our program: at any point, we cannot refer to a package that has not yet been registered.

Finally, you can access all the packages currently registered with the function list-all-packages. Lisp allows for multiple packages to co-exist with each other. A symbol name is unique within a package, but the same symbol name can exist within another package without conflict.

3. Revisiting the “Current Package”

Whilst multiple packages can and do exist, at any given time, only one package is active or current in the Lisp system. The current package is, by definition, the one that is the value of the global variable *PACKAGE*.

The current package is used by the Lisp reader to translate strings into symbols. If a symbol string does not exist in the current package, an error will be returned.Whenever we define new symbols, such as via defparameter or defun, they are interned into the current package.

You can see what the current package is by typing *PACKAGE* into your REPL.

4. Working Outside the Current Package

To access symbols in other packages (noting that these packages must be registered first before their use), a package qualifier (in the format of package-name:) is required to preceed the symbol name.

As an example, foo:bar when seen by the Lisp reader, refers to the symbol whose name is bar in the package whose name is foo.

5. Understanding Internal & External Symbols

To add an important point to the above foo:bar example:

• This is true only if bar is an external symbol of foo, that is, a symbol that is supposed to be visible outside of foo
• A reference to an internal symbol requires the intentionally clumsier syntax foo::bar

External symbols are part of the package’s public interface to other packages. External symbols are chosen with care and are advertised to users of the package.

Internal symbols are for internal use only. Most symbols are created as internal symbols; they become external only if they appear explicitly in an export command for the package.

A symbol name within a package can be either external or internal in that package, but not both.

6. Revisiting Package Qualifiers

A package qualifier containing only a single colon must refer to an external symbol — one the package exports for public use.

A double-colon qualifier can refer to any symbol from the named package, although its a good idea to avoid accessing these symbols as are they typically meant for internal use only (and hence not exported).

External packages that have been exported by the package can be accessed with a single colon qualifier:
CL-USER> (my-package:external-function)
Internal packages can be accessed with a double colon qualifier (although this should be avoided). Note that packages do not restrict access to any functions, they are simply a mechanism to manage namespaces:
CL-USER> (my-package::internal-function)

Note that all symbols in the current package can be accessed without a package qualifier; the above discussion is only relevant when we are accessing symbols in packages other than the current package.

Part 2: Taking a Break

We covered a lot of concepts in the above. Let’s take a break for a bit and perhaps you can re-read the above so that you understand the basic principles better. It took me some time to get my head around this topic.

In the next part, we will first introduce three packages that are built into the Common Lisp standard, and we will then learn how to write our own packages and switch between packages.

Part 3: Basics of Packages (contd.)

7. Built-in Packages

There are three packages built into the Common Lisp standard:

• COMMON-LISP: The COMMON-LISP package stores all the functions that are defined in the Common Lisp standard. It is a read-only package (i.e. you cannot add your own symbols to it)
• CL-USER: The CL-USER package is the default current package when you start a new Lisp session. It contains all the symbols in the COMMON-LISP package, together with any user generated symbols. We use CL-USER to add our own symbols to the base set provided by the language standard
• KEYWORD: The KEYWORD package allows us to access keyword symbols without an explicit keyword: package qualification, such as in the below example:

Below two are equivalent:
CL-USER> keyword:a
:A
CL-user> :a
:A

8. Defining Our Own Packages

We can define and register our own packages with the defpackage macro. The basic syntax of this macro is recorded in the below example.

(defpackage :my-package
  (:use :cl :other-package-1 :other-package-2)
  (:export :symbol-1
           :symbol-2
           :symbol-3))

9. Exporting Symbols with :EXPORT

The :export command within defpackage is used to define and export external symbols of our package. In our example, symbol-1 , symbol-2 and symbol-3 are external symbols that have been exported, such that:

• Any package that inherits my-package can access these symbols without a package qualifier
• We can refer to them in any package with a single colon package qualifier

As a reminder, all un-exported, or internal, symbols must be accessed with a double colon package qualifier.

10. Inheriting Packages with :USE

The :use command within defpackage is used to inherit packages. In our example above, we have inherited the COMMON-LISP package :cl and two other packages, :other-package-1 and :other-package-2.

• When we inherit packages, all of the external symbols in the inherited package will be treated as internal symbols in the package we are defining
• For example, all of the external symbols in :cl are now internal symbols in my-package and we can thus access all COMMON-LISP standard functions within this package without a package qualifier (i.e. we can do (+ 1 3) instead of the longer form (cl:+ 1 3) when writing code within my-package
• Thus it is a good idea to always inherit :cl within the packages we define

TheUSE-PACKAGE command is very similar to :use within DEFPACKAGE in that it inherits symbols from a given package. It takes two forms:

;; With only one argument, use-package will add my-package-2 to the use list of the current package:
(use-package my-package-2)
;; With two arguments, use-package will add my-package-2 to the use list of my-package-1:
(use-package my-package-1 my-package-2)

Finally, you can get a package’s use list by calling the function package-use-list.

As some general advice, it is better to avoid inheriting packages outside of :cl and perhaps some of your own standard libraries that you use very frequently (and thus want to avoid having to type the package qualifier each time).

Rather it is better to call symbols from packages explicitly with a package qualifier as:

(1) It is easier for us to understand where a symbol is located (as the package qualifier tells us this)

(2) It avoids name clashes where multiple packages use the same symbols

11. Switching Between Packages

The IN-PACKAGE command is used to switch the current package. As an example, in the below, we switch the current package to my-package.

(in-package my-package)

12. Interning Symbols in a Package

If you recall, whenever we define new symbols, such as via defparameter or defun, they are interned (entered) into the current package. Thus, when we want to intern symbols in a particular package, we need to first switch to that package via an in-package form (see examples below).

Part 4: Example

There are a lot of slightly disjointed concepts in the above. Let’s now bring it all together with an example. This example will also show you how to combine and load multiple lisp files together.

We will work with the following files:

1. Utilities.lisp: A file that contains some of our own user-defined functions
2. Quant.lisp: A file that does some calculations and makes use of utilities.lisp
3. Main.lisp: A file that references the two above
4. Packages.lisp: A file that combines the above and brings everything together into a cohesive whole so that it can be run as one program

The source code for the files, with annotations, are as follows.

1. Utilities.lisp

Below is the source code for our basic utilities.lisp. It contains one function that allows us to sum all the elements of a list.

(in-package :utilities)
(defun list-sum (my-list)
  (reduce #'+ my-list))

At the start of each of our lisp files, we should put one in-package form to switch to the relevant package (in this case, utilities). Thus, all symbols defined subsequent to this form will be interned in the utilities package.

Each file should contain exactly one IN-PACKAGE form, and it should be the first form in the file other than comments. If you violate this rule and switch packages in the middle of a file, you can confuse others who don’t notice the second IN-PACKAGE.

It is fine to have multiple files switch to the same package, each with an identical IN-PACKAGE form at the start of their code. This just means all the symbols defined in these files will be interned in the package that has been switched to.

2. Quant.lisp

Below is the source code for our second file, quant.lisp. It loads the above utilities.lisp file and also loads the cl-csv package from Quicklisp (the de facto library manager for Common Lisp).

This file then loads a csv file and calculates the sum of a list.

(in-package :quant)
(ql:quickload 'cl-csv)
;; Change the below path to where you sae your utilities file:
(load "/Users/ashokkhanna/utilities.lisp")
;; Change the below path to where you save your csv file:
(defparameter my-filename "/Users/ashokkhanna/my-csv.csv")
(defparameter csv-file (cl-csv:read-csv (pathname my-filename)))
(defparameter my-sum (list-sum (list 1 2 3 4 5)))

Note the following:

• The first form in-package is used to switch the current package to quant. As a reminder, all of our files should have an in-package form at the top
• We reference the list-sum function in the utilities package without a qualifier → We will see later that we make the quant package inherit the utilities package, allowing us to refer to the latter’s symbols without a package qualifier
• In contrast, we reference the read-csv function of cl-csv with a package qualifier → We will see later that we did not inherit this package and thus are required to reference its functions with a package qualifier

3. Main.lisp

Below is the source code for our third file, main.lisp. It loads the above quant.lisp file, which means it also indirectly loads utilities.lisp.

(in-package :main)
;; Change the below path to where you save your quant file:
(load "/Users/ashokkhanna/quant.lisp")
(print (quant::list-sum (list 1 2 3 4 5 6 7 8 9 10)))
(print csv-file)
(print quant::my-sum)

Note the following:

• We will see shortly that we did not export list-sum or my-sum from quant.lisp and therefore need to use a package qualifier to reference both, specifically with a double colon since they are internal symbols of quant.lisp
• We did however export csv-file from quant.lisp and can reference it without a package qualifier

4. Packages.lisp

Below is the source code for the final and most important file for today, packages.lisp.

(in-package :cl)
(defpackage :utilities
  (:use :cl)
  (:export :list-sum))
(defpackage :quant
  (:use :cl :utilities)
  (:export :csv-file))
(defpackage :main
  (:use :cl :quant))

;; Change filepath to where you save your main.lisp:
(load "/Users/ashokkhanna/main.lisp")

Note the following:

1. We define our three packages utilities, quant and main here, and not in their own files
2. Note how we define the packages in order: we need to define utilities first as it is referenced in the quant package declaration; similarly, quant must be defined before we define main
3. Each package uses (inherits) the standard CL library via :use :cl
4. In addition, quant inherits from utilities. Hence, as we noted above, we can reference the exported or external symbols of utilities within the quant package without a package qualifier
5. The utilities package exports the symbol list-sum. The quant package exports the csv-file package but not list-sum. Hence main.lisp can directly access csv-file but requires a double colon package qualifier to access list-sum (more technically, a double colon package qualifier is required if we want to reference list-sum from the quant package, i.e. quant::list-sum. However, we could also reference it directly from the utilities package, in which case we only need a single colon package qualifier as it is an exported symbol of this package, i.e. utilities:list-sum also will work).

Finally note that our packages.lisp file starts with (in-package :cl). This is because defpackage is defined in the CL package, so to make sure we can access it, we switch the current package to :cl.

5. Running as One Program

With all of our files written and referenced correctly, we can run our program by simply loading and running packages.lisp. This file will add all the package definitions to the package registry and then load main.lisp, which in turn loads quant.lisp, which in turn loads utilities.lisp.

Note how we segregated package definitions from the actual code we write. This is because we need to define our packages in order, so it is easier to do this together in one place.

Conclusion

This concludes our brief guide on packages and loading multiple files together. There are a lot more nuances to this subject, which you can read online in various blogs, tutorials and stack overflow posts. But at least for today we have been able to achieve the basic task of compiling and loading multiple files together. You may need to read the above a few times to fully grasp the concepts, unfortunately I find the Lisp package system a bit complex and hard to follow. I hope this guide helped in this regard.

Source code for the above example can be found here. Please star the repo if you like this guide :) I suggest experimenting with various options, try and deliberately break the code to see how things work.

If you like this post, please press the “clap” button :) You can also drop me an e-mail at ashok.khanna@hotmail.com for any comments or feedback.

Further Reading

There is a lot more to cover in this topic, but the good news is you are well on your way to becoming a super professional programmer who can create large files that work with each other. Some recommended next readings:

• Using package-local nickname to be able to write your own shorthand names for some package qualifiers → A great resource is this: https://gist.github.com/phoe/2b63f33a2a4727a437403eceb7a6b4a3
• Understanding the difference between difference between a system (as in asdf) and a package (what we discussed above) and also the finer points of QuickLisp. This post was quite helpful and you should also google some of these concepts on your own :): https://www.reddit.com/r/lisp/comments/kt0sxn/basics_of_lisp_packages_ie_working_with_multiple/gikbd8v?utm_source=share&utm_medium=web2x&context=3

I’ll update this section as I get more useful readings to add. Thanks!