Why not to overload functions in Common Lisp

A short note on the nuances of avoiding overloading functions in Common Lisp

In my first post for 2022, I want to make the case for why we should not overload functions in Common Lisp. The most insightful parts of this post were taken from an old comp.lang.lisp thread, where Pascal Bourguignon’s answers were most insightful and deserve much of the credit for the below.

We define function overloading to be the ability to associate single function names with multiple functions, and directing function calls to relevant functions based on the arguments of the calls. It is not a concept native to Lisp, but rather one borrowed from other languages such as C++.

Function overloading can be implemented in Lisp through two mechanisms:

• Lambda Lists: Lambda lists and the ability to specify &optional and &key parameters (and in particular, suppressing keyword argument checking through &allow-other-keys) can be (ab)used to implement function overloading
• Generic Functions: Generic functions allow us to implement multiple methods for a given function and in a sense overload them as long as the methods share congruent lambda lists

Whilst generic functions can be (ab)used to implement function overloading, there is an importance nuance between the two concepts. Generic functions are most commonly associated with performing largely the same task for a given function name, but being able to accept different types of parameters. On the other hand, function overloading is typically used to associate distinct, unrelated functions or tasks with a particular function name and determining the appropriate task based on the number and type of arguments supplied.

Thus, whilst we can use generic functions to implement a subset of function overloading (namely when lambda lists are congruent), they were not developed with this purpose in mind.

This leads us to the topic of today’s note, on why we should not overload functions in Common Lisp. The reasons to avoid doing so are as follows.

Avoiding Indirection and Fighting Against the Language

More than any other language, Lisp and its dialects are known for being developed from a small set of primitives which have been combined to implement more advanced functions, which are combined to develop more functions and so forth. Typically, one can inspect any Lisp function and directly follow the chain from its source code definition to the very primitives of the languages that implement it. The MACROEXPAND function in Lisp is a visible example of this introspection.

Unfortunately, function overloading introduces indirection to this chain, as one needs to stop and determine which of the overloaded functions need to be applied in a particular context. This reduces the ability to introspect your code and also introduces some complexity in reading it as Lisp progammers will not be used to deciphering overloaded functions in a codebase (as it is not a native concept to Lisp). Moreover, there are no standard tools in the language to help offset this indirection.

The end result is more complex and difficult to read code that goes against the conventions of the language (namely, one that is built up from primitives in a direct chain).

But what about Generic Functions? They also introduce indirection!

It is true that generic functions also cause a layer of indirection in. This was one of the reasons I was avoiding CLOS for some time, and I am sure others may feel the same to some degree.

This is a very important question and there are a few points to be made here.

• First, CLOS and how it is implemented is part of the ANSI Common Lisp Standard and should be well understood by any experienced Lisp programmer. Thus, it will be easier for us to parse the indirection caused by CLOS generic functions as we are used to it.
• In addition, CLOS and MOP provide significant tools to introspect generic functions, again helping overcome the layer of indirection introduced by generic functions. An equivalent set of tools do not exist for custom user-defined function overloads.
• Despite both of these points, it is important to note that generic functions are not meant to be used to overload disparate functions within the same function name. Indeed, the fact that lambda lists must be congruent points to this direction.
• Rather, generic functions and CLOS in general represent an organisational tool to collect associated concepts in objects and generic functions. It is commonly stressed that generic functions should not be used for function overloading, but rather to represent a single process at a higher level of abstraction.
• To this point, Sonya Keene in Object-Oriented Programming in Common Lisp notes that the documentation string of a generic function should cover the overall purpose of the function, what it does and what it returns at a high-level.
• In some ways, this doc string helps reduce some of the indirection caused by generic functions as it summarises the purpose of the function without requiring the reader to determine which method to read to gain this understanding.

Thus a well-organised codebase that makes use of CLOS and generic functions as an organisational tool will not suffer greatly from the indirection introduced by generic functions because they encourage the reader to think about the code at a level above the indirection (namely, not in terms of figuring out which method applies to the problem, but rather the overall process and aim of the generic function).

On the other hand, there is no link between a higher level abstraction and overloaded functions, and the latter only creates confusion as there is no clear indication what an overloaded function does without parsing its arguments.

Conversely, generic functions are meant to be understood without parsing their arguments, and in this lies the key distinction between the two concepts and why it is okay to have a layer of indirection in the latter but not the former.

Function Overloading is Not a Good Idea

The preceding points related to why function overloading is not a good idea in Lisp, but not whether it was good or bad by itself. In this post, we argue that function overloading is a bad idea because it introduces imprecision that requires context to resolve and this imprecision should be separated from the core language.

One of the main arguments for function overloading is that English (and other languages) overload the meaning of certain words and thus overloading functions allow us to keep a closer link to natural language.

An example in the earlier referenced comp.lang.lisp thread is the word “draw”, which can take on different meanings depending on context:

• Draw a shape
• Draw a conclusion
• Draw from a bank account

Given the reuse of words in English to convey different things, one can argue it makes sense to reuse function names for different processes, where the names naturally are appropriate for those different processes.

Whilst this makes sense, why stop at simply function overloading? Why not go down the route of full natural language processing?

As pointed out in the earlier referenced comp.lang.lisp thread, the LOOP macro in many ways is an example of programming constructs that moves in that direction.

Whilst the LOOP macro proves very useful once you learn its unique syntax, it suffers from the same fate as function overloading and generic functions do — the additional layer of syntax processing creates a layer of indirection which makes it difficult to introspect the LOOP macro without spending considerably more time versus more “Lispy” alternatives. Indeed the complexity in understanding the LOOP macro for many may serve as a caution against the use of context-based natural language based terminology (of which functional overloading is a mere subset).

In a perfect world, it would make sense to seperate this layer of context-based syntax processing (language inference) from the underlying language, allowing one to keep the precision of the language and encapsulating it within a clearly defined layer of inference. This would represent a more modular and clean design.

Lisp is well known for the simplicity of its syntax parsing and processing, and we should strive to maintain this simplicity and all the benefits of introspection and modularity it confers by not adding unnecessary language parsing.

Now, there are always trade-offs to be made. The benefits from the LOOP macro and generic functions far outweigh the indirection they introduce and it makes sense for them to be within Lisp. However, given how easy it is to avoid overloading functions, it should be a relatively straightforward conclusion from this to avoid overloading functions and maintain a level of simplicity and elegance in our codebases. Any choice for context-based language inference should be developed in a domain-specific language built upon top of Lisp, and not within Lisp.

Packages Exist to Manage Namespace Conflicts

Perhaps the most convincing reason to avoid function overloading in Common Lisp is because we do not have to. The Common Lisp Package System provides an excellent mechanism to manage namespaces and we can re-use function names, and simply need to intern them in separate packages. By doing so, we also create a clear separation between functions sharing the same name, which is useful for a programmer making use of both. This is much better than hiding this separation behind function overloading. Indeed, function overloading arose in other languages which did not have as strong as a solution to namespace conflicts as Common Lisp, and it is an acceptable and viable approach in those languages. But in Lisp, we don’t need to :-)

Concluding Remarks

In this short post, I hope I convinced you to avoid overloading functions in Common Lisp. Thanks for reading and Happy New Year!