Lisp & the Web

Introductory Reference Guide to Creating Web Applications with Common Lisp & Google Compute Engine

18 min readApr 15, 2021

This is a very quick (which given the word length sounds like an understatement; but, given the complexity of the topic, is true) guide to ceating web applications via a Lisp Web Server on Google Compute Engine. Hunchentoot will be our server of choice.

Our server will be connected to a PostgreSQL database to perform Create, Read, Update and Delete operations and I will give a sneak peak into storing and authenticating users.

I may write a more detailed guide in the future, but for now I am just documenting my setup notes. A familiarity with Common Lisp and Emacs is assumed. Apologies if the below is lacking explanation, its very much a step by step reference (mostly for me!).

If you like the guide, please star the repo and also leave a comment here letting me know if it worked smoothly for you. Hopefully it did.

Step 1: Creating a GCE VM Instance

First, log into GCE Console (or create an account if you do not have one). Select Create Instance as follows:

Click the “Create Instance” Button at the top. Sometimes GCE Console is confusing to navigate, but hopefully you find this page relatively quickly

You then need to enter the name of the instance and other settings. For now, enter the name, change the boot disk to Ubuntu 20.04 LTS and select Allow HTTP traffic and Allow HTTPS traffic:

Settings used for the VM instance in this tutorial

Note in the top right there is an estimate of the monthly charge incurred. I believe new joiners get $300 of GCE Credits, but yes, using GCE is not free or cheap, and should only be used when you have an actual project in mind. Better to do most of your prototyping and development on your local machine for free and then migrate to GCE when you are ready to deploy.

Click on the Management, security, disks, networking, sole tenancy link towards the bottom of the page and navigate to the Networking tab. Click on the drop-down under Network interfaces:

Navigate to Network Interfaces under “Management, Security, Disks, Networking, Sole Tenancy”

Now we will switch to a static IP by selecting Create IP address under External IP:

This brings up a menu where you type in a name for your static IP (this is not your IP address, just an identifier).

After you click on RESERVE, a static IP will be allocated to you, which can be seen thereafter:

You can then press the Create button to create your VM instance:

Step 2: Installing Lisp & Emacs

2.1 Remote into your Machine

GCE will take a minute or so to initialise your instance. Once this is done, SSH into it as follows.

By the way, I like GCE over AWS because their remote desktop experience feels much faster

2.2 Install Emacs & SBCL

Run the following commands from the console, selecting yes wherever prompted.

$ sudo apt-get update$ sudo apt-get install emacs$ sudo apt-get install sbcl

2.3 Install QuickLisp

Then we have to install QuickLisp, the legendary package manager for Common Lisp (note that in the curl commands below, it is a capital O and not a 0):

$ curl -O https://beta.quicklisp.org/quicklisp.lisp$ curl -O https://beta.quicklisp.org/quicklisp.lisp.asc$ sbcl --load quicklisp.lisp
Then when prompted (at the "*" prompts, i.e. whenever there is a pause in the installation):* (quicklisp-quickstart:install)* (ql:system-apropos "vecto")* (ql:quickload "vecto")* (ql:add-to-init-file)[Press Enter]* (quit)

2.4 Install SLIME

Now, we need to install SLIME, the amazing Common Lisp IDE built on Emacs. First log into Emacs:

$ emacs

Then type in the following to bring up your .emacs file:

C-x C-F~/.emacs (usually the ~/ is already prefilled for you)

Now install the MELPA repository by adding the following into your .emacs file and then saving via C-x C-s:

(require 'package)(add-to-list 'package-archives '("melpa" . "https://melpa.org/packages/") t)
(package-initialize)

Then refresh the package list with either M-x package-refresh-contents or M-x package-list-packages (the latter worked for me) (note that you may need to use ESC as the key for Meta — I needed to on a Mac). I also had to exit my Emacs with C-x C-c and then run it again (i.e. restarting it to allow the above to take effect), but perhaps you don’t need to.

Then go to the package list via M-x package-list and scroll down to slime (since you are in a terminal, the mouse probably won’t work. I used page down to quickly scroll). Press enter on slime to move to the installation page, from where you can press enter on [Install] to install it. By press enter on, I mean you need to move the cursor to the word and then press enter.

If you don’t see SLIME here, either you are unable to connect to Melpa or the package list hasn’t refreshed. Google is your friend

You may need to navigate to the window which has the [Install] text. To do so, type C-x o (this alternates between available windows). This all may be a bit jarring since you are not in a GUI, but in the long run it will make you better with Emacs shortcuts :)

After SLIME is installed, add the following to the bottom of your .emacs file:

(setq inferior-lisp-program "sbcl")

At this point, your .emacs file should look something like this (I removed the comment lines that will likely appear in your version):

(require 'package)(add-to-list 'package-archives '("melpa" . "https://melpa.org/packages/") t)(package-initialize)(custom-set-variables                                                               '(package-selected-packages (quote (slime))))(custom-set-faces
)(setq inferior-lisp-program "sbcl")

Step 3: Testing it all Works

Restart Emacs. Now try M-x slime and see if it brings up a Lisp REPL. Test it out with the following:

(print "Hello, World!")

Now let’s test out whether QuickLisp works correctly. Try the following:

(ql:quickload :hunchentoot)

It should download and load the Hunchentoot package. If either of the these two items do not work or you get errors, you will need to google and fix it before continuing.

Step 4: Installing PostgreSQL

Returning to our terminal, let us install PostgreSQL via the following:

$ sudo apt-get install postgresql

Now lets quickly go through some PostgreSQL basics, and setup a database which we will use later in our webserver.

4.1 Superuser Access & Creating Users

You can log into PostgreSQL as a superuser with the following command.

$ sudo -i -u postgres

This will change the terminal prompt to something like:

postgres@medium-test-server:~$

where you can enter PostgreSQL commands. You can quit PostgreSQL by typing exit at the prompt:

postgres@medium-test-server:~$ exit

Now we can create user, e.g. testuser, from Ubuntu’s terminal with the following:

$ sudo -u postgres createuser --login --pwprompt testuser

It will bring up a prompt for password, for now please use mypassword as our example Lisp server will access our database with this password.

4.2 Logging into PSQL

psql is a terminal-based front-end to PostgreSQL. We can log into it by entering in psql at the postgres prompt:

postgres@medium-test-server:~$ psql

This will change the terminal prompt to:

postgres=#

which you can quit by typing quit:

postgres=# quit

4.3 Creating Databases

We can create a database with the following command from Ubuntu’s terminal:

sudo -u postgres createdb --owner=testuser testdb

4.3 Logging in as a ‘Normal’ User to a Database

PostgreSQL permissions and accesss settings can get complex at times. The below should work however to log into PostgreSQL as a normal user (e.g. testuser that we create above):

$ psql -U testuser -h 127.0.0.1 testdb

4.4 Starting / Stopping Restarting

The following commands are used to start, stop and restart a PostgreSQL service. Note that reload is used to reload the configuration without stopping the service.

$ sudo service postgresql start
$ sudo service postgresql stop
$ sudo service postgresql restart
$ sudo service postgresql reload

4.5 Viewing Background Processes

It’s very useful to see what background processes are running on our VM instance. We can do this with the following, which can be exited with F10.

$ htop

HTOP is like Windows Task Manager. In the above, you can see that PostgreSQL is running nicely in the background

Step 5: Installing our Lisp Server

It is time now to install our lisp server. The code we will use is an adaption of Vetle Roeim’s excellent guide on implementing a blog in Common Lisp, which was the guide that introduced me to Common Lisp. His guide is much more detailed and a worthwhile read in its own right. However I have adapted it to PostgreSQL, so I suggest you read his guide, and use my code.

Note that the purpose of my guide is not teach or walk through Common Lisp. That said, if you carefully read the code we will use, you will learn a fair bit of how it works. It is actually pretty straightforward (once you get the hang of things).

If you are new to the language, you can read my introductory guide or one of the many free materials available on the internet.

5.1 Downloading the Repo

As a first step, download our example web server code with the following:

$ git clone https://github.com/ashok-khanna/lisp-notes

Let’s now copy the files into our home directory, e.g. by something like following (entered when within our home directory folder)

$ cp -R common-lisp-by-example/Tutorials/example-server/. .

5.2 Loading & Running the Repo

Open Emacs, and the start a new SLIME session with M-x slime. You can then load our web server file directly from the REPL via:

(load "blog-server.lisp")

The first time you do this will take a bit of time as QuickLisp will download all the necessary dependencies. If everything is successful, you should not get any error messages. Our server is now live on port 4242 (we specify the port in our code).

5.3 Reloading the Server & Associated Errors

Our example repo is very basic code without error checking & management. It creates a table myBlogTable within the testdb we created earlier, and then inserts some records. Thus, every time we reload the server we will get errors stating that the table and these records already exist. Simply read the error prompts and select to ignore the errors, such as selecting 1 in the below to continue.

This is not pretty, but I didn’t want to add too much complexity to the files at this stage, and you also now have experience of live debugging in Lisp.

Step 6: Installing NGINX and Accessing our Server

Assuming no errors in the above, our web server is up and running, but we cannot view it. This will be difficult to do as we are currently only in terminal mode in our remote session into our GCE instance. We will now open up our server to the world and access it from the public internet.

6.1 Installing NGINX

Currently, our lisp server is accessible on localhost:4242. We will use NGINX as another server to connect our lisp web server to the internet. This process is called reverse proxying. We use NGINX because it will make our life easier later on when we add SSL certificates.

First, without closing our existing remote session, start a new SSH session into our VM machine. Whilst we are at it, note our VM machine’s external IP address from this page (e.g. for me it is 35.224.222.114. You will have a different IP, as allocated to you by GCE).

Enter the following at the terminal prompt to install nginx:

$ sudo apt-get install nginx

6.2 Setting up NGINX

Nginx stores its configuration file in /etc/nginx/sites-enabled/default. Let’s delete this file with the following:

$ sudo rm /etc/nginx/sites-enabled/default

We will now create our own configuration file, but in a different folder (sites-available):

$ sudo emacs /etc/nginx/sites-available/node

Paste the following code into this file. We can ignore the “example.com” bit for now, we will change that later. Note that port 80 is used for HTTP access, whilst port 443 is used for HTTPS access. We will get to HTTPS in the next section.

server {
   listen 80;
   server_name example.com;
   location / {
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header Host $http_host;
      proxy_pass http://127.0.0.1:4242;
   }
}

Now, we want to link the configuration file in sites-enabled (which is where nginx looks) to the above file in sites-available, done via the following symlink:

$ sudo ln -s /etc/nginx/sites-available/node /etc/nginx/sites-enabled/node

As a final step, restart the nginx server for the configuration file to take effect:

$ sudo service nginx restart

6.3 Accessing our Web Server

Let’s now test everything out! Recalling your VM instance’s IP address from earlier (or check your GCE console), open up a browser and navigate to http://external-ip-address/. E.g. for me it was:

http://35.224.222.114/

If everything works to plan, you should get the following webpage:

6.4 Creating Entries

Our web server is very barebone, but functional. We will now show how to Create, View and Edit database entries, using blog posts as an example.

If you navigate to http://external-ip-address/create/, you can add entries to the database. For me this was: http://35.224.222.114/create/.

Try it! You should get something like the following:

6.5 Viewing Entries

You should now get an error such as the following(in this case I created an entry with title “test”), as our web server is redirecting you to localhost and not the actual external IP address:

For now, we can manually correct this and view the entry by typing in http://external-ip-address/view/?[title-of-post] (for me in this case: http://35.224.222.114/view/?test). You can also view the posts from the home page of our server.

6.6 Editing Entries

Editing entries is very similar to creating them, except this time you instead navigate to http://external-ip-address/edit/. Also note that editing will only work on entries that exist, and have no effect on entries that do not exist.

Below is an example screenshot.

6.7 Deleting Entries

I will leave deleting entries to you as an exercise :)

Step 7: Correcting the LocalHost Issue

The above issue with our lisp server redirecting to localhost, which is not accessible publicy is annoying and a deal-breaker. There are two ways around this.

Option 1

Update the code in blog-server.lisp to reference our external IP address and not localhost. In my example, I would update the code snippets to the following, such as in the following (note that this is not the only place where this change needs to be made, you will be able to figure it out when you read the code):

(defun save-new-blog-post ()
  "Read POST data and modify blog post."
  (insert-record (hunchentoot:post-parameter "title
    (hunchentoot:post-parameter "body"))
  (hunchentoot:redirect
    (format nil "http://35.224.222.14/view/?~A"
      (make-url-part (hunchentoot:post-parameter "title")))))

Now let me show you one of the beauties of Lisp. After making the above change, you can simply re-evaluate the above function, without restarting your lisp server, by navigating to its end and entering C-x C-e. If you now try to create a new blog entry, the server will automatically point to the external IP address and not localhost. No need to recompile or stop the server!

Option 2

Option 2 is doing the same as Option 1, but using an actual domain as in the below. Here I will use one of my own domains, you will need to replace this with a domain you own.

(defun save-new-blog-post ()
  "Read POST data and modify blog post."
  (insert-record (hunchentoot:post-parameter "title
    (hunchentoot:post-parameter "body"))
  (hunchentoot:redirect
    (format nil "https://ashok-khanna.com/?~A"
      (make-url-part (hunchentoot:post-parameter "title")))))

As part of Option 2, we need to link our domain to our external IP address, which is the topic of the next section. Note how I edited the above to https, we will also add SSL security in the next section.

Step 8: Adding a Domain and Adding SSL

We will now attach a domain to our IP address. To do this section, you need to own a domain name. I will use my domain ashok-khanna.com for this guide, whenever you see this, replace it with your own domain. Note also that I use Google Domains, which I highly recommend, however you should be able easily to adapt the steps for your registar if it is different.

8.1 Creating a Cloudflare Site

We will also use Cloudflare for our server. Cloudflare sits between website visitors and our web server, providing invaluable caching and speed improvements, reducing the overall load on our servers (and hence their cost!). Cloudflare also provides us with some basic server-side analytics and overall quality of life improvements.

First, log into Cloudflare and then select Add a Site :

Enter your domain name in the prompt:

Select the free plan for now and press continue:

Press continue on the next screen:

You will then be prompted to change name servers:

Now, go to your registrar (Google Domains for me), and make these name server changes. You may be prompted to turn off DNSSEC, just accept all prompts.

Once this is done, return to Cloudflare and select “Done, check nameservers”. Note that name server changes can take some time, so just be patient.

Cloudflare will now prompt you for some settings, make sure to select the following:

Automatic HTTPS Rewrites — We want this

Always Use HTTPS — We want this

The remaining settings shouldn’t matter, just select the defaults. Now simply wait a bit until the name server changes take effect and Cloudflare confirms our site is active on their platform. Once active, it should look like this:

8.2 Setting up an SSL Certificate on our Machine

We will now set up an SSL certificate on our VM instance using CertBot. This part is a bit finicky and thus we use NGINX as our reverse proxy server to streamline it as much as possible.

First, let us edit our NGINX settings by entering the following in our terminal:

$ sudo emacs /etc/nginx/sites-available/node

Update the example.com to our domain name, both in www and non-www forms:

server {
   listen 80;
   server_name www.ashok-khanna.com ashok-khanna.com;
   location / {
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header Host $http_host;
      proxy_pass http://127.0.0.1:4242;
   }
}

Note that while the above only listens on port 80 (HTTP), the Certbot utility that we use shortly will update the above code to correctly account for port 443 (HTTPS).

Restart our NGINX server to effect the changes:

$ sudo service nginx restart

8.3 Resetting NGINX

I find this part of the guide a bit finicky and had some errors when I deviated from the path we discuss here. Whilst Google is your best friend, I found it very helpful to uninstall, purge and reinstall NGINX when it had errors. This can be achieved by entering the following command and then repeating the steps of 6.2 and 8.2 above.

$ sudo apt-get purge nginx nginx-common

Don’t this now, but if you have issues with NGINX, definitely try this out.

8.4 Pointing our Domain to our IP

Let’s go back to Cloudflare now. We will now add DNS records to point our domain to our domain name. There are two main types of records:

A records map a name to one or more IP addresses when the IP are known and stable
CNAME records map a name to another name. It should only be used when there are no other records on that name

In this case, we will add A records. Navigate to “DNS” in Cloudflare and select “Add record”:

Add the following two A records and click save. The IP address you enter in these records is the IP address of your VM instance (below is an example for my instance). Note that for www.ashok-khanna.com, we will enter www as the name, while for ashok-khanna.com (i.e. the root domain), we will enter @ as the name.

Once done, your DNS should look like the following:

8.5 Setting up SSL

We will now set up our SSL certificate through CertBot. First, navigate to the SSL/TLS page in Cloudflare and ensure the encryption mode is set to “Flexible”:

Now, go back to our linux terminal and enter the following to install certbot:

$ sudo apt-get install certbot python3-certbot-nginx

Then run Certbot with the following. You will be prompted for your e-mail (enter it), to agree to the terms of service (agree to it), whether you want your e-mail address shared with the Electronic Frontier Foundation (EFF) (your choice).

$ sudo certbot --nginx

You will then be prompted to select which names you wish to activate HTTPS for. Select both by pressing enter:

Assuming you set up your A records correctly and configured Cloudflare correctly (read the above steps very carefully), Certbot will successfully generate an SSL certificate and prompt you whether you want HTTP to HTTPS redirects — Select Yes by entering 2 and then Enter:

Congratulations! Your SSL certificate is live. Note that the SSL certificate will expire, so you may need to generate it again, refer below notes. However I believe it will auto renew in our setup.

8.6 Updating Cloudflare

Once we have setup our SSL certificate, navigate back to Cloudflare and change the SSL/TLS certification to “Full”:

8.7 Testing it out!

The moment of truth is now — navigate to your domain and see if everything works correctly. If not, read through the above and debug. Test out both the www and non-www versions, and note also that all requests are now secured through HTTPS.

Step 9: Adding User Authentication

As penultimate item, let us look at some sample code to add users and authenticate them to our web app. The below is courtesy of linnilbobo on Reddit. I changed the port from 8000 to 4242 in the version in our example repo.

Exit SLIME and restart Emacs. Restart SLIME, but this time run:

(load "auth-server.lisp")

Revisit your custom domain and play around with our new server. Again, it is very barebone, but if you study the code, you will be able to figure out many things and how to extend it something useful. There is a lot of useful code here, including hashing passwords and easy handlers for hunchentoot.

Currently any users added will only exist as long as the server is live. One suggestion is to store the users in a database table, thus allowing you to have persistent storage of users and their passwords.

Step 10: Serving Static Websites

As a final item, lets add the ability to serve static websites to our server. The example code in the repo is already updated, but basically we need to set the following to the location of our static files (in our case, we store the static files within the folder www_ within our home directory).

;;main------
(setf web-server 
   (hunchentoot:start 
      (make-instance 'hunchentoot:easy-acceptor 
:address "127.0.0.1" 
:port 4242 
:document-root #p"www_/"
:persistent-connections-p t
:read-timeout 3.0 
:write-timeout 3.0
:access-log-destination nil
:message-log-destination nil)))

Note that in our example repo, we have two files, test.html and index.html, within the www_ folder. Usually an index.html file is loaded as the default file for a website. However, in our example code, this is overriden by the below (if you remove the below, index.html will become the default home page again):

(hunchentoot:define-easy-handler (index :uri "/") (info)
    (setf (hunchentoot:content-type*) "text/html")
      (let ((the-user (hunchentoot:session-value :user)))
         (format nil 
"<h4>welcome-to-the-small-demo</h4><hr>
<p>info:~A</p>
<p>user:~A</p>
<a href=\"/sign-up\">click-here-to-sign-up</a><br>
<a href=\"/sign-in\">click-here-to-sign-in</a><br>
<a href=\"/sign-out\">click-here-to-sign-out</a><br>" info the-user)))

Concluding Remarks

Our example servers were very barebone, but contained all the raw functionality to build very advanced web applications.

You can see that there are so many different parts that we need to join together to build web applications. Hopefully the above helped, and you can now focus on the fun stuff — building the actual web page and application!

If you like the guide, please star the repo.

Appendix

Here are some additional personal notes for my own setup, which you may be able to use, but are not central to the guide.

SSH into GCE via MacOS Terminal

First we need to obtain an SSH key pair for our Mac (you may already have one — in which case do not do the following as it will cause issues with your other programs).

In Terminal, first I cleared all my old keys (this is an optional step that you probably should never do, but I’m listing here for my own notes):

$ ssh-add -D

We can list all identities with the following:

$ ssh-add -l

I had to then follow the following steps: https://cloud.google.com/compute/docs/instances/managing-instance-access

To create a new one: