I've just released a new version of Declt, my Texinfo reference manual generator for ASDF systems. This release contains only one bugfix: when trying to create links to source files, Declt now checks whether the files actually exist or not.

Tracking this bug down had the side-effect of exhibiting a misfeature of SBCL's introspection facility: the COPY-<struct> functions (automatically generated by defstruct calls) have their definition source set to target-defstruct.lisp which is an SBCL source file. It would make more sense to set it to the file containing the defstruct call instead, as is already the case for constructors, predicates and accessor functions. Patch sent to the SBCL developers.

UPDATE: since the original publication of this blog entry, Nikodemus Siivola and I have done some more work on various other aspects of Common Lisp indentation, and I must say that the result is pretty satisfactory. Nikodemus has merged all the changes into Slime, and I have done so for XEmacs. If you're an XEmacs user, you don't need to use the slime-indentation contrib to get these improvements. Simply upgrade the "prog-modes" package and load cl-indent.el.

I have just modified XEmacs to improve the indentation of Common Lisp code. This change involves two things: the support for multiple method qualifiers in a call to defmethod and, much more importantly, a cleaner and more flexible scheme for indenting lambda lists. The patch has also been submitted to the GNU Emacs developers. Below is a more detailed description of what you get with these changes.

Method qualifiers

Until now, only one method qualifier was understood. Below are some examples demonstrating that the one method qualifier and the argument list are indented by 4 characters, and the method's body only by 2:

(defmethod foo :around ()        (defmethod foo :around
  do-this)                           ()
                                   do-this)
(defmethod foo
    :around ()                   (defmethod foo
  do-this)                           :around
                                     ()
                                   do-this)

Now let's add a second method qualifier:

(defmethod foo comb :around ()        (defmethod foo	    
	   do-this)			  comb :around () 
					  do-this)

Woops. Neither is correct. But now, you get this instead:

(defmethod foo comb :around ()        (defmethod foo	    
  do-this)				  comb :around () 
					do-this)

Three more examples to show how confused we were:

(defmethod foo comb :around     (defmethod foo comb     (defmethod foo 
  ()				    :around		    comb       
  do-this)			  ()			    :around    
				  do-this)		  ()	       
							  do-this)

And how better we just got:

(defmethod foo comb :around     (defmethod foo comb     (defmethod foo
    ()				    :around		    comb      
  do-this)			    ()			    :around   
				  do-this)		    ()	      
							  do-this)

Indeed, you can see that everything between the method's name and its body is now correctly indented by 4.

Lambda Lists

The next round of changes deals with the formatting of lambda-lists. As such, this will apply everywhere a lambda-list is expected, such as defun, defgeneric, defmethod etc. First of all, here are two examples showing that we were not very clever before:

(defun foo (mand1 mand2
	    &optional opt1
	    opt2
	    &rest args
	    &key key1 key2
	    (key3 val3) &aux aux1
	    aux2)
  do-this)

(defun foo (mand1 mand2 &optional opt1
	    opt2
	    &rest args
	    &key key1 key2
	    (key3 val3) &aux aux1
	    aux2)
  do-this)

Basically, everything was blindly aligned at the beginning of the lambda-list. Here is what you get now:

(defun foo (mand1 mand2
	    &optional opt1
	      opt2
	    &rest args
	    &key key1 key2
	      (key3 val3) &aux aux1
			    aux2)
  do-this)

(defun foo (mand1 mand2 &optional opt1
			  opt2
	    &rest args
	    &key key1 key2
	      (key3 val3) &aux aux1
			    aux2)
  do-this)

The difference is that keyword parameters are indented with respect to their corresponding keyword. The amount of indentation is provided by a new customizable user option named lisp-lambda-list-keyword-parameter-indentation (oh boy, what a mouthful). If you prefer, you can also have the parameters vertically aligned with each other. Set the new customizable user option named lisp-lambda-list-keyword-parameter-alignment to t and you will get this instead:

(defun foo (mand1 mand2
	    &optional opt1
		      opt2
	    &rest args
	    &key key1 key2
		 (key3 val3) &aux aux1
				  aux2)
  do-this)

(defun foo (mand1 mand2 &optional opt1
				  opt2
	    &rest args
	    &key key1 key2
		 (key3 val3) &aux aux1
				  aux2)
  do-this)

Finally, just as you could align keyword parameters together, you can also align the keywords together. Set the new customizable user option named lisp-lambda-list-keyword-alignment to t, and you will get this (only the second example differs):

(defun foo (mand1 mand2 &optional opt1
				  opt2
			&rest args
			&key key1 key2
			     (key3 val3) &aux aux1
					      aux2)
  do-this)

These are in fact my preferred settings, although both alignment options default to nil. Here is a final example demonstrating how you could format a long lambda-list with plenty of arguments:

(defun foo (mand1 mand2
	    &optional
	      opt1 opt2 opt3
	    &key
	      key1 key2 key3
	    &aux
	      aux1 aux2 aux3)
  do-this)

These indentation problems have been a huge peeve of mine for quite a long time. I hope you will find the changes useful!

Dynamic Languages Symposium 2011

Co-located with SPLASH 2011 In association with ACM SIGPLAN

Portland, Oregon, USA, October 24, 2011

http://www.dynamic-languages-symposium.org/dls-11/

Call for papers

The 7th Dynamic Languages Symposium (DLS) at SPLASH 2011 is a forum for discussion of dynamic languages, their implementation and application. While mature dynamic languages including Smalltalk, Lisp, Scheme, Self, Prolog, and APL continue to grow and inspire new converts, a new generation of dynamic scripting languages such as Python, Ruby, PHP, Tcl, Lua, and JavaScript are successful in a wide range of applications. DLS provides a place for researchers and practitioners to come together and share their knowledge, experience, and ideas for future research and development.

DLS 2011 invites high quality papers reporting original research, innovative contributions or experience related to dynamic languages, their implementation and application. Accepted Papers will be published in the ACM Digital Library.

Areas of interest include but are not limited to:

• Innovative language features and implementation techniques
• Development and platform support, tools
• Interesting applications
• Domain-oriented programming
• Very late binding, dynamic composition, and runtime adaptation
• Reflection and meta-programming
• Software evolution
• Language symbiosis and multi-paradigm languages
• Dynamic optimization
• Hardware support
• Experience reports and case studies
• Educational approaches and perspectives
• Object-oriented, aspect-oriented, and context-oriented programming

Submissions and proceedings

We invite original contributions that neither have been published previously nor are under review by other refereed events or publications. Research papers should describe work that advances the current state of the art. Experience papers should be of broad interest and should describe insights gained from substantive practical applications. The program committee will evaluate each contributed paper based on its relevance, significance, clarity, and originality.

Accepted papers will be published in the ACM Digital Library.

Papers are to be submitted electronically in PDF format. Submissions must not exceed 12 pages and need to use the ACM format, templates for which can be found at http://www.acm.org/sigs/pubs/proceed/template.html.

Important dates

Submission of papers: June 17, 2011 (hard deadline) Author notification: July 19, 2011 Final versions due: August 19, 2011 DLS 2011: October 24, 2011 SPLASH 2011: October 22-27, 2011

Program chair

Theo D'Hondt, Software Languages Lab, Vrije Universiteit Brussel, Belgium

Program committee

• Andrew Black, Portland State University, USA
• William R. Cook, University of Texas at Austin, USA
• Marc Feeley, University of Montreal, Canada
• Roberto Ierusalimschy, PUC-Rio, Brazil
• Michele Lanza, University of Lugano, Switzerland
• Hidehiko Masuhara, University of Tokyo, Japan
• Mira Mezini, University of Darmstadt, Germany
• Mark Miller, Google, USA
• Manuel Serrano, INRIA Nice, France
• Laurence Tratt, Middlesex University, UK
• David Ungar, IBM, USA
• Didier Verna, EPITA Research and Development Laboratory, France

The final programme for the 4th European Lisp Symposium is now online! 2 full days of keynotes, technical papers, demonstrations, lightning talks and panels with a special focus on performance, efficiency, parallelism, concurrency, distribution...

Early bird registration is still available until March 12th so hurry!

Hope to see you there.

The 4th European Lisp Symposium is now open for registration! See the website for details.

The early registration deadline is March 12. There is also a reduced fee for students and accompanying persons.

The complete program will be available shortly.

Hope to see you there!

Clon version 1.0b19 has just been released. This version comes with a couple of fixes related to error handling and a switch from the GPL to the BSD license. Grab it here.

Clon v1.0b18 is now out. Compared to the previous version, the only small bit of change is the fact that the CLISP implementation now depends only optionally on cffi, whereas that dependency was mandatory before. Doing this is actually quite simple but raised the question of optional ASDF system dependencies, a topic on which there's quite a bit to be said.

A bit of context first. Clon has a terminal autodetection feature which allows it to automatically highlight its output on a tty (see --clon-highlight=auto) and also compute the output line width in case the COLUNMS environment variable is not set. This feature requires an ioctl call which is beyond Lisp. Since one of my goals is to have as few dependencies as possible (zero being the ideal number), I looked at compiler-specific solutions to this problem. The result is that SBCL, CMU-CL, CCL and ECL provide solutions almost out-of-the-box: SBCL has a grovel contributed module, CMU-CL and CCL already have system bindings and ECL has its own ffi interface. The ABCL port doesn't support terminal autodetection yet, but that's another (Java) story. The only black sheep in the picture is CLISP which, as far as I can tell, neither has a native binding for ioctl, nore any built-in grovel facility, hence the need for cffi.

Previously, my system definition file was like that:

#+clisp (asdf:operate 'asdf:load-op :cffi-grovel)
 
(asdf:defsystem :com.dvlsoft.clon
  #| ... |#
  :depends-on (#+clisp :cffi)
  :components ((:file "package")
			#+clisp
			(:module "clisp"
			   :depends-on ("package")
			   :components ((cffi-grovel:grovel-file "constants")))
			(module "src"
                          :depends-on ("package" #+clisp "clisp")
                          :components #| ... |#)))

And then later, I had a file with the concerned utility function:

(defun stream-line-width (stream)
  #+clisp (#| CLISP implementation |#)
  #| etc. |#)

After that, I started looking at ways to make the dependency on cffi optional. After all, it makes sense to avoid that dependency at the cost of not being able to autodetect terminals.

One thing I fell on almost by accident is ASDF's :weakly-depends-on keyword. Here is an interesting thread about it. It was an accident because that feature is not documented :-( People, however, have very mitigated feelings about it, as shown in this other thread (syntax and semantics both appear somewhat shaky and underspecified). Besides, the aparent behavior is that if A weakly depends on B and B is not present, then A is operated anyway. So I could probably have my "src" module weakly depend on the "clisp" one, but that doesn't change the fact the "clisp" module should not be included at all if cffi is not there.

In the same thread, somebody proposes another kind of dependency called :contigent-on which looks closer to what I would need for the "clisp" module: a module the contingency of which is not present will not be processed. This new kind of dependency doesn't seem to be implemented yet, however.

So, all of this seems a bit frightening. Too borderline for my taste. Fortunately, my solution is much simpler, although probably not universal.

This first thing to do is only attempt to load cffi-grovel:

(eval-when (:load-toplevel :execute)
  #+clisp (handler-case (asdf:operate 'asdf:load-op :cffi-grovel)
	    (asdf:missing-component ()
	      (format *error-output* "~
*********************************************************************
* WARNING: ASDF component CFFI-GROVEL not found.
* Clon will be loaded without support for terminal autodetection.
* See section A.1 of the user manual for more information.
*********************************************************************"))))

After that, if loading it were successful, we end up with cffi as a feature, so we can just conditionalize on that:

(asdf:defsystem :com.dvlsoft.clon
  #| ... |#
  :depends-on (#+(and clisp cffi) :cffi)
  :components ((:file "package")
			#+(and clisp cffi)
			(:module "clisp"
			   :depends-on ("package")
			   :serial t
			   :components ((cffi-grovel:grovel-file "constants")))
			(module "src"
                          :depends-on ("package" #+(and clisp cffi) "clisp")
                          :components #| ... |#)))

One last problem remains however: what to do in the source code, for the feature-dependent parts. Conditionalizing an ASDF system may indeed lead to trouble: for instance, what would happen if the function stream-line-width was compiled with cffi around, and later used in a context where it is not? To be on the safe side, what you really need is to dynamically check for the feature. One possible solution is this:

1. move all feature-dependent code to the "clisp" module and make that a protocol,
2. everytime you need to access the feature, dynamically check whether the protocol functions are fbound.

In my specific case, what I did was to implement a CLISP-specific version of stream-line-width, called clisp/stream-line-width and put it in a new file in the "clisp" module, now defined as follows:

#+(and clisp cffi)
	       (:module "clisp"
		 :depends-on ("package")
		 :serial t
		 :components ((cffi-grovel:grovel-file "constants")
					(:file "util")))

Then, the original function is rewritten like this:

(defun stream-line-width (stream)
  #+clisp (when (fboundp 'clisp/stream-line-width) (clisp/stream-line-width stream))
  #| etc. |#)

So now I think I'm on the safe side, and Clon has zero mandatory dependency again...

UPDATE: ABCL now supports the POSIX-compliant use of --, so I've modified this blog to reflect that. I've also added a final trick to emulate the existence of a proper argv[0].

Creating standalone executables in Common Lisp is already a portability minefield, as the standard doesn't specify anything in that regard. The situation is even worse with ABCL because it is a Java-based implementation of Common Lisp, and Java has no notion of standalone executable at all. In this article, we will try to figure out how far we can go in that direction nevertheless. This work comes from my wish to port Clon to ABCL. In order to make it work, ABCL itself needed some changes, so if you want to try out the code provided below, beware that you need at least revision 13156 from the ABCL Subversion trunk (or version 0.24.0). Let's start by writing a great Common Lisp application for ABCL, and call it greet.lisp:

(defun main ()
  (format t "Hello,~{ ~A~}!~%" extensions:*command-line-argument-list*)
  (extensions:exit :status 0))
 
(main)

As you guessed, extensions:*command-line-argument-list* stores the list of command-line options passed to the program. Now, let's try it out:

$ abcl --noinform --noinit --nosystem --load greet.lisp John Doe
Hello, John Doe!

So far so good: ABCL-specific options were eaten by ABCL. Not very standalone however. Let's create a wrapper script around that, and call it greet:

#! /bin/sh
exec abcl --noinform --noinit --nosystem --load greet.lisp ${1+$@}

In fact, we have a problem:

$ ./greet --help me
Parameters:
--help displays this help
--noinform suppresses the printing of version info
--eval <form> evaluates the <form> before initializing REPL
--load <file> loads the file <file> before initializing REPL
--load-system-file <file> loads the system file <file> before initializing REPL
--batch enables batch mode. The --load, --load-system-file and --eval parameters are handled, and abcl exits without entering REPL
--noinit suppresses loading a .abclrc startup file
--nosystem suppresses loading the system startup file

Woops! We were expecting to get Hello, --help me! weren't we? In other words, our own command-line options may conflict with ABCL's which is not nice. One thing we can do with a recent ABCL version is to use -- to separate ABCL's own command-line arguments from ours. Our script becomes:

#! /bin/sh
exec abcl --noinform --noinit --nosystem --load greet.lisp -- ${1+$@}

And we now get this:

$ ./greet --help me
Hello, --help me!

This is better, although we're not actually very standalone yet: our Lisp file is a separate beast, and the abcl thing is also just a wrapper script (see below). If we want to be more standalone, we will need to go the Java way.

The first thing to understand is that there's actually no such thing as an ABCL executable. ABCL is packed into a JAR (Java archive) file, a zip file, really, and what the wrapper does is simply to call java -jar /path/to/abcl/abcl.jar options... The JAR file contains a Main class with a main function the purpose of which is to launch a Lisp interpreter. This is done by calling the method Interpreter.createDefaultInstance, which, amongst other things, handles the command-line options. There is also a method called Interpreter.createInstance which does not handle the command-line at all, so it seems that this is the one we could use, provided that we find a way to set extensions:*command-line-argument-list* manually.

The following code does exactly this. Let's call it Greet.java.

import org.armedbear.lisp.*;
 
public class Greet
{
    public static void main (final String[] argv)
    {
	Runnable r = new Runnable ()
	    {
		public void run()
		{
		    try
			{
			    LispObject cmdline = Lisp.NIL;
			    for (String arg : argv)
				cmdline = new Cons (arg, cmdline);
			    cmdline.nreverse ();
			    Lisp._COMMAND_LINE_ARGUMENT_LIST_.setSymbolValue (cmdline);
 
			    Interpreter.createInstance ();
			    Load.load ("greet.lisp");
			}
		    catch (ProcessingTerminated e)
			{
			    System.exit (e.getStatus ());
			}
		}
	    };
 
	new Thread (null, r, "interpreter", 4194304L).start();
    }
}

Our Greet class mimics what ABCL's Main class does, only with some tweaks of our own:

• we build the value of extensions:*command-line-argument-list* manually from the actual argv passed to the function main, by consing equivalent Lisp objects and then reversing the list. That way, we completely bypass ABCL's options processing code.
• as mentioned earlier, we use Interpreter.createInstance instead of Interpreter.createDefaultInstance.
• we also take the opportunity to load our lisp file directly with the function Load.load.

Finally, the try/catch construct intercepts calls to extensions:exit and effectively leaves the Java environment.

Let's compile this beast now:

$ javac -cp  /path/to/abcl/abcl.jar Greet.java

The -cp option to javac, the Java compiler, tells it where to find ABCL classes. This command gives us a Greet.class file, but also a second one named Greet$1.class. That's because our code creates an "anonymous inner class", whatever that means, in the Java jargon. Now, how do we bypass ABCL's main function and use or own instead? This is actually very simple. The Java command-line lets you specify (read: override) the name of the class in which to find the main function. So it turns out that we don't even have to modify ABCL itself for this to work. We can simply go:

$ java -cp  /path/to/abcl/abcl.jar:. Greet John Doe
Hello, John Doe!

and it works. By the way, we can also go:

$ java -cp  /path/to/abcl/abcl.jar:. Greet --help me
Hello, --help me!

and see that it still works. Since ABCL doesn't process its command-line anymore, there's no question about -- or whatever. Note that this time, we need to add the current directory to the class path option in order for Java to find the files Greet.class and Greet$1.class. Note also that with our custom interepreter, there's no more need for all those "standalone" options like --noinit, --noinform etc. Wrapping this in a shell script is left as an exercise, although not a difficult one.

Still, we're not quite standalone yet. For example, it's not cool to have these two Greet class files floating around like this. All other class files are in the ABCL JAR file, so we can think of adding ours in too. If we want to do that, we also need to state that the main class in the JAR file should be our Greet class, not ABCL's Main one. This can be done by modifying the so-called JAR manifest.

Let's create a manifest file first and call it Greet.txt:

Main-Class: Greet

It says what it says: our application's Main class (where the main function is) is Greet. Now let's make a copy of abcl.jar and add our class file to it:

$ cp /path/to/abcl/abcl.jar greet.jar
$ jar umf Greet.txt greet.jar Greet.class Greet$1.class
Jan 22, 2011 5:50:32 PM java.util.jar.Attributes read
WARNING: Duplicate name in Manifest: Main-Class.
Ensure that the manifest does not have duplicate entries, and
that blank lines separate individual sections in both your
manifest and in the META-INF/MANIFEST.MF entry in the jar file.

The u stands for "update", the m stands for "manifest" and the f stands for target (JAR) "file". The manifest and JAR files must appear in the same order as their corresponding options. Don't worry about the warning. That's because we're changing the Main class (ABCL had one already).

Let's try it out:

$ java -jar  greet.jar John Doe
Hello, John Doe!

Cool. But what about the Lisp file? It's still floating around. As a matter of fact, the JAR file may contain anything you want, not just class files. ABCL itself has a bunch of lisp files in its JAR file already. So we can put it in there just as well:

$ jar uf greet.jar greet.lisp

The last question is: how to have ABCL load a lisp file that's in its own JAR? The answer is: dig into the code and figure out that there is a method called Load.loadSystemFile which does just that. ABCL uses it to load its own boot.lisp for instance. We hence need to modify our main function and replace the line that calls Load.load with this one:

Load.loadSystemFile ("/greet.lisp", false, false, false);

Note the leading slash. It is needed because we have stored greet.lisp at the root of the JAR file. Let's update our JAR file once again:

$ javac -cp /path/to/abcl/abcl.jar Greet.java
$ jar uf greet.jar Greet.class Greet$1.class

And voilà! What you get this time is your application completely packed up in the JAR file. You can move this file around, distribute it... everything's inside.

Here's a final additional trick and I think we're done. As you know, ABCL stores the command-line argument list in extensions:*command-line-argument-list*. One potential problem is that there's no notion of a program name (argv[0]) in ABCL (the program is in fact always Java). Since we're going standalone, however, maybe we would like to have that. Something like, say, a variable named extensions:*argv0*. So here's a way to do it (Clon does this as part of its built-in dump facility): in the Greet class, replace the line

Interpreter.createInstance ();

with this:

Interpreter interpreter = Interpreter.createInstance ();
interpreter.eval ("(defvar extensions::*argv0* \"greet\")");
interpreter.eval ("(export 'extensions::*argv0* 'extensions)");

So that's it I guess. It seems to me that this is as far as you can go in the direction of standalone ABCL executables. Of course, this example is overly simple. We didn't try to incorporate full ASDF systems in the archive, and stuff. But you get the idea... I've attached my custom interpreter class below if you want to play with it.

Have fun!