I'm missing an obvious part of ASDF / quickload / sly, and it's starting to drive me nuts.

It's mostly about the developer experience, but as I can't go past the very first steps, I'm unable to work on anything more than short examples in a short-lived repl.

So, don't hesitate to assume that I'm misunderstanding something very basic.

Also, I've spend a couple of days trying to wrap my head around this, so at this point I've probably been looking "past" the solutions a couple of time (reading exactly the right sentences of the right manual several times, but not managing to make my brains register.) So, really, have mercy on the noob.

Following the "Working with projects" section of the Common Lisp cookbook, I was able to create an asdf project.

.html#working-with-projects

The project has three source files:

• foobar.asd
• src/main.lisp
• test/main.lisp

The asd file contains two defsystem clauses.

In particular, one of the clauses defines a dependency on the "rove" library for test runner.

(defsystem "foobar/tests"
  :author ""
  :license ""
  :depends-on ("foobar"
               "rove")
  :components ((:module "tests"
                :components
                ((:file "main"))))
  :description "Test system for foobar"
  :perform (test-op (op c) (symbol-call :rove :run c)))

There is one source file for tests:

(defpackage foobar/tests/main
  (:use :cl
        :foobar
        :rove))
(in-package :foobar/tests/main)

;; NOTE: To run this test file, execute `(asdf:test-system :foobar)' in your Lisp.

(deftest test-target-1
  (testing "should (= 1 1) to be true"
    (ok (= 1 1))))

If I open the foobar/test/main.lisp file, and I want to run the tests, then as I understand it :

• the rove library must be downloaded in my computer
• the rove library must be loaded in my reply
• the foobar system must be loaded in my reply
• the foobar/tests system must be loaded in my system

One way to do that is:

• open the file
• start a repl (m-x slime or similar)
• explicitly type and (ql:quickload "rove") in the REPL
• then, explicitly type '(ASDF:load-system "foobar")' in the repl
• then, explicitly type '(ASDF:test-system foobar)'

So it seems, like, if my project depends on 10 libraries, it would mean that I would start every work session by quickload-ing all 10 libraries, then asdf-loading things for 5 minutes (including multiple typos in the reply.)

1. Is that correct ? (Or is there a proper sequence of repl / slime / sly commands that will trigger all the quicklisp and ASDF functions ?

2. Is that how people work ? Open up emacs, go through all their files, and quickload stuff as they go, ASDF load system as they go, etc... If I put the content of those files on a repo, and I tell you to extend the code, what do you do past 'git clone' ?

I'm missing an obvious part of ASDF / quickload / sly, and it's starting to drive me nuts.

It's mostly about the developer experience, but as I can't go past the very first steps, I'm unable to work on anything more than short examples in a short-lived repl.

So, don't hesitate to assume that I'm misunderstanding something very basic.

Also, I've spend a couple of days trying to wrap my head around this, so at this point I've probably been looking "past" the solutions a couple of time (reading exactly the right sentences of the right manual several times, but not managing to make my brains register.) So, really, have mercy on the noob.

Following the "Working with projects" section of the Common Lisp cookbook, I was able to create an asdf project.

https://lispcookbook.github.io/cl-cookbook/getting-started.html#working-with-projects

The project has three source files:

• foobar.asd
• src/main.lisp
• test/main.lisp

The asd file contains two defsystem clauses.

In particular, one of the clauses defines a dependency on the "rove" library for test runner.

(defsystem "foobar/tests"
  :author ""
  :license ""
  :depends-on ("foobar"
               "rove")
  :components ((:module "tests"
                :components
                ((:file "main"))))
  :description "Test system for foobar"
  :perform (test-op (op c) (symbol-call :rove :run c)))

There is one source file for tests:

(defpackage foobar/tests/main
  (:use :cl
        :foobar
        :rove))
(in-package :foobar/tests/main)

;; NOTE: To run this test file, execute `(asdf:test-system :foobar)' in your Lisp.

(deftest test-target-1
  (testing "should (= 1 1) to be true"
    (ok (= 1 1))))

If I open the foobar/test/main.lisp file, and I want to run the tests, then as I understand it :

• the rove library must be downloaded in my computer
• the rove library must be loaded in my reply
• the foobar system must be loaded in my reply
• the foobar/tests system must be loaded in my system

One way to do that is:

• open the file
• start a repl (m-x slime or similar)
• explicitly type and (ql:quickload "rove") in the REPL
• then, explicitly type '(ASDF:load-system "foobar")' in the repl
• then, explicitly type '(ASDF:test-system foobar)'

So it seems, like, if my project depends on 10 libraries, it would mean that I would start every work session by quickload-ing all 10 libraries, then asdf-loading things for 5 minutes (including multiple typos in the reply.)

1. Is that correct ? (Or is there a proper sequence of repl / slime / sly commands that will trigger all the quicklisp and ASDF functions ?

2. Is that how people work ? Open up emacs, go through all their files, and quickload stuff as they go, ASDF load system as they go, etc... If I put the content of those files on a repo, and I tell you to extend the code, what do you do past 'git clone' ?

• build
• common-lisp
• dependency-management
• asdf
• quicklisp

Share Follow edited Mar 4 at 12:29 Rainer Joswig 140k1111 gold badges228228 silver badges354354 bronze badges asked Mar 4 at 8:17 phtrivierphtrivier 13.4k99 gold badges5252 silver badges8282 bronze badges 2

• I suggest to leave aside the test framework integration into ASDF until you're good. You can still run Rove tests with its own function, independently of ASDF. You don't require the integration of the two. So, for later! (I very rarely do it myself) – Ehvince Commented Mar 5 at 13:12
• How did this question get 20k and climbing views in 4 days? That's orders of magnitude more than normal for common lisp questions – Shawn Commented Mar 9 at 3:03

Add a comment  | 

2 Answers 2

Sorted by: Reset to default 2

So it seems, like, if my project depends on 10 libraries, it would mean that I would start every work session by quickload-ing all 10 libraries, then asdf-loading things for 5 minutes

Normally you should only have to quickload your project, and all the transitive dependencies will be loaded.

There is also the convention of having a separate test system that exists because supposedly during tests you may need more dependencies that the ones for the library you publish. Knowing that, if you want to run all tests, you need only call:

(ql:quickload "foobar/tests")

And since system "foobar" is a dependency, it will also be loaded automatically.

Once it is loaded, you can execute various operations on systems, like asdf:test-system.

Is that correct ? (Or is there a proper sequence of repl / slime / sly commands that will trigger all the quicklisp and ASDF functions ?

There might be Slime shortcuts but I don't use them so I can't say, but you only need to quickload the root element of your dependency tree.

Is that how people work ? Open up emacs, go through all their files, and quickload stuff as they go, ASDF load system as they go, etc... If I put the content of those files on a repo, and I tell you to extend the code, what do you do past 'git clone' ?

So I went to the Awesome CL Software page and picked something relatively small (large projects tend to have more setup requirements, hopefully they document it). I cloned ALIW (A Lisp in Wonderland), some Wiki software which I don't know. It seems to be a bit old and not maintained.

I go to the local-projects directory for quicklisp, which will simplify the discovery of the system. I could have used a different approach but this is the simplest:

$ cd $HOME/quicklisp/local-projects

There, I clone the project:

$ git clone https://github/vy/aliw

I see that there is an aliw.asd file which defines a system named "aliw", with a lot of dependencies. I happen to have a Lisp instance already running, and from the Slime REPL, I write the following:

CL-USER> (ql:quickload "aliw")

Then, a lot of text is emitted showing that quicklisp is fetching, compiling and loading code (all the dependencies), until I reach my first error:

; caught ERROR:
;   READ error during COMPILE-FILE:
;   
;     Package HUNCHENTOOT-MP does not exist.
;   
;       Line: 74, Column: 27, File-Position: 3006
;   
;       Stream: #<SB-INT:FORM-TRACKING-STREAM for "file /home/christophe/quicklisp/local-projects/aliw/src/specials.lisp" {1005C5C7E3}>

There are also available restarts:

COMPILE-FILE-ERROR while
compiling #<CL-SOURCE-FILE "aliw" "src" "specials">
   [Condition of type UIOP/LISP-BUILD:COMPILE-FILE-ERROR]

Restarts:
 0: [RETRY] Retry compiling #<CL-SOURCE-FILE "aliw" "src" "specials">.
 1: [ACCEPT] Continue, treating compiling #<CL-SOURCE-FILE "aliw" "src" "specials"> as having been successful.
 ....

I do not select any restart option, instead I'll fix the code and resume loading later.

The repository was archived some time ago was not kept up-to-date, there is effectively no hunchentoot-mp package. Seeing that function being called was make-lock, which is used to make a mutex, I fix this by editing the file and writing instead: bt:make-lock, referencing the bordeaux-threads library, which is well-known and provides locks. I also edit the .asd file to add bordeaux-threads library.

(asdf:defsystem :aliw
  :version +aliw-version+
  :depends-on (:bordeaux-threads ;; << here
               :cl-difflib
               :cl-fad
               :cl-ppcre
               :cl-who
               :flexi-streams
               :hunchentoot
               :md5
               ...

Note that Hunchentoot already depends on bordeaux-threads but it is better to document the dependencies that you really need in the project, not rely on hidden transitive dependencies.

Since it is unlikely to be the only error, I also fix all other occurences of make-lock and use bt:with-recursive-lock-held wherever necessary. In the buffers that I visit in Emacs, slime-mode allows me to autocomplete things that are already loaded in the system at this point.

Now, I can resume loading the system, and I do invoke the Retry ASDF operation restart because I edited the .asd file.

SBCL warns me about a symbol being exported, due to this pattern:

(defpackage :aliw-asd
  (:use :cl :asdf))

(in-package :aliw-asd)

(defparameter +aliw-version+ "0.1.5")

(export '+aliw-version+)

When you load the file twice, the defpackage notices that +aliw-version+ is exported, but this is not declared by the package declaration. A better way to do this is to write:

(defpackage :aliw-asd
  (:use :cl :asdf)
  (:export #:+aliw-version+))

(in-package :aliw-asd)

(defparameter +aliw-version+ "0.1.5")

(asdf:defsystem :aliw ...)

I restart again after making this change, and fix the remaining errors about hunchentoot:start-server (which has been renamed start for a long time). I do the same for stop-server.

There is a warning about hunchentoot:*handle-http-errors-p* being undefined, so now I have another Yak to shave, I clone Hunchentoot and see that it was removed in 2011. I remove the code in "aliw" that uses it and make a note to send a patch to Hunchentoot.

Now, everything is loaded. Since I hacked a bit the code while loading it, I want to make sure that everything works. If I open a new terminal and start sbcl, I can quickload "aliw" almost instantly (compilation produces object files that can be loaded), without error, so my fixes seem to be good (nb. after having looked at it more, the code compiles without problem, but I would have to make more changes to have something that runs).

There was a lot of stuff to do, but notice that I was able to fix the code while I was in the process of loading it, and I only had to invoke restarts to continue compiling the code as necessary. Usually with libraries that are distributed by Quicklisp this works out of the box because the maintainer tests the distribution.

I hope this makes sense.

is there a proper sequence of repl / slime / sly commands that will trigger all the quicklisp and ASDF functions ?

You have an .asd file.

Start a REPL: this starts a Lisp image.

When you do (ql:quickload :stuff), Quicklisp looks on a couple directories if it finds such a system: ~/quicklisp/local-projects/ and ~/common-lisp/ (typical ASDF directory).

If your project is not in one of these directories, you have to instruct your lisp image about your project.

You can do so by loading the .asd file: this defines a system.

• to compile and load the code into the .asd, use C-c C-k (M-x slime-compile-and-load-file)
• or use (load "myfile.asd") but it's advised to use (asdf:load-asd "/path/to/myfile.asd").

Now ql:quickload can load it.

If you want to avoid this last step (load the .asd), you can place your project into ~/quicklisp/local-projects/, or create a symlink, or do this:

;; .sbclrc
(pushnew "~/projets/myproject/" asdf:*central-registry* :test #'equal)

in your ~/.sbclrc. Now everytime SBCL starts, it adds a directory to where ASDF (and thus Quicklisp) look for projects. You can ql:quickload it right away.

---

If I put the content of those files on a repo, and I tell you to extend the code, what do you do past 'git clone' ?

If I cloned it anywhere (on a non-meaningful directory for ASDF/Quicklisp), I open the .asd, type C-c C-k, quickload the system, and start working.