[Openmcl-devel] Debian packaging for CCL

[R. Matthew Emerson]

On Jul 17, 2012, at 1:09 AM, Faheem Mitha wrote:

> Thanks. This brings up the question - what code should I be using, and
> from where, to build the binary?
> 
> Currently, I have been using
> http://svn.clozure.com/publicsvn/openmcl/release/1.8/linuxx86/ccl, per
> the documentation in
> 
> http://ccl.clozure.com/manual/chapter2.2.html#obtaining-via-svn
> 2.2.3.2.2. Downloading a Release Version
> 
> which appears to be equivalent to the tarball in
> ftp://ftp.clozure.com/pub/release/1.8/ccl-1.8-linuxx86.tar.gz
> 
> Debian tends to prefer upstream tarballs, so this latter tarball would
> appear to be the version to choose. However, the problem is that you
> ship binaries along with the source code in these versions. Granted,
> this is useful to build on a system which does not already have a
> working CCL compiler installed, but it has the complicating factor
> that on build the existing binaries, in this case lx86cl, lx86cl.image
> are overwritten by the build, which causes problems. Debian expects
> that there is a clean target in debian/rules (the debian build
> script), which returns the upstream source to a pristine condition, as
> it originally was before build. This is much more difficult if the
> binaries are overwritten on build, as is the case here. This is
> probably a question to ask Debian, but perhaps you have an opinion
> too.
> 
> Note that I have worked around the existence of the binaries by
> writing the build files elsewhere, but this is been a pain, and
> obviously it would be preferable not to have to do so.
> 
> However, using just the sources as you have listed them, then raises
> the question - what compiler do I use to build with - do I use the
> binary packaged version of CCL in debian to build the source for
> itself? That seems a little weird.

This is a very long reply.  I seem to be developing the
tendency to talk too much.

Let me review how ccl's sources are organized and how ccl is
built.

In the URLs I'll be using, I'll refer to the trunk, but if you
substitute "release/1.8/" for "trunk/", then everything should
still be valid.

The source files for ccl are found at the following URL:

http://svn.clozure.com/publicsvn/openmcl/trunk/source

Now, this source code has to be compiled by something, of course.

As described in the manual and elsewhere, ccl consists of two
parts: the lisp kernel and the heap image.

The lisp kernel contains the GC, thread support, and other
runtime support functions.  It is an ordinary C (and assembly
language) program. The only slightly odd thing about it is that
it uses the m4 macro processor.  It's easy to build:

cd ccl/lisp-kernel/linuxx8664 && make

Of course, you'd use linuxx8632 if you wanted to build for 32-bit
x86; you'd use linuxarm for ARM, linuxppc for for 32-bit PowerPC,
etc.  You get the idea.

It's convenient for make's default target to overwrite the
existing lisp kernel binary in the top-level ccl directory, but
if it would help, I'd be willing to add some other target so that
you could say "make build" or something that would leave the
binary in the lisp-kernel/linnuxx8664 directory.

Note that since it's possible to build the lisp kernel binary
with the normal Unix C toolchain, it's not strictly necessary to
distribute a lisp kernel binary for bootstrapping.

Anyway, so much for the lisp kernel.

The rest of CCL is written in lisp, so it needs to be compiled by
a lisp compiler, and that lisp compiler needs to be CCL.  In
principle, one could write the lisp code that implements CCL so
that it could be compiled by other Common Lisp compilers, but we
don't make any effort to do that.

As Tim Bradshaw mentioned, this is nothing very unusual for
compilers that are written in the language that they themselves
compile.  In the case of GCC, it used to be the case (and maybe
still is) that you could use the (possibly pre-ANSI C) system C
compiler to compile the GCC sources, giving you a version of GCC
which you would then use to compile itself.  When I used
workstations running Sun OS 4.1.x, I did this all the time.
Sun's /usr/bin/cc was pretty old and simple-minded (and didn't
support ANSI C), and was generally considered to be fit only for
relinking the kernel and compiling GCC.  The fact that Solaris 2
didn't include any sort of C compiler was yet another reason for
Sun OS 4 people to hate it when it came out---it made it that
much harder to get GCC running on it.  Sun did sell a good C
compiler that was (and may still be) better than GCC, but after
getting Solaris 2 shoved down our throats, there wasn't much
willingness to give Sun more money to restore a feature (viz., a
C compiler) that we already had in Sun OS 4.  Solaris eventually
did get better, by the way, and I actually like it now.  (And I
like where it's going with the illumos project.)

But I digress.  I was supposed to be talking about the heap
image.

The heap image contains all the lisp code and data that make up
CCL.  At startup time the lisp kernel maps the heap image into
memory, and starts executing code in it.  Since the lisp compiler
is contained in the heap image, we have to distribute a heap
image file so that the ccl can compile itself.

In addition to the heap image file, we also need the interface
databases, which make the #_ and #$ and #& reader macros work.
These reader macros are used in the CCL sources, so these
databases have to be present for CCL to compile itself. Users
don't normally rebuild the interfaces, but it's not terribly
difficult to do: http://trac.clozure.com/ccl/wiki/BuildFFIGEN
explains how.

So, we're now looking at getting three things at a minimum
before we can possibly rebuild ccl:

1. The source code

2. A heap image

3. The interface databases

In order to make getting a working ccl as easy as possible, we
arranged things so that a user can do a single svn checkout, and
get everything needed to run ccl.

svn co http://svn.clozure.com/publicsvn/openmcl/trunk/linuxx86/ccl

To see what's going on with this, browse the repository via the
Trac:

http://trac.clozure.com/ccl/browser/trunk/linuxx86/ccl

You'll see that this URL refers to a directory that contains lisp
kernel and heap image binaries, and uses svn externals to bring
in the interface databases and the various source code
subdirectories.

There's a lot to be said for having a single-command way to
get ccl.

Unfortunately, this arrangement (binaries, plus externals) it
makes actually hacking on ccl somewhat awkward.  That's the
reason for the scripts in scripts/http-to-ssh and
scripts/http-to-svn.  The externals are always fetched via
"http://" URLs, and our svn server isn't configured to allow
write access via "http://".  Those scripts switch the URLs to use
either "svn+ssh://" or "svn://" so that it's possible to commit.

The other unfortunate side effect is that rebuilding CCL from
this one-step checkout will make svnversion report a number like
"15212M", where the "M" means that the working copy has been
modified.  This is because the working copy contains the
binaries, and once those binaries are changed by being rebuilt,
svn sees them as having been modified.  Ideally, we'd like to see
the "M" only when the sources have been locally modified.

Now, I hack on CCL itself, which most people probably do not do.
For most, the single-command method is convenient and effective.
But, that said, what I myself actually do is this:

1.  Check out the sources.  I use the svn:// scheme so that I can
    commit.  (Note that the "publicsvn" component of the URL used
    for the http: scheme is not used with the svn: scheme.)

    svn co svn://svn.clozure.com/openmcl/trunk/source ccl

2.  Get bootstrapping binaries from somewhere.

    One way to get them is via the following checkout:

    svn co --ignore-externals \
     svn://svn.clozure.com/openmcl/trunk/linuxx86/ccl bootstrap
    cp bootstrap/* ccl/

3.  Get interface databases from somewhere.

    cd ccl
    svn co svn://svn.clozure.com/openmcl/trunk/x86-headers
    svn co svn://svn.clozure.com/openmcl/trunk/x86-headers64

At that point, I can rebuild CCL in the usual way, e.g., with
(rebuild-ccl :full t).  I can commit easily, because externals
are not in the way.  Rebuilding the binaries doesn't result in
a version number like "14531M" unless I actually have local
modifications to the sources.

I admit that we are abusing svn externals, but having a one-step
way to install CCL seems to me to be very important, and I'm
willing to complicate the life of CCL hackers a bit in order to
make that one-step installation work.