[Openmcl-devel] Debian packaging for CCL

[Faheem Mitha]

Hi Matthew,

Thanks for the helpful reply.

On Tue, 17 Jul 2012, R. Matthew Emerson wrote:

>
> On Jul 17, 2012, at 1:09 AM, Faheem Mitha wrote:
>
>> Thanks. This brings up the question - what code should I be using, and
>> from where, to build the binary?
>>
>> Currently, I have been using
>> http://svn.clozure.com/publicsvn/openmcl/release/1.8/linuxx86/ccl, per
>> the documentation in
>>
>> http://ccl.clozure.com/manual/chapter2.2.html#obtaining-via-svn
>> 2.2.3.2.2. Downloading a Release Version
>>
>> which appears to be equivalent to the tarball in
>> ftp://ftp.clozure.com/pub/release/1.8/ccl-1.8-linuxx86.tar.gz
>>
>> Debian tends to prefer upstream tarballs, so this latter tarball would
>> appear to be the version to choose. However, the problem is that you
>> ship binaries along with the source code in these versions. Granted,
>> this is useful to build on a system which does not already have a
>> working CCL compiler installed, but it has the complicating factor
>> that on build the existing binaries, in this case lx86cl, lx86cl.image
>> are overwritten by the build, which causes problems. Debian expects
>> that there is a clean target in debian/rules (the debian build
>> script), which returns the upstream source to a pristine condition, as
>> it originally was before build. This is much more difficult if the
>> binaries are overwritten on build, as is the case here. This is
>> probably a question to ask Debian, but perhaps you have an opinion
>> too.
>>
>> Note that I have worked around the existence of the binaries by
>> writing the build files elsewhere, but this is been a pain, and
>> obviously it would be preferable not to have to do so.
>>
>> However, using just the sources as you have listed them, then raises
>> the question - what compiler do I use to build with - do I use the
>> binary packaged version of CCL in debian to build the source for
>> itself? That seems a little weird.
>
> This is a very long reply.  I seem to be developing the
> tendency to talk too much.
>
> Let me review how ccl's sources are organized and how ccl is
> built.
>
> In the URLs I'll be using, I'll refer to the trunk, but if you
> substitute "release/1.8/" for "trunk/", then everything should
> still be valid.

> The source files for ccl are found at the following URL:

> http://svn.clozure.com/publicsvn/openmcl/trunk/source

> Now, this source code has to be compiled by something, of course.

> As described in the manual and elsewhere, ccl consists of two
> parts: the lisp kernel and the heap image.

> The lisp kernel contains the GC, thread support, and other
> runtime support functions.  It is an ordinary C (and assembly
> language) program. The only slightly odd thing about it is that
> it uses the m4 macro processor.  It's easy to build:

> cd ccl/lisp-kernel/linuxx8664 && make

> Of course, you'd use linuxx8632 if you wanted to build for 32-bit
> x86; you'd use linuxarm for ARM, linuxppc for for 32-bit PowerPC,
> etc.  You get the idea.

> It's convenient for make's default target to overwrite the
> existing lisp kernel binary in the top-level ccl directory, but
> if it would help, I'd be willing to add some other target so that
> you could say "make build" or something that would leave the
> binary in the lisp-kernel/linnuxx8664 directory.

Thanks for the offer, but my feeling is that building directly from
source with some extra "bits" as necessary, as you do at the end of
the message, is what would be preferred by Debian. If this is the
case, then writing the binaries in the same place you currently put
them would not be a problem, since in the case of the source, they
would not already be there, and hence would not be overwritten.

It sounds from advice I've received on debian mentors and elsewhere,
that Debian expects these extra bits to be packaged separately as
"throwaway" binary packages, though I'm sketchy on details.

> Note that since it's possible to build the lisp kernel binary
> with the normal Unix C toolchain, it's not strictly necessary to
> distribute a lisp kernel binary for bootstrapping.

True.

> Anyway, so much for the lisp kernel.
>
> The rest of CCL is written in lisp, so it needs to be compiled by
> a lisp compiler, and that lisp compiler needs to be CCL.  In
> principle, one could write the lisp code that implements CCL so
> that it could be compiled by other Common Lisp compilers, but we
> don't make any effort to do that.

I'm curious if this is intrinsically difficult, or just done that way
for historical reasons.

> As Tim Bradshaw mentioned, this is nothing very unusual for
> compilers that are written in the language that they themselves
> compile.  In the case of GCC, it used to be the case (and maybe
> still is) that you could use the (possibly pre-ANSI C) system C
> compiler to compile the GCC sources, giving you a version of GCC
> which you would then use to compile itself.  When I used
> workstations running Sun OS 4.1.x, I did this all the time.  Sun's
> /usr/bin/cc was pretty old and simple-minded (and didn't support
> ANSI C), and was generally considered to be fit only for relinking
> the kernel and compiling GCC.  The fact that Solaris 2 didn't
> include any sort of C compiler was yet another reason for Sun OS 4
> people to hate it when it came out---it made it that much harder to
> get GCC running on it.  Sun did sell a good C compiler that was (and
> may still be) better than GCC, but after getting Solaris 2 shoved
> down our throats, there wasn't much willingness to give Sun more
> money to restore a feature (viz., a C compiler) that we already had
> in Sun OS 4.  Solaris eventually did get better, by the way, and I
> actually like it now.  (And I like where it's going with the illumos
> project.)

The first C compiler I ever used was on a Sun machine in 1997. It was
definitely a Sun compiler, but I don't know what the OS was.

> But I digress.  I was supposed to be talking about the heap
> image.

I think it would be nice to have a similar system for a Lisp
compiler. I.e. have a basic CCL CL compiler that can be compiled by
any other CL compiler, which can then be used to compile the full CCL
compiler. The current system seems to be problematic if one wants to
port CCL to a new arch, for example.

> The heap image contains all the lisp code and data that make up
> CCL.  At startup time the lisp kernel maps the heap image into
> memory, and starts executing code in it.  Since the lisp compiler
> is contained in the heap image, we have to distribute a heap
> image file so that the ccl can compile itself.
>
> In addition to the heap image file, we also need the interface
> databases, which make the #_ and #$ and #& reader macros work.
> These reader macros are used in the CCL sources, so these
> databases have to be present for CCL to compile itself. Users
> don't normally rebuild the interfaces, but it's not terribly
> difficult to do: http://trac.clozure.com/ccl/wiki/BuildFFIGEN
> explains how.
>
> So, we're now looking at getting three things at a minimum
> before we can possibly rebuild ccl:
>
> 1. The source code
>
> 2. A heap image
>
> 3. The interface databases
>
> In order to make getting a working ccl as easy as possible, we
> arranged things so that a user can do a single svn checkout, and
> get everything needed to run ccl.
>
> svn co http://svn.clozure.com/publicsvn/openmcl/trunk/linuxx86/ccl
>
> To see what's going on with this, browse the repository via the
> Trac:
>
> http://trac.clozure.com/ccl/browser/trunk/linuxx86/ccl
>
> You'll see that this URL refers to a directory that contains lisp
> kernel and heap image binaries, and uses svn externals to bring
> in the interface databases and the various source code
> subdirectories.
>
> There's a lot to be said for having a single-command way to
> get ccl.
>
> Unfortunately, this arrangement (binaries, plus externals) it
> makes actually hacking on ccl somewhat awkward.  That's the
> reason for the scripts in scripts/http-to-ssh and
> scripts/http-to-svn.  The externals are always fetched via
> "http://" URLs, and our svn server isn't configured to allow
> write access via "http://".  Those scripts switch the URLs to use
> either "svn+ssh://" or "svn://" so that it's possible to commit.
>
> The other unfortunate side effect is that rebuilding CCL from
> this one-step checkout will make svnversion report a number like
> "15212M", where the "M" means that the working copy has been
> modified.  This is because the working copy contains the
> binaries, and once those binaries are changed by being rebuilt,
> svn sees them as having been modified.  Ideally, we'd like to see
> the "M" only when the sources have been locally modified.
>
> Now, I hack on CCL itself, which most people probably do not do.
> For most, the single-command method is convenient and effective.
> But, that said, what I myself actually do is this:
>
> 1.  Check out the sources.  I use the svn:// scheme so that I can
>    commit.  (Note that the "publicsvn" component of the URL used
>    for the http: scheme is not used with the svn: scheme.)
>
>    svn co svn://svn.clozure.com/openmcl/trunk/source ccl
>
> 2.  Get bootstrapping binaries from somewhere.
>
>    One way to get them is via the following checkout:
>
>    svn co --ignore-externals \
>     svn://svn.clozure.com/openmcl/trunk/linuxx86/ccl bootstrap
>    cp bootstrap/* ccl/

This is just the lisp kernel and the heap image. Maybe I could just
use the heap image, since, as you mention, the lisp image can be built
from the source.

> 3.  Get interface databases from somewhere.
>
>    cd ccl
>    svn co svn://svn.clozure.com/openmcl/trunk/x86-headers
>    svn co svn://svn.clozure.com/openmcl/trunk/x86-headers64

Are these interface headers precompiled?

> At that point, I can rebuild CCL in the usual way, e.g., with
> (rebuild-ccl :full t).  I can commit easily, because externals
> are not in the way.  Rebuilding the binaries doesn't result in
> a version number like "14531M" unless I actually have local
> modifications to the sources.

This sounds like a reasonable way to go. As I said above, it seems the
Debian way in this case is to:

1) package the source as a source package, which consists of
*.orig.tar.gz, *.dsc (Debian source control file) and *.debian.tar.gz
(Debian specific changes to the source, sometimes only the debian
directory).

2) Package other build time dependencies as binary packages. This
would correspond to your bootstrapping binaries and headers as
above. I assume these would count as "special" binaries in the sense
that the binaries would be obtained from upstream and not built by
Debian. I think it would make sense to have both the binaries and the
headers in one binary package.

These "special" binary packages would be a one-time thing to get CCL
into Debian, since once version n-1 of CCL was in the archive, it
could be used to compile version n of CCL without requiring anything
more than the source.

See comments from debian-mentors below.

###################################################################
quote from  "The Fungi" on debian-mentors
###################################################################
Having not done this myself, I've seen it discussed ad nauseum on
debian-devel over the years. First you have to work out how to
bootstrap the compiler on each arch you're targeting. Most compilers
have a minimal bootstrapping method documented using another compiler
(C or in CCL's case perhaps another CL compiler). As a non-DD you'd
likely need to work closely with someone with access to porter boxes
for the various architectures, even if only to help test cross-built
binary versions of CCL (hopefully it comes with a good set of
automated validation tests?). Testing is essential for something like
a compiler, because you want to have some confirmation your package
build sequence works everywhere and won't be as likely to annoy the
buildd admins or porters with a bunch of arch-specific FTBFS problems.

Once you manually work out how to get your CCL packages building
reliably, you'd coordinate closely with a sponsor (maybe someone from
the Debian Common Lisp Team?) to get temporary/throwaway binary
packages uploaded for each arch so that they can be subsequently
pulled in on the buildds as a build-depends of your proper
source-built packages.
###################################################################

> I admit that we are abusing svn externals, but having a one-step
> way to install CCL seems to me to be very important, and I'm
> willing to complicate the life of CCL hackers a bit in order to
> make that one-step installation work.

Personally I prefer (as do many other people) distributed version
control systems. I use mercurial myself.

The next thing I'll do is put my packaging of CCL somewhere
public. This will not be the version that is used, since is uses the
release default version that contains binaries, but there are a bunch
of minor issues which are unrelated to the bootstrapping issue that I
might as well deal with now. I don't know if you use Debian, but I'd
appreciate testing and/or comments.

                                                      Regards, Faheem