[Openmcl-devel] Porting

[Gary Byers]

When he was working on the x8632 port, Matt wrote a wiki page (on the
trac wiki) that described the process.  I took a quick look for it
and couldn't find it, but I'm sure that it's still there and someone
may remember the URL ...

It's probably true that >99% of the lisp code in (for instance) an
x8664 FreeBSD image is identical to the code in an X8664 Linux image;
the other <1% - things that depend on C library and OS differences -
generally has to be right or things will fail (sometimes
spectacularly, sometimes subtly.)  We treat all code in FASL files and
heap images as if it was platform-specific, even though it's likely
that only a very small percentage of it is.

A "platform" (combination of architecture, word size, and OS) can be
identified by a small integer where those 3 pieces of information are
encoded in bitfields within the integer.  (At least one of those
fields is getting kind of small.)  Constants that define the values
used in these fields are defined in "ccl:compiler;backend.lisp" and
in "ccl:lisp-kernel;lisp.h".  One of the first things that you'd want
to do is to define PLATFORM_OS_NETBSD/PLATFORM-OS-NETBSD consistently
and uniquely in these two files.

What you're generally going to need to do is to cross-compile a set
of NetBSD fasl files and a NetBSD bootstrapping image from some host
machine on which CCL already runs.  Once you've done that, you copy
everything over to the target system, tell the NetBSD kernel to
start with the bootstrapping image, watch it load a few dozen FASL
files, save a full heap image, then sit back and congratulate yourself.
(This may take several iterations, and it's sometimes more practical
to NFS-mount the host's CCL directory on the target to avoid having
to copy things around as often.)  In theory, it should be possible
to use any platform on which CCL already runs as the host; in practice,
it may work better if the host and target share the same word size
and endianness.

Another of the first things that you'd need to do is to generate a
set of interfaces (.cdb files) for the target platform.  It's -probably-
very easy to port the interface translator: you can use the trunk version
(as described at <http://trac.clozure.com/ccl/wiki/BuildFFIGEN>).  You
probably need to add a clause to the Makefile and may need to add a couple
of files to the src directory; those files would probably be functonally
identical to the files that exist for other platforms.)  Once you get
this working, create a "populate.sh" script for NetBSD and use it to
create a tree full of .ffi files.  (You may find that a lot of the things
in "/usr/include/**/*.h" depend on a lot of ... unspecified ... other
things;  some .h files implicitly depend on <sys/types.h> and in other
cases it's not at all clear what the dependencies are.  Wrestle with
this until you're convinced that the translator can process a large
subset of the headers - large enough to define the fairly vanilla things
that CCL uses.  You might have to comment out (e.g.) Kerberos support
if you can't figure out what its headers depend on; it wouldn't be the
first time that'd happened.)

When changing code in CCL itself, you may find it convenient to a) be
in the CCL package b) disable the CERRORs that try to prevent casual
redefinition of builtin functions. Calling
(SET-DEVELOPMENT-ENVIRONMENT) will do these things.  I do this in my
init file and I don't like or use SLIME; I don't honestly know how
well SLIME deals with this or whether there are any issues at all.

A lot of platform-specific attributes (by no means all) are
encapsulated in a structure called a BACKEND.  The existing BACKENDs
for x8664 platforms are defined in
"ccl:compiler;X86;X8664;x8664-backend.lisp".  The backend definitions
in that file are conditionalized so that only the native backend is
ordinarily defined; you probably want to add similar conditionalization
when editing that file, but you also want to mouse on the DEFSTRUCT 
(e.g., select it and do C-M-x or equivalent in Emacs) while running
the host.  You want to ensure that the new backend is defined in the
host and that it's on *KNOWN-BACKENDS* and *KNOWN-X8664-BACKENDS*.
Once it is, call

? (ccl::fixup-x8664-backend)

just in case that function actually does something useful.

One attribute of a BACKEND is its "foreign type data" (FTD), which
basically describes how the FFI interacts with the backend.  There's
a large CASE form in the function SETUP-X8664-FTD; add a clause
describing your backend's FTD and call CCL::SETUP-X8664-FTD on the
new backend.

All x8664 UNIX systems that I'm aware of use the same ABI (Win64 is
very different.)  There are several files in "ccl:lib;" whose names
start with "ffi-" and that describe some of the details of how foreign
function calls and callbacks work; the files for x8664 unices are
functionally identical but define interfaces in terms of the BACKEND's
FTD's interface-package name.  Make a copy of one of these files, save
the copy in "ccl:lib;", change the package appropriately, and load the
copy on the host.

Fetch the .ffi files generated on the NetBSD system and install them
in the "cross-" headers directory for the target on the host, then do:

? (require "PARSE-FFI")
...
? (ccl::parse-standard-ffi-files :libc :netbsdx8664) ; the second arg
                                                      ; is the name of the BACKEND
....
? (ccl::parse-standard-ffi-files :libc :netbsdx8664) ; parse again to resolve
                                                      ; forward-referenced constants.

This should give you a set of .cdb files that describes NetBSD's C library
functions and data structures that can be used on the host.

(If you're still following this, you're almost at the point where you'll get
some positive feedback for all of this tedium.  Almost.  Some.  No sense in
giving up now ...)

Edit "ccl:xdump;xx8664-fasload.lisp"; note that it defines some BACKEND-XLOAD-INFO
structures for various X8664 platforms.  Define a new one for NetBSD; it's likely
to be identical to most of the others except for the name of the bootstrapping
image and the file type of the startup file.  The IMAGE-BASE-ADDRESS field of
this structure is the address into which the kernel will try to map the image;
most/all x8864 platforms can use #x300000000000, but that's ultimately OS-dependent.
It should match the IMAGE_BASE_ADDRESS defined in the kernel "platform" file
for netbsd/amd64.

One more thing: in "ccl:compiler;nx1.lisp", there's an ECASE in the
definition DEFNX1 NX1-FF-CALL that affects how the compiler frontend
deals with foreign-function calls.  NetBSD (I'm almost certain) uses the same
ABI as the other x8664 Unices, and the ECASE needs to be updated.

Edit "ccl:lib;compile-ccl.lisp".  A few functions there have CASE or ECASE
forms that define which sets of platform-specific files are used on the target
platform.  Extend those [E]CASE forms to deal with NetBSD.  (One of the cases
involves the "ffi-" file you created earlier.

Edit "ccl:lib;systems.lisp"; add a definition for that ffi- file.

You may find this to be a convenient time to do:

? (rebuild-ccl)

on the host.  The new image will include the backend definition and incorporate
the changes you've made to the sources.

In the new host image (setup for cross-compilation), ensure that the NetBSD-specific
FFI file is loaded.  Then do:

? (ccl::cross-compile-ccl target t) ; where target is the name of the netbsd backend

That probably won't run to completion the first time; some source files (especially
things that contain a lot of FFI stuff) will need to be conditionalized in order
to even compile cleanly; most sources contain pretty vanilla CL code and should
cross-compile without incident. You may want to use restarts to skip over some
of the problem files; you can call CCL::CROSS-COMPILE-CCL without providing a
second arg of T to only try to recompile things that were skipped on a previous
attempt.  It may take several iterations, but you want to reach the point where
all files can be cross-compiled.

Among other things:

- "ccl:lib;foreign-types.lisp" needs to create/initialize the native FTD
- "ccl:level-1;l1-boot-2.lisp" needs to load the platform-specific "ffi-" file.
- some callbacks define in "ccl:level-1;x86-trap-support.lisp" need to be
   able to access fields in the ucontext and mcontext structures used to
   represent signal contexts.  No two OSes (so far) provide equivalent
   definitions of these structures; FreeBSD and NetBSD may or may not
   provide very similar definitions.

Once at that point, you can do:

? (ccl::require-modules ccl::*x8664-xload-modules* t)

? (ccl::cross-xload-level-0 target :force) ; where target is the name of the
                                            ; "xload backend" for NetBSD defined
                                            ; earlier.  It's often the same
                                            ; name as the BACKEND structure.

At some point, the definition of *features* (in "ccl:level-0;l0-init.lisp")
needs to ensure that target-specific featues that describe the NetBSD platform
when that's also the host platform.

At least in theory, most of the code that's compiled here isn't very
OS-dependent.  The stuff that deals with shared libraries (in
"ccl:level-0;l0-cfm-support.lisp") is an exception.  NetBSD uses ELF
object files and ELF dynamic-linking technology, but there are some
platform-specific differences in how that technology is implemented.
(It's mostly "Linux" and "platforms that try harder to be standards-
compliant" so far.)  Opening a shared library returns a "handle"; some
of the code here deals with ELF "link_map" structures.  On Linux a
handle and a link map are the same thing; on other platforms, there
are platform-specific ways to map a handle to the associated link_map
or vice versa.

At some point, you'll have an alleged NetBSD bootstrapping image and
a set of (alleged) NetBSD FASL files, and you'll be able to see if
you can run the image and have it load the FASLs and save a full image.
You'll get to see if the kernel actually works: can it map the image
into memory ?  Can it handle exceptions generated by lisp code ?  (There'll
be a lot of those, and there are OS-dependencies involved in handling them.)
If you aren't comfortable debugging low-level code in GDB and want to pursue
this, you'll probably get more comfortable/facile with that kind of debugging
pretty quickly.  (Note that it's almost certainly necessary to use a 
.gdbinit file that tells GDB what signals to ignore.  See the .gdbinit
files defined for other platforms.)

At some (somewhat later) point, things'll work: the NetBSD version will
be able to compile itself and CCL on NetBSD should be as functional as
it is on other platforms.

I'm fairly familiar with all of this stuff (I'm disturbingly familiar
with some of it) and I think that it usually takes me about a week to port
CCL to a new Unix variant.  I'm not trying to make this sound any more
daunting than it is, but if you were expecting to be able to run a Linux
image with a NetBSD kernel you've probably been discouraged from that idea.
If you have commercial interest in having CCL ported to NetBSD, Clozure
could be persuaded to take money to do it for you.  It's possible (though
probably not too likely) that we'd just do the port ourselves someday.
(There was slight interest expressed in having CCL running on Solaris;
porting to Solaris exposed some problems that could have affected other
platforms, so the port was worthwhile even though the demand was slight.)

If you're interested in doing this yourself (as an exercise or for other
reasons), I think that all of the above is essentially correct.  I may
be forgetting some details, but it's likely that details that I'm forgetting
are similar to those that I described.  The overall effort involved is
probably similar to what's involved in porting a medium-to-large application
program to another OS.








On Fri, 13 May 2011, rjs at fdy2.demon.co.uk wrote:

>
> Is there any documentation on how best to debug a port to a new platform ?
>
> I'm trying to build CCL for NetBSD/amd64.
>
> I have built the lisp-kernel stuff and tried running it with a Linux
> X86-64 image but it just drops into the debugger.
>
> Robert Swindells
> _______________________________________________
> Openmcl-devel mailing list
> Openmcl-devel at clozure.com
> http://clozure.com/mailman/listinfo/openmcl-devel
>
>