[Openmcl-devel] ClozureCL on iPhone

[Gary Byers]

On Wed, 2 Nov 2011, Andrew Shalit wrote:

> What you're asking for is a cross-compiler and cross-dumper.

A cross-compiler and cross-dumper are basically the tools that we use tp
bootstrap CCL on a new platform.  Here's a little file that turns a CCL
running on an x86 on MacOS (or Linux, or FreeBSD) into a cross-compiler
and cross-dumper that targets ARM Android devices.  (A similar little
script could be used to target "ARM Darwin" - aka iOS -, but I don't
have such a script handy at the moment.)

;;; -start of file-
(in-package "CCL")

(defpackage "ARM-ANDROID" (:use))
(defpackage "ARM-LINUX" (:use))
(defpackage "ARM-DARWIN" (:use))

(defun load-android-backend ()
   (update-modules '(arm-arch arm-asm arm-lap arm-backend arm-vinsns arm2) t)
   (setup-arm-ftd *androidarm-backend*)
   (update-modules '(arm-lapmacros arm-disassemble ffi-androidarm) t)
   (update-modules *arm-xload-modules* t))

(load-android-backend)
;;; -end of file-

If it looks at first glance like there isn't much to that, you're right.
On a second glance, you might ask what these "modules" are, realize that
they're ways of naming source files, and note that there's a lot of ARM-
specific code in some of those files.  That's work that's essentially
done.

Assuming that we actually did want to cross-compile for Android (if only
because I can actually find the file above as opposed to the iOS-specific
version) and assuming that we have a set of Android ARM interface files
in our host system's CCL directory (there's a set in svn and for the time
being we can assume that that's where they come from), we could load the
above file and then do:

? (ccl::cross-compile-ccl :androidarm t)
and watch a few dozen .aafsl files get compiled from CCL sources and a few
warnings (stemming from the fact that some Android-specific function calls
are getting compiled on a non-Android machine) scroll by.

We could then do:

? (ccl::cross-xload-level-0 :androidarm :force)

and that'll compile a few dozen more lisp sources into .aafsl files,
"cross-load" them into the host lisp, and "cross-dump" a heap image
file (named something like "aarm-boot"). Moving more quickly than we
have been so far, we use tools from the Android NDK to cross-compile
the kernel, use the Android Debug Bridge to copy the kernel (aarmcl,
aarmcl.so), bootstrapping image, and fasl hierarchy to some directory
on the device that we can write to (/data/local/ccl seems to work on
my phone), and then use "adb shell" to connect to the device, and then
(actual unretouched transcript):

[~] gb at antinomial> adb shell
* daemon not running. starting it now on port 5037 *
* daemon started successfully *
$ cd /data/local/ccl
$ ./aarmcl -I aarm-boot
Unhandled exception 4 at 0x5002c540, context->regs at #xbe9a47a8
? for help
[10772] Clozure CL kernel debugger:

It seems that some idiot (me) has modified the shared library initialization
code and forgotten to conditionalize the changes for Android.  If that idiot
ever gets around to fixing that and whatever other bitrot has been introduced
in the last year or so, we'd see a bunch of FASL (.aafsl) files get loaded
and then we could do:

? (save-application "aarmcl.image")
$ ./aarmcl
Welcome to whatever version this is!

The point of this lengthy example is to remind me to fix the shared-library
initialization stuff someday, since it's really embarrassing when examples
like this crash.  A secondary point is to try to put some of the discussion
of the last few days in context.

If we take the little script above and make a couple of simple changes to
it (replace ffi-androidarm with ffi-darwinarm, *androidarm-backend* with
- IIRC - *darwinarm-backend*) we can go through essentially the same steps
as are outlined above; when we get to the point where we'd like to build
a kernel for darwinarm, we might (or might not) say "what's wrong with the
idiot who wrote this Makefile ? It seems to assume an older version of Xcode;
don't they realize that every Xcode release moves and renames everything
arbitrarily ?"  Ahem.)

Let's assume that we can get past that.  What do we do next ?

Assuming that we could figure out how to copy the kernel, bootstrapping
image, and FASL hierarchy to an iDevice and that we'd digitally signed
the kernel and (if possible) the heap image, having paid Apple for the
right to do so.  Assume further that we could actually run the kernel
(I'd know how to do this under GDB; there are lots of people at Clozure
that do iOS development and they may know of a better way) and assume
that we can actually run that code; it'll start loading FASL files (assuning
that iOS will let the application touch the filesystem.)

FASL files are basically bytecode streams.  A fairly common bytecode
sequence in a FASL file might say "interpret the next N bytes in the
stream as ARM machine insructions.  Make an object of type FUNCTION
out of those bytes and some associated data, do whatever you need to
do to get the machine's instruction and data caches in synch, and call
that function."  That's pretty basic and very common operation; it's
equivalent to what happens when you load a .lisp source file, only the
compilation to machine code has already happened.  It can't work on
(stock) iOS, because stock iOS doesn't allow memory to be both
writable and executable.

Let's back up a bit.  When we "cross-loaded" a bunch of FASL files and
"cross-dumped" a bootstrapping heap image, what really happened ?
FASL files contain bytecodes that create certain kinds of objects,
call functions, and otherwise have side-effects on the running lisp
image.  Some of those side-effects are simple and predictable: the
bytecode produced for DEFUN will create a FUNCTION object (from
immediate data in the FASL file) and typically cause some SYMBOL to
have that FUNCTION as its global function definition.  Emulating these
side effects (those typically associated with DEFUN) is fairly simple,
and the cross-loader and cross-dumper can build a heap image in memory
on the "host" machine (where that image contains the functions defined
with DEFUN in a set of FASL files) and that's all comparatively simple.
Other operations in FASL files involve creating a (target) function and
funcalling it, and as Bill has pointed out doing that in this context
basically requires that you have the ability to emulate arbitrary ARM
code on the host machine, and that sounds like a fairly extensive 
project.

(If you don't try to emulate arbitrary code on the host, you could
just store that code in the image and arrange to run it when the image
startes.  That's basically what [CROSS-]XLOAD-LEVEL-0 does; of course,
that arbitary code starts loading other FASL files, which try to put
code in memory and execute it.  As we all know, this leads to street
crime and Apple thoughtfully protects us from that.)

One last aside: a C source file contains a sequence of (function,
variable, and type) declarations and function and variable definitions
(and I'm ignoring preprocessor macros and pragmas and other nonsense
and intentionally oversimplifying things here).  A C object file (a .o
file) contains some initialized data (representing the variable definitions)
and some machine code (representing the function definitions), along with
linkage and relocation information.  The machine code may target a machine
with a different architecture than that on which the code was compiled.

A .o file doesn't (generally) contain "arbitrary toplevel forms" ("static
initializers" aren't arbitrary enough to qualify, but nice try!) or the
notion that it's "loaded" into a running environment and modifies that
environment.

Compare what can be expressed in a .o file with (for instance) what a
DEFCLASS macro expands into.  If you haven't thought about this and are
interested enough to still be reading this, please think about it.  Anyone
who says "I don't understand why you don't just cross-compile" is hereby
condemned to write lisp source code that consists of nothing but simple
variable definitions and functions defined by DEFUN until they do understand.
I can't think of a better way to make that point.

Suppose that we developed some sort of emulation technology and that
that was adequate to deal with the issue.  (It wouldn't likely be, and
I'm oversimplifying again) and we could somehow build a working image
enturely on the host machine (a Mac, whatever.)  As Gail pointed out,
you'd be tossing away a lot of the nominal advantages of programming in
lisp (incremental development and interactive debugging) in the first place,
and unless you could do that you might as well be programming in Xcode.
(I don't remember whether Gail characterized things exactly that way, but
I would.  What's the point ?)

The ability to interactively develop and debug by (in some sense/some way)
defining code on the device seems to be incredibly important and I think
that Gail said this very clearly.  Being able to run evaluated code might
satisfy this requirement, but as she noted that involves technology (an
evaluator) that we don't have.  A bytecode compiler and interpreter might
be a more viable solution, but that also involves technology that we don't
have and have never had.

Those solutions might lead somewhere, though we'd still need to address
some of the cross-compilation/side-effect/emulation issues described above.
I've been advocating the idea of assuming that all development was done
natively on jailbroken devices.  There are risks involved in that, including
the risks associated with depending on a group of (benign) hackers known
by their IRC nicknames.  (As a grownup, I admit that I'm somewhat embarrassed
to find myself thinking "gee, I hope that MuscleNerd comes up with an
untethered solution soon.")  Apple may decide to take some sort of real
stand against it (rather than just spreading FUD and doing very little to
actually prevent it.)

Different people may evaluate those risks differently, but I think that
being able to develop natively in full-blown CCL has a lot of advantages
and don't personally assess the risks as being very great.  I think that
this is likely the most expedient way to get anywhere, but other approaches
can eventually be made to work they might be more attractive to people
who assess those risks differently.

Apple has sometims claimed that jailbreaking could lead to a number of
unfortunate consequences.  I may be misremembering the details; I don't
know that they claimed that the earth "would" spin off its axis or merely
noted that it "could" do so.  Likewise, I don't know that they claimed
that jailbreaking would lead to a general decay in moral fiber and an
increase in street crime or claimed that it was a byproduct of that
moral decay; my memory of that's a bit fuzzy.

I'm quite sure, however, that if I see one more message that says "why
can't you just cross compile ..." there will be some acts of senseless
violence.







> When CCL is running on a Macintosh with an Intel processor, the compiler produces code for the Macintosh and Intel processor.  You want the compiler to have a special mode where it can create iOS ARM code even though it is running on a Macintosh.
>
> Furthermore, the process of creating a Lisp program almost always involves the creation of initial datastructures, which then get dumped out.  These initial datastructures are created by running Lisp code.  So you'd want to be able to run Lisp code on the Macintosh (or whatever) that would somehow create data structures suitable for the iOS ARM.  That's just not the way most Lisps (including CCL) work.
>
> This is all technically possible, but it would be a lot of work.
>
>
> On Nov 2, 2011, at 7:59 AM, Lynbech Christian wrote:
>
>> I am getting a little confused.
>>
>> I understand that getting a full-fledged CCL, including compiler and
>> REPL, up and running on an iDevice is pretty difficult.
>>
>> But if one looked at the (presumably) simpler problem of just getting a
>> standlone application transferred to the device, ie one that was
>> developed and dumped with CCL but without any dynamic development
>> environment, would we still need heavy handed extensions of CCL (like
>> compilling to C)?
>>
>>
>> ------------------------+-----------------------------------------------------
>> Christian Lynbech       | christian #\@ defun #\. dk
>> ------------------------+-----------------------------------------------------
>> Hit the philistines three times over the head with the Elisp reference manual.
>>                                        - petonic at hal.com (Michael A. Petonic)
>
> _______________________________________________
> Openmcl-devel mailing list
> Openmcl-devel at clozure.com
> http://clozure.com/mailman/listinfo/openmcl-devel
>
>