[Openmcl-devel] Using Cocoa with MCL

[Gary Byers]

On Sat, 10 May 2003, Yannick Versley wrote:

> > There might be some way to do what #_setjmp does (establish something
> > like a CATCH frame that #_longjmp can do something like THROW to).  It
> > probably won't ever "work" to use #_setjmp from lisp (as if #_setjmp
> > was just a normal function call and not magic),
> Well, you *can* just call _setjmp like a normal function. It then sets up
> a jump buffer to the FFI entry/exit routine, and, ideally, an error would
> land you not at the point where you called _setjmp, but somewhere in
> Lisp-land where you called the function that threw the exception, and
> you can (almost) easily check for exceptions via some unspectacular C
> functions. It's a hack and it seems to sometimes collide with either the
> Lisp scheduler or the GC.

All of the complexity (and, unfortunately, there's a lot of it) on FFI
entry and exit is there to keep the GC happy.  (We don't want to make
the GC unhappy).  It is happy if it can find all pointers to lisp objects
(including pointers into the middle of lisp code vectors, i.e., return
addresses.)  Compiled lisp code follows strict conventions (register use,
etc) that enable the GC to reliably find all pointers to lisp objects;
C code doesn't. _setjmp basically just captures its caller's registers
(and some other state); we don't want (the GC doesn't want ...) foreign
data structures (jmp_buf's) to contain pointers to relocatable lisp
objects.

Fortunately, they don't: all FF-CALLs sort of "bounce off a trampoline":
the lisp code calls some assembly-language code in the lisp kernel.
Several dozen very careful instructions later, all registers that could
point to lisp objects are saved where the GC can find them and the
kernel routine calls the foreign code.  In 0.14, some other thread might
be consing away while the current thread is in the midst of all of this;
that has to work reliably (and as far as I know, it does.)

If we do (#_setjmp buf) from lisp, "buf" winds up getting filled with
the (mostly random) register values in effect when the trampoline called
it.  Two registers that aren't random are the register that contains
the return address (points right after the "call" [bctrl] instruction
in the trampoline) and the stack pointer (points to a frame that contains
the foreign function's outgoing arguments, with a lisp frame that tells
the trampoline how to return to lisp code underneath it.).  The trampoline
function pops both of these off of the stack, so the value of "SP" in
the jmp_buf is already pointing to garbage (a deallocated stack frame.)

If we then do (#_longjmp buf val) in the same stack context in which
we did the original (#_setjmp), the trampoline will make the ffcall
to _longjmp with (coincidentally) SP pointing to the same place as
it pointed when _setjmp was called.  (More accurately, the value of
SP saved in the jmp_buf is -coincidentally- valid again.)  _longjmp
will restore all registers saved by _setjmp and return to _setjmp's
caller, which happens to be exactly the same place (the trampoline)
as _longjmp's caller.  The trampoline pops the top frame off of
the stack (fortunately, SP is actually pointing to a stack frame)
and returns to the lisp frame underneath it, which happens to be
#_longjmp's caller.  That's not what we expect, and it's blind
luck (an artifact of having made the #_setjmp and #_longjmp calls
in the same stack context) that it wasn't catastrophic.

Moral: it doesn't seem possible to use #_setjmp at all reliably.
Fortunately, I don't think that we need to call #_setjmp from lisp
in order for lisp to particpate in ObjC's exception handling system.

>
> > but it seems like it
> > may be possible to do something like:
> >
> > (rlet ((buf :jmp_buf))
> >   (catch :some-tag
> >    (%%associate-jmp-buf-with-current-catch-frame buf)
> >    (progn
> >      (whatever)))) ; that association needs to be broken; perhaps
> >                      this'd work better if it was nested differently
> >
> > If a jmp_buf was "associated" with a catch frame (and this all worked),
> > then #_longjmp to that jmp_buf would behave exactly like a THROW to
> > that frame's tag.  There are lots of gory details and complications,
> > but this might (with suitable macrology to hide those details) make
> > it possible to write ObjC-visible exception handlers in lisp.

> This would mean recreating almost all of the black magic that
> currently works in the FFI entry/exit point. I looked at it, understood
> some of it and concluded that it's not a feasible thing for people like me
> who are happy to understand half of the weird things going on at assembly
> level.

I'm hoping to be able to recreate the black magic that happens in
#_setjmp, and leverage existing black magic in other cases.

Whenever lisp code does CATCH, a "catch frame" is pushed on a stack.
The frame is conceptually something like a jmp_buf, only it's a lisp
data structure.  A thread's catch frames are linked together; a
frame contains a "tag" field (an arbitrary lisp object used to
identify it) and the values of interesting registers at the time
it was created.  THROW does a linear search of a thread's catch
frames and sort of "returns to" the first one it finds whose tag
matches.

Since catch frames are allocated on a stack, it's safe to pass
their addresses to foreign code.  (Since they have dynamic extent,
it wouldn't be safe for that foreign code to hang on to those
addresses too long, but it'd be OK to keep them in some foreign
data structure whose extent is similar.)

Since a catch frame has a fixed address, we can represent it as
a MACPTR (details elided).  If we're willing to ignore a few
more details, we can write a lisp function:

(defun throw-to-catch-frame (value catch-frame-address)
  (let* ((tag (dig-catch-tag-out-of-frame catch-frame-address)))
    (throw tag value)))

Or, better yet:

(defcallback throw-to-catch-frame (:signed-fullword value
                                   :address catch-frame-address)
  (let* ((tag (dig-catch-tag-out-of-frame catch-frame-address)))
    (throw tag value)))

Now, if we have a way to pass that callback and a frame address
to foreign code, that foreign code can effectively transfer control
to a lisp CATCH construct; all of the black magic so far is just
the regular catch/throw stuff and the existing FFI.

In the context of ObjC exception handling, what we'd sort of like
to be able to do is to register an ObjC handler and associated
jmp_buf such that when something #_longjmp's to that jmp_buf
control is transferred to lisp in some known, well-defined way;
we want that jmp_buf to look like it returns to the THROW-TO-CATCH-FRAME
callback that we just defined.

We've seen (in painful detail) that it doesn't work to use #_setjmp
to initialize a jmp_buf, so I guess that we'll just have to initialize
it ourselves.  We don't really care too much about what goes in the
saved register values in that jmp_buf; all that really matters is
that SP is valid, that the return address points to THROW-TO-CATCH-FRAME
or a little bit of glue, and that we can get R4 register (in which
C passes a function's second argument) pointing to the right catch
frame.  (I looked: R4 isn't saved in a jmp_buf, so we need an extra
instruction or two of glue.)

(Using #_setjmp won't work, but Treachery and Deceit should work just
fine ...)


>
> > The next nasty surprise awaiting us is the observation that merely
> > transferring control to the right place (via #_longjmp) isn't
> > adequate: THROW "unwinds the stack" on the way to its target,
> > discarding intervening CATCH frames and executing intervening
> > UNWIND-PROTECT cleanup forms.  If there's a reasonably sane way to
> > patch #_longjmp (kind of sounds like an oxymoron already) so that it
> > does the right thing, it seems like it'd be worth the effort to do
> > so.
> I was thinking of simply wrapping the lisp function that invokes an ObjC
> method (and is called by all ObjC method invocations) with the
> _setjmp/error checking mechanism and rethrowing occuring exceptions on the
> Lisp side. The _setjmp business would be concentrated in _one_ location
> and there would not be any need for much macrology or patching the
> longjmp C routine.

This sounds like the right idea: at any point where we switch between
lisp and ObjC code, there's a barrier that keeps exceptions from
being blindly thrown too far.  Having these barriers at the points
where Lisp<->C transitions occur might help ensure that any necessary
black magic happens correctly.  I have a sneaking suspicion that
really doing this right is a little harder than it may sound at first
glance, but I also suspect that it's doable and would be of great
value.

(Ask me again after I've spent a few days in GDB ...).

>
> > The ObjC runtime maintains a per-thread LIFO list of exception
> > handlers; THROW is ordinarily blissfully unaware of this.  I'm not
> > sure how to effect this, but throwing "past" an ObjC exception
> > handler should ideally remove that handler from the per-thread
> > list.
> If we wrap Lisp implementations of ObjC methods with a wrapper that
> handles Lisp exceptions and rethrows them as ObjC exceptions (in some sane
> way), we wouldn't have to think about unwinding ObjC handlers, but we
> would still have the same problems as the Java/ObjC wrapper currently has
> (means: Lisp exceptions thrown through ObjC code get horribly munged).

ObjC can't really get its evil, GC-unaware hands on a lisp condition
object, but it might be able to pass some sort of identifier ("handle")
around.

>
> Possibly, all this would be easier if we had some kind of native interface
> like JNI in Java that allows C code to interact with Lisp data and code in
> a clean and easy way.

If there were an LNI for OpenMCL, it'd mostly consist of C functions
that tell you that you can't do what you're trying to do.

OpenMCL's runtime system (with the exception of the GC itself and
some exception-handling code) isn't written in C and isn't too
easily accessible from C.

> And, yes, a pony for me too ;-)

Some of this stuff is complicated, but I'm fairly sure that the pony
won't be necessary.

>
> Regards,
> Yannick
>
>

_______________________________________________
Openmcl-devel mailing list
Openmcl-devel at clozure.com
http://clozure.com/cgi-bin/mailman/listinfo/openmcl-devel