[Openmcl-devel] sharing data between fortran and OpenMCL

[Gary Byers]

[Cyrus posted this reply from another email account.  I happened to
notice this, went to the mailing list administration page - which I
don't do as often as I should - to "approve" the message, and spent
the next half hour or so watching a beachball cursor as Safari and
Apache and Mailman struggled to present a few hundred spam messages
with an occasional on-topic message thrown in.

I'm not sure if Cyrus' message ever made it to the list; I was CC'ed
separately.

If anyone wonders "why do other lisps allow you to pass the addresses
of Lisp objects to foreign code, while OpenMCL doesn't ?", I'll try
to explain that briefly without noirish analogies.

At one point in MCL's history, there was a mechanism for doing this;
the rules for using the mechanism were basically "you can do this
safely, as long as you're sure that the foreign code you're calling
won't call back to lisp and trigger a GC, which might render that
address invalid."  That was a fairly easy rule to follow, because
not too many C or Fortran functions that people wanted to pass large
lisp arrays to were written to accept lisp callback arguments.

I would guess that implementations that still offer this functionality
are offering it under similar rules, whether those rules are
explicitly stated or not.

It's possible to break those rules and live to tell about it: the
only thing that's really fatal is the case where the foreign code
calls back into lisp AND the lisp triggers a GC AND the GC moves
the object in question.  The odds of all of these things occurring
at the same time are usually very low, unless you're trying to
do a demo for an important customer/project sponsor, etc.

There are sometimes additional platform-specific considerations.  It's
very hard on x86 hardware to do a fully relocating GC; such a GC
usually depends on the lisp following conventions so that the GC can
tell which registers definitely have lisp objects in them and which
registers definitely don't.  The x86 archtitecture doesn't offer many
registers and doesn't make such register-partitioning schemes easy,
so many/most GCs on x86 implementations adopt a hybrid strategy:
they can relocate things that are referenced only from the heap
reliably ("those roots are not ambiguous", in GC-speak), and have
to avoid relocating objects that appear to be referenced from the
stack/registers.  (I know that CMUCL and SBCL adopt this partially
conservative GC strategy on x86; I'm not sure about commercial
implementations.)

Since the arguments to the foreign function call are on the stack/
in registers throughout the lifetime of the call, those objects are
protected from relocation (by some combination of accident, design,
and concessions to a register-starved but commercially popular
architecture.)

So, the amended rules for passing pointers to lisp objects to foreign
code are:

 a) don't do anything that could trigger a GC between the time that
    that pointer is obtained and the time that the ff-call returns
 b) if you can't really guarantee that, do this on an x86 with a
    partially conservative garbage collector, where most of the things
    that you'd try to do would work because the GC can't really
    be too sure of what you're doing anyway, kinda.

A preemptively scheduled lisp makes (a) harder to guarantee; the
behavior of the calling thread might be predictable, but the
behavior of other threads in the system might not be (OpenMCL's
initial housekeeping thread might wake up from a nap, look at
the time, and try to cons a bignum ...).  If you think that lisps
should use partially conservative GCs on platforms where they aren't
practically forced to, I think that the short answer is "no" and
the long, explicit answer is "no!".

Since OpenMCL is preemptively scheduled and is not forced to use a
partially conservative GC strategy, the only thing that that could
work reliably is something like:

(without-gcing
  (#_operate_on_big_array (%pointer-from-lisp-array *the-big-array*)))

I'd be willing to implement this when I get a little time; I'm
very strongly tempted to say that %POINTER-FROM-LISP-ARRAY should
signal an error if the GC's enabled, or should  otherwise be closely
coupled with the act of disabling the gc:

(with-pointer-from-lisp-array (ptr *the-big-array*) ; implicitly disables GC
  (#_operate_on_big_array ptr)
  ) ; illegal to reference ptr after this point



On Fri, 1 Oct 2004, Cyrus Harmon wrote:

>
> Sure, but in comparison with C memory management, the garbage
> collection really means two things, one obvious, one less so. The first
> is that memory just appears and disappears when you need it and no
> longer need it. That's really nice. The second is that objects in
> memory can move around on you to make the whole system work better.
> That's also nice, but poses some issues when dealing with non-lisp
> components, be they system resources, other code libraries, etc...
>
> Getting even further off-topic, one of the things that bothers me about
> lisp is the notion that the lisp way is the right way and everything
> else doesn't matter. One of the things that I _really_ like about
> OpenMCL is the interface database and the relative ease with which one
> can interface with foreign code. It's a fact of life that we live in a
> heterogeneous world. Back to my particular corner of the world. There's
> some really nice fortran code for dealing with big matrices out there.
> I don't want to (and probably can't) rewrite this stuff in lisp. On the
> other hand, I don't want to write my systems code, web server, db
> access, etc... in fortran. I want to use lisp. OpenMCL seems to be
> pretty useful here and I'm just trying to sort out all of the details
> in how one would use this stuff in a heterogeneous environment. In
> SBCL, and, I think, ACL, there are some tricks one can do to make it so
> one can avoid round-trip copies of the data as you move between lisp
> and the fortran code. It would be nice if I could do the same thing in
> OpenMCL and I think I can, I'm just trying to sort through all of the
> issues and trying to figure out the best way to proceed here.
>
> Also, you shouldn't be so dismissive of a trumped-up portable assembly
> language. There's lots of really nice trumped-up (not-so) portable
> assembly language at the core of OpenMCL I'm discovering. One of the
> interesting things (to me) about lisp is that you can basically write
> lispy lisp, c-like lisp, or even assembly language in the same overall
> framework. This is pretty helpful if you need to interact with the
> c-world or the assembly-world, I think.
>
> Cyrus
>