[Openmcl-devel] Allocate heap and call C question

[Gary Byers]

On Mon, 5 Jul 2004, Andrew P. Lentvorski, Jr. wrote:

> I have been trying to allocate a chunk of memory from the heap rather
> than the stack (I would eventually like to move a vector around of
> reasonable size) and pass it back and forth to C.
>
> What I would like to do is:
>
> CL-USER> (setq a (make-array 4 :element-type :signed-long))
> #(0 0 0 0)
> CL-USER> (setq amp (unknown-function-to-create-a-macptr-to-memory a))


There is (intentionally) no such function: MACPTRs don't point to lisp
memory.  Lisp objects can and do move around (as a result of GC
activity) at any time.

If you poke around (as you have), you might encounter the function
CCL::%ADDRESS-OF, which returns the address of a heap-allocated lisp
object as an unsigned integer.  As soon as that address has been
obtained, it might become invalid (which means that the value is
useless for any purpose other than associating a more-or-less random
but generally unique number with the printed representation of an object.)

Since there's no reliable way to pass a lisp object to foreign code,
you generally have to do something like what you're trying to do by
allocating a foreign array, copying the lisp array's contents to that
foreign array, passing the foreign array to foreign code, and possibly
copying the foreign array's elements back to the lisp array.  (The
first few of these steps is basically what things like WITH-CSTRS do.)

People have sometimes objected to this, on the grounds that that
copying can be expensive when the arrays are large.  I'm very
reluctant to expose any "reliable" general means of passing a lisp
object directly to foreign code, since I see the things that can go
wrong when foreign code points at relocatable lisp objects as being
(a) often subtle and (b) bad.

Both sets of concerns are justified. A possible compromise might be
something like::

(with-pointers-to-lisp-objects ((ptr-var lisp-object)*) &body body)

where the body is executed in an environment where each PTR-VAR is
bound to a MACPTR that (magically) points to the corresponding lisp
object (an array, most likely).

The constraints would be

 a) those pointers would only be valid during the extent of the body
    (the results would be undefined if any lisp or foreign code retained
    those pointers in any way.)
 b) the GC would be disabled trhoughout the whole construct (execution
    of the body and of all of the lisp-object initialization forms.)

A typical use might be something like:

(with-pointers-to-lisp-objects ((p big-array-of-floats))
  (external-call "_fft" :address p)))

Note that the GC would be disabled for all threads if any thread was
inside a WITH-POINTERS-TO-LISP-OBJECTS, and this can be undesirable.
(Whether it is undesirable or not depends on the application, and
the developer of that application should have enough rope to hang
themselves ...)


FWIW, in the code in your message you were calling %GET-UNSIGNED-LONG
(and SETF thereof) on successive byte indices; you'd typically want
to copy 32-byte words that are 4 bytes apart:

(defun copy-unsigned-byte-32-vector-to-pointer (vector pointer)
  "Assumes that pointer is big enough ..."
  (declare (type (simple-array (unsigned-byte 32) (*)) vector))
  (dotimes (i (length vector))
    (setf (%get-unsigned-long pointer (* i 4)) (aref vector i))))