[Openmcl-devel] FFI question: passing aligned pointersfa

[Gary Byers]

On Tue, 7 Dec 2010, Artem Mironov wrote:

> Hello Mattew,
>
> On Mon, Dec 6, 2010 at 10:23 PM, R. Matthew Emerson <rme at clozure.com> wrote:
>>
>> On Dec 3, 2010, at 5:14 AM, Artem Mironov wrote:
>>
>>> I'm new to CCL and to Lisp in general. I have a question about passing
>>> C pointers to/from C functions using (external-call ...).
>>>
>>> A C pointers returned by system's malloc() are aligned by at least 8 bytes.
>>> Such pointers can be safely passed to/from lisp as fixnums, without boxing
>>> into macptr.
>>>
>>> I'm looking for something like this:
>>>
>>> (external-call "c_function" :fixnum 0 :int 7 :fixnum)
>>>
>>> Is there such feature or where to look to try to implement it?
>>
>> Are you worried about consing a lot of macptrs? ?In many cases, macptrs can be stack-allocated.
>>
>> http://ccl.clozure.com/ccl-documentation.html#Referencing-and-Using-Foreign-Memory-Addresses
>>
>
> I use code like following
>

And in this code as it's written, the compiler has no way of knowing that
the function IOQ-NEW will always return a pointer, and the function's
pretty much obligated to always return a boxed object.


> (defun ioq-new (flags) (external-call "ioqnew" :int flags :address)

What if this was written as a macro instead ?

(defmacro ioq-new (flags) `(external-call "ioqnew" :int ,flags :address))

(In many cases, you could get a similar effect by declaiming IOQ-NEW
to be INLINE before defining it.  In this case, the macroexpansion
does a better job of avoiding boxing an address just so that it can
be unboxed and stored in a stack-consed MACPTR object.)

If you don't somehow make the pointer-creating operation (the EXTERNAL-CALL)
visible to the compiler, there isn't much that it can do to avoid the
boxing (that's necessary in general) in some specific case.



> My application performs huge amount of IO (tens of Mb per second).
> This particular case isn't problem, but inside (loop) of example above,
> a lot more functions with take/returns :address are called.
>
> All pointers used in this example are page aligned.

And they have well-defined extents (lifetimes), and are therefore good
candidates for stack-allocation.

>
> The other issue that my application passed a lot of (signed 64) values.
> This file offsets all of them are 4096 bytes aligned too.
>
> Also, as I mentioned above, majority of foreign pointers (ones returned by
> system's malloc()) are at least 8 byte aligned and some implementations
> (like FreeBSD) are 16 byte aligned.
>
> So passing around such pointers are fixnums seems to me quite natural.

Maintaining code that uses this idiom can be quite unnatural.

If all addresses were word-aligned and if there wasn't some native pointer
type, then things would probably be more tractable than I'm claiming them
to actually be.  If you have to deal with a mixture of aligned and non-aligned
addresses, then the resulting code is often much more difficult to understand
and maintain.  (Some people may disagree with this claim.  Everyone's entitled
to their opinion, no matter how obviously wrong it may be ...)  If the motivation
for representing addresses as FIXNUMs is to avoid consing - and if there are other
ways of doing that - then it really seems preferable to avoid the idiom.



>
> I'm slowly getting around CCL sources and gathering information in order
> to implement such kind of external-call.

If you're really convinced that you want to do this, I'd suggest doing it
at a slightly higher level and (among other things) making the fixnum <->
pointer coercion explicit.

One way to coerce an aligned address into a fixnum with the same bits
is to store the address in a memory word and reference the lisp object
at that word.  The internal function (CCL::%GET-OBJECT ptr offset)
does the latter; it can be extremely unsafe (so we'll check the low bits
of the word at PTR+OFFSET before calling it.)  (CCL::%SET-OBJECT ptr offset value)
does the inverse, and in our use of these things here "offset" is always 0.



(defmacro aligned-ptr-to-fixnum (ptr)
   (let* ((p (gensym)))
     `(ccl:%stack-block ((,p target::node-size)) ; allocate room for a native word
        (setf (ccl:%get-ptr ,p) ,ptr)
        ;; Be paranoid.
        (unless (zerop (logand (ccl::%get-unsigned-byte ,p
                                 #+big-endian-target (1- target::node-size)
                                 #+little-endian-target 0)
                               target::fixnummask))
          (error "Pointer not word-aligned."))   ; don't evaluate PTR twice here
        (ccl::%get-object ,p 0))))

(defmacro with-aligned-ptr-for-fixnum ((ptr f ) &body body)
   "Execute BODY with PTR effectively bound to an aligned pointer
    equivalent to the fixnum F."
   (let* ((p (gensym)))
     `(ccl:%stack-block ((,p target::node-size))
        (ccl::%set-object ,p 0 ,f)
        (symbol-macrolet ((,ptr (ccl::%get-ptr ,p)))
          , at body))))


(defun malloc-fixnum (size)
   "Call (#_malloc size) and return a fixnum representation of the result."
   (aligned-ptr-to-fixnum (#_malloc size)))

(defun free-fixnum (f)
   "Treat the fixnum F as an aligned pointer and free it."
   (with-aligned-ptr-for-fixnum (p f)
     (#_free p)))

>
> But %ff-call is very integrated into CCL system. It's not clear where to add
> new code.

I really think that it's appropriate to do this at about the level that it's
done above, where it's clear that you're doing a kind of coercion between
two disjoint types of things.