[Openmcl-devel] getting CCL ready for CS graphics courses
Gary Byers
gb at clozure.com
Thu Jan 8 15:59:13 PST 2009
#_ basically hashes the function name which follows and tries to find
an entry for that name in some "functions.cdb" file (where an "entry"
is a sort of byte-coded representation of the function's argument and
return types.) #_ parses the encoded argument into a little lisp
structure and adds that structure as the value of the key
OS::|caseSensitiveName| in a hash table. The final read-time actions
are to define OS::|caseSensitiveName| as a macro (all of the things
defined via #_ share the same macro definition, named something like
CCL::%EXTERNAL-CALL-EXPANDER) and return OS::|caseSensitiveName|.
So, reading:
(#_read fd buf n)
tries to ensure that some foreign function definition info is associated
with OS::|read| and that OS::|read| is defined as the magic macro (when
processing the #_) and the reader returns:
(os::read fd buf n)
When macroexpanded, that turns into something like:
(external-call "read" :int fd :address buf :some-kind-of-int n :some-kind-of-int)
which eventually macroexpands down to %FF-CALL.
What you presumably want to do is something similar. only avoiding
the use of #_ and defining (possibly inlined) functions or macros
with lispier names (gl-color-3f rather than |glColor3f|, etc.)
It's possible to enumerate all of the keys in a .cdb file - I think that
the function/macro that does this is called CCL::CDB-ENUMERATE-KEYS, of
all things; this returns a list of all strings used as keys in the .cdb
file (in random order). If you do:
(ccl::cdb-enumerate-keys (ccl::db-functions (use-interface-dir :gl)))
you'll get a list of > 7000 strings; that includes everything that's
defined in the :gl functions.db file.
Unfortunately, that list will include the names of things that are
referenced in some GL-related header file (including a lot of Carbon
and C library stuff defined in agl.h). If a heuristic is that all
real GL function names start with the string "gl" (and few non-GL
function names do), you can pass a second "filter/predicate" argument
to CDB-ENUMERATE-KEYS that decides whether or not to include a given
string in the list. (This may or may not be easier than generating
the full list and post-processing it.)
However we do it, if we restrict the set of defined keys to names
that start with "gl", we get a list with ~1136 entries.
The enclosed function OBTAIN-GL-FUNCTION-INFO will return a hash
table whose keys are GL function names (mixed-case strings) and
whose values are structures (CCL::EXTERNAL-FUNCTION-DEFINITIONs);
accessors with the prefix (defstruct :conc-name) CCL::EFD- can
be used to access the fields of an EXTERNAL-FUNCTION-DEFINITION
structure. If we look at one of these (chosen at random based
on my fading memory of OpenGL):
? (defvar *gl-info* (obtain-gl-function-info))
? (describe (gethash "glColor3f" *gl-info*))
#S(CCL::EXTERNAL-FUNCTION-DEFINITION :ENTRY-NAME "glColor3f"
:ARG-SPECS (:<GL>FLOAT
:<GL>FLOAT
:<GL>FLOAT)
:RESULT-SPEC :VOID :MIN-ARGS 3)
Type: CCL::EXTERNAL-FUNCTION-DEFINITION
Class: #<STRUCTURE-CLASS CCL::EXTERNAL-FUNCTION-DEFINITION>
ENTRY-NAME: "glColor3f"
ARG-SPECS: (:<GL>FLOAT :<GL>FLOAT :<GL>FLOAT)
RESULT-SPEC: :VOID
MIN-ARGS: 3
we see that there isn't a whole lot there. If the MIN-ARGS field has
any useful purpose anymore, I don't remember what that purpose would
be; assuming that we can think of a reasonable way to map the foreign
name ("glColor3f") to a lisp symbol (maybe something like OPENGL:GL-COLOR3F,
maybe something better ...) it's not too hard to imagine tha we could
automatically generate:
(declaim (inline opengl::gl-color3f))
(defun opengl::gl-color3f (arg-0 arg-1 arg-2)
(ccl:external-call "glColor3f" ; the CCL::EFD-ENTRY-NAME
:<gl>float arg-0
:<gl>float arg-1
:<gl>float arg-2
:void))
A few things worth noting:
1) Any information about parameter names that might have existed in the .h
file in the declaration of glColor3f is long gone:
extern void glColor3f (GLfloat red, GLfloat green, GLfloat blue);
It might be nice to be able to generate more meaningful names.
2) Argument and result type specifiers might be primitive, built-in
things (like :int and :void) or might be things that're themselves
defined in the .cdb files. You could:
a) yank those things in and predefine them (by enumerating the
keys in the "types.cdb" file for :gl)
b) let CCL:EXTERNAL-CALL look them up at macroexpand time
(a) can be machine-dependent. (I'm not sure how true this is
for OpenGL types, but it can generally be true that the definition
of something like a 64-bit signed integer can be sensitive to word-size
issues, structure types and bitfields and other things can have
machine-dependent alignment and packing rules and may be sensitive
to endianness, etc.)
(b) should do the right thing, but depends on details of CCL's FFI
to do so.
3) I'm not sure how often this comes up in OpenGL, but if the last
arg-spec in a function definition is the symbol :VOID (which is
otherwise meaningless as an argument type), that means "the function
accepts an indefinite number of extra arguments." (Using the "void"
type as a sort of &rest marker is actually how GCC represents this
internally, and that gets passed through the ffi translation process.)
I checked, and this doesn't seem to happen in OpenGL. Never mind.
4) A lot of OpenGL functions take arguments and return results
by value (as glColor3f does), so the idea of treating them as
simple lisp functions makes sense. Other cases may be harder
to map directly: glColor3fv takes a "vector" (C pointer) that's
presumed to point to 3 :<GL>floats. If you want to support
that, you may have to decide whether to expose the FFI (e.g.,
make the caller stack-cons a "vector of 3 floats" and pass
the raw pointer in, or do it for them (e.g., have the function
accept a lisp vector and copy its contents to a foreign vector
that the function allocates itself.) It's probably hard to
automate the translation process, and it might be wise to
just do the low-level thing and discourage students from using
the "v" functions, at least initialy.
5) Lastly: I'm sure that this (generating lisp FFI bindings for
OpenGL) has been done before (probably many times) and there may be
things out there that are accurate, complete, and relatively portable
(using CFFI or UFFI or some other FFI abstraction layer.) We did
a fairly large project with a fairly large OpenGL component a
few years ago (everyone, rush out and buy a copy of InspireData
now), and at that time we had to manually define interfaces for
OpenGL things that weren't present in the vendor's OpenGL interface.
That was 4-5 years ago, and the situation may have improved a lot
since then; if someone's already put a lot of thought into how
to solve some of these issues - like naming conventions - it might
be wise to look on common-lisp.net to see what approaches others
have taken.
On Thu, 8 Jan 2009, Alexander Repenning wrote:
> I promise some technical questions, mostly aimed at Gary I assume, but first
> a motivational, short rant. Unlike some of the voices on comp.lang.lisp I do
> not think that CL is either doomed or is about to take off like crazy by
> adjusting itself (e.g., its syntax). Education is key. Get the CS
> undergraduates excited about CL somehow and CL could actually gain new ground
> instead of just holding on to its legacy developers. But how? Intro to CS
> courses, e.g., Scheme based on Abelson are probably doomed given that even
> Abelson himself is no longer using Lisp at MIT for that course. Instead, I
> claim, it makes more sense to have a compelling package fitting a course such
> a graphics course where students typically follow the OpenGL red book and do
> weekly programming assignments. The point of these courses it to learn about
> graphics and to have a nice tool. Running your event, animation loop and
> change your code on the fly -WOW. Yes, Lisp can do this. I have been teaching
> these kinds of course and I tell you a good percentage of students started
> to really like to program Lisp because of this experience. At the time the
> MCL license and sales "strategy" became a show stopper. CCL could become this
> kind of tool. If it can be delivered in a way that I, as prof, have my
> students download Mac or Windows versions and they can open up OpenGL windows
> and start typing in OpenGL code right away. Then, when the projects become a
> bit more complex CL even delivers good speed in contrast to most scripting
> languages that typically used to make OpenGL programming simple.
>
> Technical challenges:
>
> In general I am not against the MCL/CCL #_<function_name> syntax to invoke
> system calls. For instance, with OpenGL, to make the obligatory RGB triangle
> your code would like this:
>
> (defmethod DRAW ((Self opengl-view))
> (#_glBegin #$GL_TRIANGLES)
> (#_glColor3f 1.0 0.0 0.0)
> (#_glVertex3f 0.0 0.6 0.0)
> (#_glColor3f 0.0 1.0 0.0)
> (#_glVertex3f -0.2 -0.3 0.0)
> (#_glColor3f 0.0 0.0 1.0)
> (#_glVertex3f 0.2 -0.3 0.0)
> (#_glEnd))
>
> but as it turns out most people prefer:
>
> (defmethod DRAW ((Self opengl-view))
> (glBegin GL_TRIANGLES)
> (glColor3f 1.0 0.0 0.0)
> (glVertex3f 0.0 0.6 0.0)
> (glColor3f 0.0 1.0 0.0)
> (glVertex3f -0.2 -0.3 0.0)
> (glColor3f 0.0 0.0 1.0)
> (glVertex3f 0.2 -0.3 0.0)
> (glEnd))
>
> Now one can argue why, at least in the case of OpenGL, this is preferable.
> Here are some reasons:
>
> - the #_ function call mechanism establishes binding at read time via a DB.
> This is good for the program, perhaps, but not good for the programmer.
> Before Lisp reads a correct opengl function call Lisp does not have an
> interned symbol relating to this function name. As a consequence symbol
> completion cannot work. In an API with close to 1000 functions/constants/
> this makes a huge difference.
>
> - the #_ syntax may not be cross platform or cross Lisp
>
> - most OpenGL apps have concentrated OpenGL call where virtually ALL the
> function calls are GL function calls. Ergo you get just about each line of
> line of code containing some #_ which after a while starts to look awkward
> and redundant.
>
> How could we address this? Here are some bad ideas:
>
> 1) define a function for each foreign function, e.g.,
>
> (defun glColor3f (R G B)
> (#_glColor3f R G B))
>
> - not good: function call overhead
>
> 2) define a macro for each foreign function, e.g.,
>
> (defmacro glColor3f (R B G)
> <call read-dispatch function via *readtable* to expand into same thing as (#_
> glColor3f ...) >
>
> -> (%FF-CALL (%REFERENCE-EXTERNAL-ENTRY-POINT
> (LOAD-TIME-VALUE (EXTERNAL "_glColor3f")))
> :SINGLE-FLOAT R
> :SINGLE-FLOAT G
> :SINGLE-FLOAT B
> :VOID)
>
> - as fast as #_ glColor3f ...)
>
>
> The biggest issue with both approaches is that I do not know the name of the
> function and its parameters to create a function or a macro.
>
> In MCL and Allegro we used these LONG lists:
>
> ....
>
> (foreign-function glvertex3f (float FLOAT FLOAT) VOID "glVertex3f")
> (foreign-function glvertex3i (INT INT INT) VOID "glVertex3i")
> (foreign-function glvertex3s (SHORT SHORT SHORT) VOID "glVertex3s")
> (foreign-function glvertex4d (DOUBLE DOUBLE DOUBLE DOUBLE) VOID "glVertex4d")
>
> ...
>
> to create bindings to OS X AGL and Windows WGL
>
> Manually maintaining these kinds of lists is tedious and error prone.
>
> What can we do? Could we have a function to parse the
> "/System/Library/Frameworks/OpenGL.framework/OpenGL" framework to generate
> foreign interfaces. CCL must have some code to generate access framework
> entries and generate interfaces.
>
>
> Alex
>
>
> Prof. Alexander Repenning
>
> University of Colorado
> Computer Science Department
> Boulder, CO 80309-430
>
> vCard: http://www.cs.colorado.edu/~ralex/AlexanderRepenning.vcf
>
>
-------------- next part --------------
(in-package "CL-USER")
(defun obtain-gl-function-info ()
(let* ((gl-function-info (make-hash-table :test #'equal))
(cdb (ccl::db-functions (use-interface-dir :gl))))
(dolist (name (ccl::cdb-enumerate-keys cdb (lambda (s) (and (> (length s) 2) (string= "gl" s :end2 2)))))
(let* ((info (ccl::db-lookup-function cdb name)))
(if info
(setf (gethash name gl-function-info) info)
(warn "Odd. There's no foreign function info for ~s" name))))
gl-function-info))
More information about the Openmcl-devel
mailing list