[Openmcl-devel] getting CCL ready for CS graphics courses

Alexander Repenning ralex at cs.colorado.edu
Wed Jan 14 17:41:59 PST 2009

thank you for the code and explanations. Some followup questions:

- how does the .cdb  file get created. I assume that is a CCL thing  
and that the .cdb file is the result of parsing the header file  
contained in the framework?

- how hard would it be to parse and include the parameter names? Could  
the header file parser be extended to build a .cdb file that would  
include parameter name info? Just seems to be a real shame to skip  
that kind of info given that it is in the header files.

- what about constants? The .cdb files does not appear to include  
"CL_..." constants such as GL_COLOR_BUFFER_BIT, yet CCL clearly has  
access to this info, e.g., #$GL_COLOR_BUFFER_BIT expands into  
DARWIN32::GL_COLOR_BUFFER_BIT of value 16384.

- type names: Can one use the type names returned by the gl-function- 
info, e.g., :single-float or does one have to use the :<gl>float?

(defun gl-vertex3f (arg-0 arg-1 arg-2)
  (ccl:external-call "_glVertex3f" ; the CCL::EFD-ENTRY-NAME
                     :single-float arg-0
                     :<gl>float arg-1
                     :<gl>float arg-2


On Jan 8, 2009, at 4:59 PM, Gary Byers wrote:

> #_ basically hashes the function name which follows and tries to find
> an entry for that name in some "functions.cdb" file (where an "entry"
> is a sort of byte-coded representation of the function's argument and
> return types.)  #_ parses the encoded argument into a little lisp
> structure and adds that structure as the value of the key
> OS::|caseSensitiveName| in a hash table.  The final read-time actions
> are to define OS::|caseSensitiveName| as a macro (all of the things
> defined via #_ share the same macro definition, named something like
> CCL::%EXTERNAL-CALL-EXPANDER) and return OS::|caseSensitiveName|.
> So, reading:
> (#_read fd buf n)
> tries to ensure that some foreign function definition info is  
> associated
> with OS::|read| and that OS::|read| is defined as the magic  macro  
> (when
> processing the #_) and the reader returns:
> (os::read fd buf n)
> When macroexpanded, that turns into something like:
> (external-call "read" :int fd :address buf :some-kind-of-int n :some- 
> kind-of-int)
> which eventually macroexpands down to %FF-CALL.
> What you presumably want to do is something similar. only avoiding
> the use of #_ and defining (possibly inlined) functions or macros
> with lispier names (gl-color-3f rather than |glColor3f|, etc.)
> It's possible to enumerate all of the keys in a .cdb file - I think  
> that
> the function/macro that does this is called CCL::CDB-ENUMERATE-KEYS,  
> of all things; this returns a list of all strings used as keys in  
> the .cdb
> file (in random order).  If you do:
> (ccl::cdb-enumerate-keys (ccl::db-functions (use-interface-dir :gl)))
> you'll get a list of > 7000 strings; that includes everything that's
> defined in the :gl functions.db file.
> Unfortunately, that list will include the names of things that are
> referenced in some GL-related header file (including a lot of Carbon
> and C library stuff defined in agl.h).  If a heuristic is that all
> real GL function names start with the string "gl" (and few non-GL
> function names do), you can pass a second "filter/predicate" argument
> to CDB-ENUMERATE-KEYS that decides whether or not to include a given
> string in the list.  (This may or may not be easier than generating
> the full list and post-processing it.)
> However we do it, if we restrict the set of defined keys to names
> that start with "gl", we get a list with ~1136 entries.
> The enclosed function OBTAIN-GL-FUNCTION-INFO will return a hash
> table whose keys are GL function names (mixed-case strings) and
> whose values are structures (CCL::EXTERNAL-FUNCTION-DEFINITIONs);
> accessors with the prefix (defstruct :conc-name) CCL::EFD- can
> be used to access the fields of an EXTERNAL-FUNCTION-DEFINITION
> structure.  If we look at one of these (chosen at random based
> on my fading memory of OpenGL):
> ? (defvar *gl-info* (obtain-gl-function-info))
> ? (describe (gethash "glColor3f" *gl-info*))
>                                     :ARG-SPECS (:<GL>FLOAT
>                                                 :<GL>FLOAT
>                                                 :<GL>FLOAT)
>                                     :RESULT-SPEC :VOID :MIN-ARGS 3)
> ENTRY-NAME: "glColor3f"
> we see that there isn't a whole lot there.  If the MIN-ARGS field has
> any useful purpose anymore, I don't remember what that purpose would
> be; assuming that we can think of a reasonable way to map the foreign
> name ("glColor3f") to a lisp symbol (maybe something like OPENGL:GL- 
> maybe something better ...) it's not too hard to imagine tha we could
> automatically generate:
> (declaim (inline opengl::gl-color3f))
> (defun opengl::gl-color3f (arg-0 arg-1 arg-2)
>  (ccl:external-call "glColor3f" ; the CCL::EFD-ENTRY-NAME
>                     :<gl>float arg-0
>                     :<gl>float arg-1
>                     :<gl>float arg-2
>                     :void))
> A few things worth noting:
> 1) Any information about parameter names that might have existed in  
> the .h
> file in the declaration of glColor3f is long gone:
> extern void glColor3f (GLfloat red, GLfloat green, GLfloat blue);
> It might be nice to be able to generate more meaningful names.
> 2) Argument and result type specifiers might be primitive, built-in
> things (like :int and :void) or might be things that're themselves
> defined in the .cdb files.  You could:
>  a) yank those things in and predefine them (by enumerating the
> keys in the "types.cdb" file for :gl)
>  b) let CCL:EXTERNAL-CALL look them up at macroexpand time
> (a) can be machine-dependent. (I'm not sure how true this is
> for OpenGL types, but it can generally be true that the definition
> of something like a 64-bit signed integer can be sensitive to word- 
> size
> issues, structure types and bitfields and other things can have
> machine-dependent alignment and packing rules and may be sensitive
> to endianness, etc.)
> (b) should do the right thing, but depends on details of CCL's FFI
> to do so.
> 3) I'm not sure how often this comes up in OpenGL, but if the last
> arg-spec in a function definition is the symbol :VOID (which is
> otherwise meaningless as an argument type), that means "the function
> accepts an indefinite number of extra arguments."  (Using the "void"
> type as a sort of &rest marker is actually how GCC represents this
> internally, and that gets passed through the ffi translation process.)
> I checked, and this doesn't seem to happen in OpenGL.  Never mind.
> 4) A lot of OpenGL functions take arguments and return results
> by value (as glColor3f does), so the idea of treating them as
> simple lisp functions makes sense.  Other cases may be harder
> to map directly: glColor3fv takes a "vector" (C pointer) that's
> presumed to point to 3 :<GL>floats.  If you want to support
> that, you may have to decide whether to expose the FFI (e.g.,
> make the caller stack-cons a "vector of 3 floats" and pass the raw  
> pointer in, or do it for them (e.g., have the function
> accept a lisp vector and copy its contents to a foreign vector
> that the function allocates itself.)  It's probably hard to
> automate the translation process, and it might be wise to
> just do the low-level thing and discourage students from using
> the "v" functions, at least initialy.
> 5) Lastly: I'm sure that this (generating lisp FFI bindings for
> OpenGL) has been done before (probably many times) and there may be
> things out there that are accurate, complete, and relatively portable
> (using CFFI or UFFI or some other FFI abstraction layer.)  We did
> a fairly large project with a fairly large OpenGL component a
> few years ago (everyone, rush out and buy a copy of InspireData
> now), and at that time we had to manually define interfaces for
> OpenGL things that weren't present in the vendor's OpenGL interface.
> That was 4-5 years ago, and the situation may have improved a lot
> since then; if someone's already put a lot of thought into how
> to solve some of these issues - like naming conventions - it might
> be wise to look on common-lisp.net to see what approaches others
> have taken.
> On Thu, 8 Jan 2009, Alexander Repenning wrote:
>> I promise some technical questions, mostly aimed at Gary I assume,  
>> but first a motivational, short rant. Unlike some of the voices on  
>> comp.lang.lisp I do not think that CL is either doomed or is about  
>> to take off like crazy by adjusting itself (e.g., its syntax).  
>> Education is key. Get the CS undergraduates excited about CL  
>> somehow and CL could actually gain new ground instead of just  
>> holding on to its legacy developers. But how? Intro to CS courses,  
>> e.g., Scheme based on Abelson are probably doomed given that even  
>> Abelson himself is no longer using Lisp at MIT for that course.  
>> Instead, I claim, it makes more sense to have a compelling package  
>> fitting a course such a graphics course where students typically  
>> follow the OpenGL red book and do weekly programming assignments.  
>> The point of these courses it to learn about graphics and to have a  
>> nice tool. Running your event, animation loop and change your code  
>> on the fly -WOW. Yes, Lisp can do this. I have been teaching these  
>> kinds of course and I tell you a good percentage of  students  
>> started to really like to program Lisp because of this experience.  
>> At the time the MCL license and sales "strategy" became a show  
>> stopper. CCL could become this kind of tool. If it can be delivered  
>> in a way that I, as prof, have my students download Mac or Windows  
>> versions and they can open up OpenGL windows and start typing in  
>> OpenGL code right away. Then, when the projects become a bit more  
>> complex CL even delivers good speed in contrast to most scripting  
>> languages that typically used to make OpenGL programming simple.
>> Technical challenges:
>> In general I am not against the MCL/CCL #_<function_name> syntax to  
>> invoke system calls. For instance, with OpenGL, to make the  
>> obligatory RGB triangle your code would like this:
>> (defmethod DRAW ((Self opengl-view))
>> (#_glBegin #$GL_TRIANGLES)
>> (#_glColor3f 1.0 0.0 0.0)
>> (#_glVertex3f 0.0 0.6 0.0)
>> (#_glColor3f 0.0 1.0 0.0)
>> (#_glVertex3f -0.2 -0.3 0.0)
>> (#_glColor3f 0.0 0.0 1.0)
>> (#_glVertex3f 0.2 -0.3 0.0)
>> (#_glEnd))
>> but as it turns out most people prefer:
>> (defmethod DRAW ((Self opengl-view))
>> (glBegin GL_TRIANGLES)
>> (glColor3f 1.0 0.0 0.0)
>> (glVertex3f 0.0 0.6 0.0)
>> (glColor3f 0.0 1.0 0.0)
>> (glVertex3f -0.2 -0.3 0.0)
>> (glColor3f 0.0 0.0 1.0)
>> (glVertex3f 0.2 -0.3 0.0)
>> (glEnd))
>> Now one can argue why, at least in the case of OpenGL, this is  
>> preferable. Here are some reasons:
>> - the #_ function call mechanism establishes binding at read time  
>> via a DB. This is good for the program, perhaps, but not good for  
>> the programmer. Before Lisp reads a correct opengl function call  
>> Lisp does not have an interned symbol relating to this function  
>> name. As a consequence symbol completion cannot work. In an API  
>> with close to 1000 functions/constants/ this makes a huge difference.
>> - the #_ syntax may not be cross platform or cross Lisp
>> - most OpenGL apps have concentrated OpenGL call where virtually  
>> ALL the function calls are GL function calls. Ergo you get just  
>> about each line of line of code containing some #_ which after a  
>> while starts to look awkward and redundant.
>> How could we address this? Here are some bad ideas:
>> 1) define a function for each foreign function, e.g.,
>> (defun  glColor3f (R G B)
>> (#_glColor3f R G B))
>> - not good: function call overhead
>> 2) define a macro for each foreign function, e.g.,
>> (defmacro glColor3f (R B G)
>> <call read-dispatch function via *readtable* to expand into same  
>> thing as (#_ glColor3f  ...) >
>>         (LOAD-TIME-VALUE (EXTERNAL "_glColor3f")))
>>       :SINGLE-FLOAT R
>>       :SINGLE-FLOAT G
>>       :SINGLE-FLOAT B
>>       :VOID)
>> - as fast as #_ glColor3f  ...)
>> The biggest issue with both approaches is that I do not know the  
>> name of the function and its parameters to create a function or a  
>> macro.
>> In MCL and Allegro we used these LONG lists:
>> ....
>> (foreign-function glvertex3f (float FLOAT FLOAT) VOID "glVertex3f")
>> (foreign-function glvertex3i (INT INT INT) VOID "glVertex3i")
>> (foreign-function glvertex3s (SHORT SHORT SHORT) VOID "glVertex3s")
>> (foreign-function glvertex4d (DOUBLE DOUBLE DOUBLE DOUBLE) VOID  
>> "glVertex4d")
>> ...
>> to create bindings to OS X AGL and Windows WGL
>> Manually maintaining these kinds of lists is tedious and error prone.
>> What can we do? Could we have a function to parse the "/System/ 
>> Library/Frameworks/OpenGL.framework/OpenGL" framework to generate  
>> foreign interfaces. CCL must have some code to generate access  
>> framework entries and generate interfaces.
>> Alex
>> Prof. Alexander Repenning
>> University of Colorado
>> Computer Science Department
>> Boulder, CO 80309-430
>> vCard: http://www.cs.colorado.edu/~ralex/AlexanderRepenning.vcf
> <gl-function-info.lisp>

Prof. Alexander Repenning

University of Colorado
Computer Science Department
Boulder, CO 80309-430

vCard: http://www.cs.colorado.edu/~ralex/AlexanderRepenning.vcf

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clozure.com/pipermail/openmcl-devel/attachments/20090114/74f90da8/attachment.htm>

More information about the Openmcl-devel mailing list