[Openmcl-devel] getting CCL ready for CS graphics courses
Alexander Repenning
ralex at cs.colorado.edu
Wed Jan 14 17:41:59 PST 2009
thank you for the code and explanations. Some followup questions:
- how does the .cdb file get created. I assume that is a CCL thing
and that the .cdb file is the result of parsing the header file
contained in the framework?
- how hard would it be to parse and include the parameter names? Could
the header file parser be extended to build a .cdb file that would
include parameter name info? Just seems to be a real shame to skip
that kind of info given that it is in the header files.
- what about constants? The .cdb files does not appear to include
"CL_..." constants such as GL_COLOR_BUFFER_BIT, yet CCL clearly has
access to this info, e.g., #$GL_COLOR_BUFFER_BIT expands into
DARWIN32::GL_COLOR_BUFFER_BIT of value 16384.
- type names: Can one use the type names returned by the gl-function-
info, e.g., :single-float or does one have to use the :<gl>float?
(defun gl-vertex3f (arg-0 arg-1 arg-2)
(ccl:external-call "_glVertex3f" ; the CCL::EFD-ENTRY-NAME
:single-float arg-0
:<gl>float arg-1
:<gl>float arg-2
:void))
alex
On Jan 8, 2009, at 4:59 PM, Gary Byers wrote:
> #_ basically hashes the function name which follows and tries to find
> an entry for that name in some "functions.cdb" file (where an "entry"
> is a sort of byte-coded representation of the function's argument and
> return types.) #_ parses the encoded argument into a little lisp
> structure and adds that structure as the value of the key
> OS::|caseSensitiveName| in a hash table. The final read-time actions
> are to define OS::|caseSensitiveName| as a macro (all of the things
> defined via #_ share the same macro definition, named something like
> CCL::%EXTERNAL-CALL-EXPANDER) and return OS::|caseSensitiveName|.
>
> So, reading:
>
> (#_read fd buf n)
>
> tries to ensure that some foreign function definition info is
> associated
> with OS::|read| and that OS::|read| is defined as the magic macro
> (when
> processing the #_) and the reader returns:
>
> (os::read fd buf n)
>
> When macroexpanded, that turns into something like:
>
> (external-call "read" :int fd :address buf :some-kind-of-int n :some-
> kind-of-int)
>
> which eventually macroexpands down to %FF-CALL.
>
> What you presumably want to do is something similar. only avoiding
> the use of #_ and defining (possibly inlined) functions or macros
> with lispier names (gl-color-3f rather than |glColor3f|, etc.)
>
> It's possible to enumerate all of the keys in a .cdb file - I think
> that
> the function/macro that does this is called CCL::CDB-ENUMERATE-KEYS,
> of all things; this returns a list of all strings used as keys in
> the .cdb
> file (in random order). If you do:
>
>
> (ccl::cdb-enumerate-keys (ccl::db-functions (use-interface-dir :gl)))
>
> you'll get a list of > 7000 strings; that includes everything that's
> defined in the :gl functions.db file.
>
> Unfortunately, that list will include the names of things that are
> referenced in some GL-related header file (including a lot of Carbon
> and C library stuff defined in agl.h). If a heuristic is that all
> real GL function names start with the string "gl" (and few non-GL
> function names do), you can pass a second "filter/predicate" argument
> to CDB-ENUMERATE-KEYS that decides whether or not to include a given
> string in the list. (This may or may not be easier than generating
> the full list and post-processing it.)
>
> However we do it, if we restrict the set of defined keys to names
> that start with "gl", we get a list with ~1136 entries.
>
> The enclosed function OBTAIN-GL-FUNCTION-INFO will return a hash
> table whose keys are GL function names (mixed-case strings) and
> whose values are structures (CCL::EXTERNAL-FUNCTION-DEFINITIONs);
> accessors with the prefix (defstruct :conc-name) CCL::EFD- can
> be used to access the fields of an EXTERNAL-FUNCTION-DEFINITION
> structure. If we look at one of these (chosen at random based
> on my fading memory of OpenGL):
>
> ? (defvar *gl-info* (obtain-gl-function-info))
>
> ? (describe (gethash "glColor3f" *gl-info*))
> #S(CCL::EXTERNAL-FUNCTION-DEFINITION :ENTRY-NAME "glColor3f"
> :ARG-SPECS (:<GL>FLOAT
> :<GL>FLOAT
> :<GL>FLOAT)
> :RESULT-SPEC :VOID :MIN-ARGS 3)
> Type: CCL::EXTERNAL-FUNCTION-DEFINITION
> Class: #<STRUCTURE-CLASS CCL::EXTERNAL-FUNCTION-DEFINITION>
> ENTRY-NAME: "glColor3f"
> ARG-SPECS: (:<GL>FLOAT :<GL>FLOAT :<GL>FLOAT)
> RESULT-SPEC: :VOID
> MIN-ARGS: 3
>
> we see that there isn't a whole lot there. If the MIN-ARGS field has
> any useful purpose anymore, I don't remember what that purpose would
> be; assuming that we can think of a reasonable way to map the foreign
> name ("glColor3f") to a lisp symbol (maybe something like OPENGL:GL-
> COLOR3F,
> maybe something better ...) it's not too hard to imagine tha we could
> automatically generate:
>
> (declaim (inline opengl::gl-color3f))
> (defun opengl::gl-color3f (arg-0 arg-1 arg-2)
> (ccl:external-call "glColor3f" ; the CCL::EFD-ENTRY-NAME
> :<gl>float arg-0
> :<gl>float arg-1
> :<gl>float arg-2
> :void))
>
> A few things worth noting:
>
> 1) Any information about parameter names that might have existed in
> the .h
> file in the declaration of glColor3f is long gone:
>
>
> extern void glColor3f (GLfloat red, GLfloat green, GLfloat blue);
>
> It might be nice to be able to generate more meaningful names.
>
> 2) Argument and result type specifiers might be primitive, built-in
> things (like :int and :void) or might be things that're themselves
> defined in the .cdb files. You could:
>
> a) yank those things in and predefine them (by enumerating the
> keys in the "types.cdb" file for :gl)
>
> b) let CCL:EXTERNAL-CALL look them up at macroexpand time
>
> (a) can be machine-dependent. (I'm not sure how true this is
> for OpenGL types, but it can generally be true that the definition
> of something like a 64-bit signed integer can be sensitive to word-
> size
> issues, structure types and bitfields and other things can have
> machine-dependent alignment and packing rules and may be sensitive
> to endianness, etc.)
>
> (b) should do the right thing, but depends on details of CCL's FFI
> to do so.
>
> 3) I'm not sure how often this comes up in OpenGL, but if the last
> arg-spec in a function definition is the symbol :VOID (which is
> otherwise meaningless as an argument type), that means "the function
> accepts an indefinite number of extra arguments." (Using the "void"
> type as a sort of &rest marker is actually how GCC represents this
> internally, and that gets passed through the ffi translation process.)
> I checked, and this doesn't seem to happen in OpenGL. Never mind.
>
> 4) A lot of OpenGL functions take arguments and return results
> by value (as glColor3f does), so the idea of treating them as
> simple lisp functions makes sense. Other cases may be harder
> to map directly: glColor3fv takes a "vector" (C pointer) that's
> presumed to point to 3 :<GL>floats. If you want to support
> that, you may have to decide whether to expose the FFI (e.g.,
> make the caller stack-cons a "vector of 3 floats" and pass the raw
> pointer in, or do it for them (e.g., have the function
> accept a lisp vector and copy its contents to a foreign vector
> that the function allocates itself.) It's probably hard to
> automate the translation process, and it might be wise to
> just do the low-level thing and discourage students from using
> the "v" functions, at least initialy.
>
> 5) Lastly: I'm sure that this (generating lisp FFI bindings for
> OpenGL) has been done before (probably many times) and there may be
> things out there that are accurate, complete, and relatively portable
> (using CFFI or UFFI or some other FFI abstraction layer.) We did
> a fairly large project with a fairly large OpenGL component a
> few years ago (everyone, rush out and buy a copy of InspireData
> now), and at that time we had to manually define interfaces for
> OpenGL things that weren't present in the vendor's OpenGL interface.
> That was 4-5 years ago, and the situation may have improved a lot
> since then; if someone's already put a lot of thought into how
> to solve some of these issues - like naming conventions - it might
> be wise to look on common-lisp.net to see what approaches others
> have taken.
>
>
>
> On Thu, 8 Jan 2009, Alexander Repenning wrote:
>
>> I promise some technical questions, mostly aimed at Gary I assume,
>> but first a motivational, short rant. Unlike some of the voices on
>> comp.lang.lisp I do not think that CL is either doomed or is about
>> to take off like crazy by adjusting itself (e.g., its syntax).
>> Education is key. Get the CS undergraduates excited about CL
>> somehow and CL could actually gain new ground instead of just
>> holding on to its legacy developers. But how? Intro to CS courses,
>> e.g., Scheme based on Abelson are probably doomed given that even
>> Abelson himself is no longer using Lisp at MIT for that course.
>> Instead, I claim, it makes more sense to have a compelling package
>> fitting a course such a graphics course where students typically
>> follow the OpenGL red book and do weekly programming assignments.
>> The point of these courses it to learn about graphics and to have a
>> nice tool. Running your event, animation loop and change your code
>> on the fly -WOW. Yes, Lisp can do this. I have been teaching these
>> kinds of course and I tell you a good percentage of students
>> started to really like to program Lisp because of this experience.
>> At the time the MCL license and sales "strategy" became a show
>> stopper. CCL could become this kind of tool. If it can be delivered
>> in a way that I, as prof, have my students download Mac or Windows
>> versions and they can open up OpenGL windows and start typing in
>> OpenGL code right away. Then, when the projects become a bit more
>> complex CL even delivers good speed in contrast to most scripting
>> languages that typically used to make OpenGL programming simple.
>>
>> Technical challenges:
>>
>> In general I am not against the MCL/CCL #_<function_name> syntax to
>> invoke system calls. For instance, with OpenGL, to make the
>> obligatory RGB triangle your code would like this:
>>
>> (defmethod DRAW ((Self opengl-view))
>> (#_glBegin #$GL_TRIANGLES)
>> (#_glColor3f 1.0 0.0 0.0)
>> (#_glVertex3f 0.0 0.6 0.0)
>> (#_glColor3f 0.0 1.0 0.0)
>> (#_glVertex3f -0.2 -0.3 0.0)
>> (#_glColor3f 0.0 0.0 1.0)
>> (#_glVertex3f 0.2 -0.3 0.0)
>> (#_glEnd))
>>
>> but as it turns out most people prefer:
>>
>> (defmethod DRAW ((Self opengl-view))
>> (glBegin GL_TRIANGLES)
>> (glColor3f 1.0 0.0 0.0)
>> (glVertex3f 0.0 0.6 0.0)
>> (glColor3f 0.0 1.0 0.0)
>> (glVertex3f -0.2 -0.3 0.0)
>> (glColor3f 0.0 0.0 1.0)
>> (glVertex3f 0.2 -0.3 0.0)
>> (glEnd))
>>
>> Now one can argue why, at least in the case of OpenGL, this is
>> preferable. Here are some reasons:
>>
>> - the #_ function call mechanism establishes binding at read time
>> via a DB. This is good for the program, perhaps, but not good for
>> the programmer. Before Lisp reads a correct opengl function call
>> Lisp does not have an interned symbol relating to this function
>> name. As a consequence symbol completion cannot work. In an API
>> with close to 1000 functions/constants/ this makes a huge difference.
>>
>> - the #_ syntax may not be cross platform or cross Lisp
>>
>> - most OpenGL apps have concentrated OpenGL call where virtually
>> ALL the function calls are GL function calls. Ergo you get just
>> about each line of line of code containing some #_ which after a
>> while starts to look awkward and redundant.
>>
>> How could we address this? Here are some bad ideas:
>>
>> 1) define a function for each foreign function, e.g.,
>>
>> (defun glColor3f (R G B)
>> (#_glColor3f R G B))
>>
>> - not good: function call overhead
>>
>> 2) define a macro for each foreign function, e.g.,
>>
>> (defmacro glColor3f (R B G)
>> <call read-dispatch function via *readtable* to expand into same
>> thing as (#_ glColor3f ...) >
>>
>> -> (%FF-CALL (%REFERENCE-EXTERNAL-ENTRY-POINT
>> (LOAD-TIME-VALUE (EXTERNAL "_glColor3f")))
>> :SINGLE-FLOAT R
>> :SINGLE-FLOAT G
>> :SINGLE-FLOAT B
>> :VOID)
>>
>> - as fast as #_ glColor3f ...)
>>
>>
>> The biggest issue with both approaches is that I do not know the
>> name of the function and its parameters to create a function or a
>> macro.
>>
>> In MCL and Allegro we used these LONG lists:
>>
>> ....
>>
>> (foreign-function glvertex3f (float FLOAT FLOAT) VOID "glVertex3f")
>> (foreign-function glvertex3i (INT INT INT) VOID "glVertex3i")
>> (foreign-function glvertex3s (SHORT SHORT SHORT) VOID "glVertex3s")
>> (foreign-function glvertex4d (DOUBLE DOUBLE DOUBLE DOUBLE) VOID
>> "glVertex4d")
>>
>> ...
>>
>> to create bindings to OS X AGL and Windows WGL
>>
>> Manually maintaining these kinds of lists is tedious and error prone.
>>
>> What can we do? Could we have a function to parse the "/System/
>> Library/Frameworks/OpenGL.framework/OpenGL" framework to generate
>> foreign interfaces. CCL must have some code to generate access
>> framework entries and generate interfaces.
>>
>>
>> Alex
>>
>>
>> Prof. Alexander Repenning
>>
>> University of Colorado
>> Computer Science Department
>> Boulder, CO 80309-430
>>
>> vCard: http://www.cs.colorado.edu/~ralex/AlexanderRepenning.vcf
>>
> <gl-function-info.lisp>
Prof. Alexander Repenning
University of Colorado
Computer Science Department
Boulder, CO 80309-430
vCard: http://www.cs.colorado.edu/~ralex/AlexanderRepenning.vcf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clozure.com/pipermail/openmcl-devel/attachments/20090114/74f90da8/attachment.htm>
More information about the Openmcl-devel
mailing list