[Openmcl-devel] Is OpenMCL ready for use?

[Gary Byers]

On Sun, 15 Dec 2002, Stonewall Ballard wrote:

> I'm becoming very encouraged by the MCL 5 beta and the OpenMCL efforts.
>

I hope that other people will be inclined to respond to this with
their opinions.

> I'm trying to convince myself that I can use one or both of these to
> develop competitive desktop applications. "Competitive" in this case
> means that users won't suspect that the app was built using Lisp,
> they won't find anything amiss in performance or size. So long as
> the compiler is good, the GC has no perceptible pauses, I can escape
> to C if necessary, and I have complete and reasonably convenient
> access to the system services, I should be able to use it.

I think that the goal (sometimes obscured by intermediate goals) of
both environments is to provide that capability.  I think that it's
also fair to say that in other development environments
(ProjectBuilder, for example) that's more obviously the focus, and
that something like PB provides a lot more infrastructure in support
of that process.  I would like to see OpenMCL provide a lot more of
that infrastructure than it does.

It's difficult to write compelling (or even interesting) applications
in any environment; it's perhaps telling that my idea of an
interesting application leads to something like OpenMCL's Cocoa demo.
I honestly don't know (or at least don't know in great detail) what
kind of infrastructure and support would be required to turn other
kinds of compelling or interesting Lisp programs into "competitive
desktop applications".  I've always thought that that question would
be easier to answer if I had my hands on one or more Lisp programs
that would be competitive desktop applications if the development
environment provided a little more/a lot more support, or got out of
the way, or both.

(If anyone has any such Lisp programs laying around, I'd be very
interested in seeing them.  Seriously.  Historically, such things
have often been heavily-guarded intellectual property.)

One of the original goals of (what became) OpenMCL was to make it
possible to develop embedded/server applications in Lisp.  That's
still an important (if not widely-shared) goal (it's important to
the people who pay me, therefore important to me.)  Some of the
issues there overlap with desktop-delivery issues.

> The Cocoa binding for OpenMCL is most interesting. Obviously,
> there's a lot to be done to make this usable, but it appears that
> the foundation is there and works.

I've been hung up on the fact that Cocoa isn't cross-platform: yes,
GnuStep exists, but it's not as mature and not widely used.  I don't
know how relevant that concern is: I don't know that many people are
interested in delivering UI-rich applications under Linux.  If the
Cocoa stuff had moved forward faster over the last several months,
that might have better served the needs of more users.  (It's not
clear that my thinking Deep Thoughts About The Cross-Platform UI Issue
has gotten too far.)

> I'm not clear on whether MCL 5 is stuck with Carbon, but I suspect
> that it is so long as it uses CFM. I have no interest in OS 9 apps,
> so I'd rather use a Mach-O based system.

There are a lot of people for whom backward compatibility is a big
concern.  MCL 5 certainly provides lots of backward compatibility;
since they seem to have that issue pretty well covered, it seems that
OpenMCL should ideally move in the other direction (faster than it's
been doing so lately.)

> My question is whether I should feel confident in moving ahead with
> OpenMCL now, or whether there's something missing that's going to
> bite me. I'm sure that I could spend too much time building out the
> Cocoa development tools, but that's part of the fun of using Lisp
> environments.

I've been finding that working on/poking around in the Cooca world is a
very good way of stress-testing the native threads that'll be in 0.14.
(It's incidentally nice to be able to say things like "wait - forever,
if need be - for an event" in one thread without affecting the behavior
or responsiveness of other threads.  It'll be even nicer when the GC
and resource-contention bugs go away ...)

> I'm most concerned about the ability of OpenMCL to create
> self-contained Cocoa apps. The package for the example Cocoa app
> worked well, but I don't see any automatic way to rip out the
> compiler or otherwise prune the image.  Is this hard to do?

I don't mean to trivialize this, but the easiest way of ripping out the
compiler from an application that you have source to is to not build it
into the application in the first place.  There's a way-out-of-date
file in the OpenMCL distribution ("ccl:level-1;runtime.lisp") that
(when it was up-to-date: note that it still assumes a binary extension
of ".pfsl", a la LinuxPPC) could be used by a modified version of the
bootrstapping image to produce an OpenMCL image that didn't contain
the compiler or other development tools.  That was done at JPL to answer
the question of "how big is an image into which application-specific
FASL files could be loaded"; at the time, the answer was "around 1MB,
without putting a whole lot of effort into reorganizing things."

The fact that that proof-of-concept once worked doesn't mean that the
problem's solved; the fact that an OpenMCL developer has control over
how their application (including OpenMCL itself) is built suggests that
there are a variety of approaches to the problem.   (This is clearly
an area in which additional infrastructure needs to be developed.)

MCL users used to send bug reports when an application from which the
compiler had been excised was no longer able to load and evaluate
lisp source files.  (The :EXCISE-COMPILER option to SAVE-APPLICATION
probably should have set *COMPILE-DEFINITIONS* to NIL, but I think
that we were so shocked to see people doing this kind of thing that
we never got to making a one-line change to SAVE-APPLICATION.)
Different users have different expectations and different levels of
sophistication; providing application-delivery infrastructure and
support that satisfies a wide spectrum of expectations and needs
sounds like a hard problem.

One approach to that problem is to present an MCL or OpenMCL image
and say "do X, Y, and Z to turn that image into your deliverable
application."  OpenMCL offers the alternate approach of saying
"here are some sources/fasls and a framework for loading them;
load the ones you need and make a deliverable application out of
them."  Neither approach is that simple in practice, but the latter
seems to offer a lot more flexibility.  It might be very hard to
do that now, but it could be made easier.

> The other concern I have is that this list seems dead lately. It's
> been two weeks since there was a post. Is OpenMCL proceeding at a
> pace to be reasonably usable within the next 4-6 months?

I don't think that this list has ever carried a high volume of traffic;
it's certainly been quiet lately, but it's always been kind of bursty.
If you look at the release history, development's also been bursty;
 releases were probably more regular in the past, but progress
(however it's measured) hasn't always been linear.

I think that there are several reasons for the current perceived lull
in development.

I was originally planning on "phasing in" support for native threads
over the course of a few releases, so perhaps 0.14 would have had
cooperatively scheduled threads based on Posix threads, 0.15 would
have changed special binding to work with a preemptive scheduler,
0.16 would have cleaned up some runtime issues, and ... sooner or
later there would have been a Big Bang and OpenMCL would have been
using preemptively scheduled native threads.

As I started to work on it, it became clear that there wasn't anything
incremental about the process: the hard parts involved (to strain the
analogy) the things that needed to happen right after the Big Bang:
the GC needs to be different in a preemptively scheduled world,
consing needs to be different, resource-contention problems need to be
dealt with differently, the FFI has to change ...

This realization (and some combination of adrenaline, caffiene, and
Wild Turkey) enabled me to get the incremental parts done a bit
quicker than expected: the Big Bang happened quite a while ago, and
I've been kind of cleaning up after it for the last few months (the GC
understands about native-thread-related issues, but has bugs; there
are mechanisms for dealing with resource-contention problems, but it
isn't always apparent that a resource-contention problem exists until
you figure out that you just got bitten by it.)

I generally have a lot of time to work on OpenMCL; in a classic case
of bad timing, I needed to get involved in other things shortly after
the Big Bang happened.  That kind of involvement's sometimes
necessary; OpenMCL's continued development is important to the people
I work for, and I don't think that anyone wants that "involvement in
other things" to distract from work on OpenMCL any more than it
(sometimes) has to.

That explanation hopefully helps to make the current sitution a little
more understandable; the fact that the last few releases have just
been collections of bug fixes shouldn't be construed as an indication
that OpenMCL isn't continuing to move forward, but the nature of the
current movement is a lot less incremental than it's been other times.
I should probably be doing a better job of communicating that than I
am, and if I'd stuck with the incremental plan that might have made
things clearer.

This all does emphasize one issue: I'm kind of the choke point or
bottleneck in a lot of OpenMCL development.  I don't know how many
other people are interested in working on OpenMCL itself, but there
are probably organizational and practical things that could make it
easier for them to do so.

> And last, I understand the difficulties in interpreting the license
> with a system like this, but as a practical matter, do I have to
> ship each app with some sort of patching facility (e.g. menu access
> to a load dialog) that lets end users change the code in the base
> image? That seems like a great way to confuse users and create a
> real support nightmare.
>

The last time I tried to think about how the LGPL applies to Lisp I
gave myself (and, presumably, anyone who read that message) a bad
headache.

If a preamble to the LGPL like the one under which Franz licensed
AllegroServe:

<http://AllegroServe.sourceforge.net/license-allegroserve.txt>

were provided as part of OpenMCL's licensing terms, would that
make our heads hurt less ?  (It might be fairly easy to get Digitool -
as primary copyright holder - to agree to that, especially if it
could be done in a way that didn't involve having to pay lawyers.)

Digitool's reasons for wanting to use the LGPL (as opposed to a
Berkeley-style license) had to do with their not wanting their
intellectual property to wind up in a competing commercial Lisp
product.

I think that that concern's understandable, but I don't know
that making a developer jump through hoops to satisfy all of the
LGPL's requirements is in anyone's interest.


> TIA for any info.

I don't know if I answered your questions; I'd be interested
in hearing what other people think.

>
>  - Stoney
>
> --
> Stonewall Ballard
> stoney at sb.org           http://stoney.sb.org/
>
>


_______________________________________________
Openmcl-devel mailing list
Openmcl-devel at clozure.com
http://clozure.com/cgi-bin/mailman/listinfo/openmcl-devel