[Openmcl-devel] RUN-PROGRAM and CPU usage

[Gary Byers]

On Sun, 30 May 2004, Tord Kallqvist Romstad wrote:

> Hello,
>
> I am the author of Gothmog, one of the strongest free chess engines
> available for OS X.  The engine itself is a console application
> written in C.  I have now started writing a Lisp library to help me in
> the development of the engine, by automating tasks like running
> matches between different versions, running test suites and doing
> statistical analysis of the results.  If the OpenMCL Cocoa bindings
> are sufficiently mature, I also hope to eventually use my library as a
> basis for a Cocoa GUI for my chess engine.
>
> My first problem is to make the communication between OpenMCL and the
> engine work.  I thought this would be quite simple, and that the
> RUN-PROGRAM function would do the trick:
>
> (defun start-engine ()
>   (run-program *engine-path* nil
>                :output :stream
>                :input :stream
>                :wait nil))
>
> This works fine, except for the problem that OpenMCL's CPU usage
> immediately jumps through the roof.  Is there any way to prevent this
> from happening?  If not, is there something else than RUN-PROGRAM
> which is likely to work better?
>

The problem isn't really RUN-PROGRAM as such; it's the mechanism used
to wait for I/O and for process termination.  The obvious solution is
to fire off a thread that watches the external process via some mixture
of #_select and some variant of #_wait.

The mechanism that OpenMCL actually uses isn't quite as reasonable, but
shouldn't consume a whole lot of CPU when nothing's happening: a few times
a second, a thread that ordinarily spends most of its time sleeping is
supposed to wake up and perform "periodic tasks": these are supposed to
be little short-lived functions (like forcing output on interactive streams),
and there's a periodic task that's supposed to watch for I/O and process
termination on anything created by RUN-PROGRAM that might still be active.

Unfortunately, rather than checking for activity on external-processes
and going back to sleep, the "periodic task" loops until all external
processes die and all I/O to and from them is complete.  Not only does
this send CPU utilization through the roof, it has the negative effect
of preventing other "periodic tasks" (like the one which tries to force
output on interactive streams) from running.  (If you could type anything
while the external-process task was hysterically polling, you might find
that output doesn't get flushed automatically.)

The loop around the body of the function (installed at the beginning of
"ccl:level-1;l1-boot-3.lisp") is probably there for historical reasons;
I don't remember when polling so hysterically was ever a good idea.  If
that LOOP is removed:

(defloadvar *external-process-task*
    (%install-periodic-task 'external-process-watchdog
			    #'(lambda ()
				(watch-file-descriptors)
				(check-all-pids))
			    50))

we're left with the other problem associated with polling: it may be a
little while (half a second or more) between the time input arrives and
the time something wakes up and notices.

That all suggests that it's time to rewrite this to let other threads
block (via #_select/#_wait*) and abandon polling. Until that happens,
it'd probably be better to poll a few times a second (and not wear
out the fans on laptops, etc.) and complain about the high latency
instead ...





> --
> Tord Romstad
>
> _______________________________________________
> Openmcl-devel mailing list
> Openmcl-devel at clozure.com
> http://clozure.com/mailman/listinfo/openmcl-devel
>
>