[Openmcl-devel] Running out of file descriptors with run-program

[Gary Byers]

On Tue, 25 Nov 2008, Daniel Dickison wrote:

> I have a program that uses run-program to call out to ImageMagick many
> times (on the order of hundreds of calls). Eventually, I run into a
> "Too many open files" error inside a call to (pipe) in run-program.
>
> I specify :wait t so it shouldn't leave FDs open after the call
> exits.  I don't specify an :input; however, I do specify stream
> objects for :output and :error, and my hunch is that FDs are left open
> when these two streams are different.  If I change *error-output*
> to :output below, then the error goes away:
>
>     (multiple-value-bind (status-key exit-code)
>         (ccl:external-process-status
>          (ccl:run-program "convert"
>                           (list (path "svg") (path "gif"))
>                           :wait t
>                           :output *verbose-output*
>                           :error *error-output*))
>       (assert (not (eq status-key :running)))
>       (unless (= exit-code 0)
>         (error "ImageMagick returned non-zero on exit: ~D" exit-
> code))))
>
> *verbose-output* is either bound to *standard-output* or a dead-end
> broadcast-stream (make-broadcast-stream), but the error happens in
> either case.  I can also reproduce the error running via SLIME or
> straight from the command line.
>
> Looking at the code for run-program and get-descriptor-for, it looks
> like only one of the stdout and stderr pipes get monitored, so one of
> them might be left dangling.  Does that sound like a plausible
> explanation?
>

We have a ticket (<http://trac.clozure.com/openmcl/ticket/318>) only
monitors one of :output/:error when they're distinct.  That's bad
enough (in that some output can disappear), but I think that you're
right in noting that we'll create two pipes in that case and only
ever monitor (and therefore only ever close) one of them.  (I've been
working on fixing that ... very intermittently.)

> Any help would be appreciated, whether it's a fix, instructions on how
> to fix, or a quick workaround.

One workaround in this case would be to to use:

     :output *verbose-output*
     :error t

That'd mean that the subprocess writes error output directly to lisp's
fd 1.  (In SLIME, that might show up in the *INFERIOR-LISP* buffer.)
The subprocess can write directly to that fd, and we shouldn't need
to create a pipe to act as an intermediary and risk neglecting to close
the :output pipe because of the bug.




>
> Daniel
>
>
>
> CL-USER> (lisp-implementation-version)
> "Version 1.2-r10552  (DarwinPPC32)"
> CL-USER> (problem-generator:generate-problems)
> ;;...snip...
> > Error: Too many open files
> > While executing: CCL::PIPE, in process listener(1).
> > Type :GO to continue, :POP to abort, :R for a list of available
> restarts.
> > If continued: Skip evaluation of (problem-generator:generate-
> problems :n 30)
> > Type :? for other options.
> 1 > :b
>  (F019AB90) : 1 (PIPE) 540
>  (F019ABB0) : 3 (GET-DESCRIPTOR-FOR #<SYNONYM-STREAM to *TERMINAL-IO*
> #x83E9EBE> #<EXTERNAL-PROCESS (convert /Volumes/D1/SVN Tree/trunk/
> research/PSLC/trunk/Base-height recog YK 2009/src/recognition/images/
> svg/yk-parallelogram-ir-b120h30t120o-90r0.svg /Volumes/D1/SVN Tree/
> trunk/research/PSLC/trunk/Base-height recog YK 2009/src/recognition/
> images/gif/yk-parallelogram-ir-b120h30t120o-90r0.gif)[NIL] (RUNNING)
> #x84872C6> '(254 4) '(253 4) [...]) 2136
>  (F019ABD0) : 5 (RUN-PROGRAM "convert" '("convert" "/Volumes/D1/SVN
> Tree/trunk/research/PSLC/trunk/Base-height recog YK 2009/src/
> recognition/images/svg/yk-parallelogram-ir-b120h30t120o-90r0.svg" "/
> Volumes/D1/SVN Tree/trunk/research/PSLC/trunk/Base-height recog YK
> 2009/src/recognition/images/gif/yk-parallelogram-ir-
> b120h30t120o-90r0.gif") [...]) 1176
>  (F019ABF0) : 6 (CONVERT-SVG-TO-GIF #S(FIGURE-
> SVG:FIGURE :SHAPE :PARALLELOGRAM :HIGHLIGHTS (:BASE-
> RIGHT :RIGHT) ...)) 220
>  (F019AC00) : 7 (SAVE-PROBLEM #S(PROBLEM-GENERATOR:PROBLEM :NUMBER
> 13 :SHAPE :PARALLELOGRAM ...) [...]) 200
>  (F019AC10) : 8 (GENERATE-PROBLEMS [...]) 668
>  (F019AC30) : 10 (CALL-CHECK-REGS 'PROBLEM-GENERATOR:GENERATE-
> PROBLEMS [...]) 72
>  (F019AC50) : 12 (FUNCALL #'#<(:INTERNAL CCL::EVAL-STRING
> CCL::STARTUP-CCL)> "(problem-generator:generate-problems :n 30)") 272
>  (F019AC70) : 13 (STARTUP-CCL [...]) 1024
>  (F019AC90) : 14 (FUNCALL #'#<(:INTERNAL (TOPLEVEL-FUNCTION
> (CCL::LISP-DEVELOPMENT-SYSTEM T)))>) 56
>  (F019ACB0) : 16 (FUNCALL #'#<(:INTERNAL CCL::MAKE-MCL-LISTENER-
> PROCESS)>) 384
>  (F019ACE0) : 18 (RUN-PROCESS-INITIAL-FORM #<TTY-LISTENER listener(1)
> [Active] #x83E9B4E> '(#)) 404
>  (F019AD20) : 20 (FUNCALL #'#<(:INTERNAL CCL::%PROCESS-PRESET-
> INTERNAL)> #<TTY-LISTENER listener(1) [Active] #x83E9B4E> '(#)) 164
>  (F019AD40) : 21 (FUNCALL #'#<(:INTERNAL CCL::THREAD-MAKE-STARTUP-
> FUNCTION)>) 172
> 1 > *error-output*
> #<SYNONYM-STREAM to *TERMINAL-IO* #x83E9EBE>
> 1 > problem-generator::*verbose-output*
> #<SYNONYM-STREAM to *TERMINAL-IO* #x83E9EDE>
> 1 >
>
> _______________________________________________
> Openmcl-devel mailing list
> Openmcl-devel at clozure.com
> http://clozure.com/mailman/listinfo/openmcl-devel
>
>