[Openmcl-devel] Running out of file descriptors with run-program

[Daniel Dickison]

I have a program that uses run-program to call out to ImageMagick many  
times (on the order of hundreds of calls). Eventually, I run into a  
"Too many open files" error inside a call to (pipe) in run-program.

I specify :wait t so it shouldn't leave FDs open after the call  
exits.  I don't specify an :input; however, I do specify stream  
objects for :output and :error, and my hunch is that FDs are left open  
when these two streams are different.  If I change *error-output*  
to :output below, then the error goes away:

     (multiple-value-bind (status-key exit-code)
         (ccl:external-process-status
          (ccl:run-program "convert"
                           (list (path "svg") (path "gif"))
                           :wait t
                           :output *verbose-output*
                           :error *error-output*))
       (assert (not (eq status-key :running)))
       (unless (= exit-code 0)
         (error "ImageMagick returned non-zero on exit: ~D" exit- 
code))))

*verbose-output* is either bound to *standard-output* or a dead-end  
broadcast-stream (make-broadcast-stream), but the error happens in  
either case.  I can also reproduce the error running via SLIME or  
straight from the command line.

Looking at the code for run-program and get-descriptor-for, it looks  
like only one of the stdout and stderr pipes get monitored, so one of  
them might be left dangling.  Does that sound like a plausible  
explanation?

Any help would be appreciated, whether it's a fix, instructions on how  
to fix, or a quick workaround.

Daniel



CL-USER> (lisp-implementation-version)
"Version 1.2-r10552  (DarwinPPC32)"
CL-USER> (problem-generator:generate-problems)
;;...snip...
 > Error: Too many open files
 > While executing: CCL::PIPE, in process listener(1).
 > Type :GO to continue, :POP to abort, :R for a list of available  
restarts.
 > If continued: Skip evaluation of (problem-generator:generate- 
problems :n 30)
 > Type :? for other options.
1 > :b
  (F019AB90) : 1 (PIPE) 540
  (F019ABB0) : 3 (GET-DESCRIPTOR-FOR #<SYNONYM-STREAM to *TERMINAL-IO*  
#x83E9EBE> #<EXTERNAL-PROCESS (convert /Volumes/D1/SVN Tree/trunk/ 
research/PSLC/trunk/Base-height recog YK 2009/src/recognition/images/ 
svg/yk-parallelogram-ir-b120h30t120o-90r0.svg /Volumes/D1/SVN Tree/ 
trunk/research/PSLC/trunk/Base-height recog YK 2009/src/recognition/ 
images/gif/yk-parallelogram-ir-b120h30t120o-90r0.gif)[NIL] (RUNNING)  
#x84872C6> '(254 4) '(253 4) [...]) 2136
  (F019ABD0) : 5 (RUN-PROGRAM "convert" '("convert" "/Volumes/D1/SVN  
Tree/trunk/research/PSLC/trunk/Base-height recog YK 2009/src/ 
recognition/images/svg/yk-parallelogram-ir-b120h30t120o-90r0.svg" "/ 
Volumes/D1/SVN Tree/trunk/research/PSLC/trunk/Base-height recog YK  
2009/src/recognition/images/gif/yk-parallelogram-ir- 
b120h30t120o-90r0.gif") [...]) 1176
  (F019ABF0) : 6 (CONVERT-SVG-TO-GIF #S(FIGURE- 
SVG:FIGURE :SHAPE :PARALLELOGRAM :HIGHLIGHTS (:BASE- 
RIGHT :RIGHT) ...)) 220
  (F019AC00) : 7 (SAVE-PROBLEM #S(PROBLEM-GENERATOR:PROBLEM :NUMBER  
13 :SHAPE :PARALLELOGRAM ...) [...]) 200
  (F019AC10) : 8 (GENERATE-PROBLEMS [...]) 668
  (F019AC30) : 10 (CALL-CHECK-REGS 'PROBLEM-GENERATOR:GENERATE- 
PROBLEMS [...]) 72
  (F019AC50) : 12 (FUNCALL #'#<(:INTERNAL CCL::EVAL-STRING  
CCL::STARTUP-CCL)> "(problem-generator:generate-problems :n 30)") 272
  (F019AC70) : 13 (STARTUP-CCL [...]) 1024
  (F019AC90) : 14 (FUNCALL #'#<(:INTERNAL (TOPLEVEL-FUNCTION  
(CCL::LISP-DEVELOPMENT-SYSTEM T)))>) 56
  (F019ACB0) : 16 (FUNCALL #'#<(:INTERNAL CCL::MAKE-MCL-LISTENER- 
PROCESS)>) 384
  (F019ACE0) : 18 (RUN-PROCESS-INITIAL-FORM #<TTY-LISTENER listener(1)  
[Active] #x83E9B4E> '(#)) 404
  (F019AD20) : 20 (FUNCALL #'#<(:INTERNAL CCL::%PROCESS-PRESET- 
INTERNAL)> #<TTY-LISTENER listener(1) [Active] #x83E9B4E> '(#)) 164
  (F019AD40) : 21 (FUNCALL #'#<(:INTERNAL CCL::THREAD-MAKE-STARTUP- 
FUNCTION)>) 172
1 > *error-output*
#<SYNONYM-STREAM to *TERMINAL-IO* #x83E9EBE>
1 > problem-generator::*verbose-output*
#<SYNONYM-STREAM to *TERMINAL-IO* #x83E9EDE>
1 >