[Openmcl-devel] How to properly close the streams created by run-program?

[Gary Byers]

It's not clear that you're neglecting to close a stream, but it seems
that RUN-PROGRAM is leaking a file descriptor in this case.

The EXTERNAL-PROCESS that RUN-PROGRAM creates needs to have its
standard input/output/error connected to real file descriptors; in a
lot of cases, the file descriptors that the child sees refer to
temporary files or pipes that may have lisp streams on the other end
of the connection (or /dev/null, or ...).  RUN-PROGRAM calls
CCL::GET-DESCRIPTOR-FOR to obtain such a file descriptor for each of
the child process's standard I/O descriptors, and CCL::GET-DESCRIPTOR-FOR
is supposed to return both the fd and an indication of whether or not
the parent process should close that fd (fds that GET-DESCRIPTOR-FOR
creates are -usually- pushed on a list of fds to be closed in the parent,
and the augmented list is returned as an additional return value.)

In the case where the child should use an existing lisp stream that's
attached to a file descriptor, GET-DESCRIPTOR-FOR can just duplicate
the stream's file descriptor.  (It might also want to check that the
fd in question is open in a mode that makes sense, but it doesn't do
so.)  In this case, the duplicate fd isn't pushed on the "close in
parent" list, and that fd is never closed.

The code in question is the FD-STREAM clause in the big ETYPECASE
in CCL::GET-DESCRIPTOR-FOR:

     (fd-stream
        (values (fd-dup (ioblock-device (stream-ioblock object)))
 	       nil
 	       close-in-parent
 	       close-on-error))

I -believe- that it'd be correct to change that to:

     (fd-stream
        (let* ((fd (fd-dup (ioblock-device (stream-ioblock object)))))
          (values fd
 	         nil
 	         (cons fd close-in-parent)
 	         close-on-error)))

That seems to stop the leak in your example and I can't think of a
reason -not- to do that; some other nearby clauses seem to exhibit
the same problem.  (I suspect that the calls to FD-DUP were introduced
at some point for some possibly good reason, but once the parent has
passed the duplicate fd to the child the duplicate should be closed.)






On Wed, 19 Apr 2006, Stefan Mandl wrote:

> Hi out there,
>
> I'm trying to roll my own little "pipe" command.
>
> This is the source code:
>
> ==================================
>
> (defparameter newline-string (format nil "~%"))
>
> (defun read-all (str)
>   (let (res)
>     (do ((line (read-line str nil 'eof)
> 	       (read-line str nil 'eof)))
> 	 ((eql line 'eof))
> 	(setf res (cons (concatenate 'string line newline-string) res)))
>     (setf res (nreverse res))
>     (apply #'concatenate (cons 'string res))))
>
>
> (defun pipe (prg1 args1 prg2 args2 input)
>   (with-input-from-string (inp input)
>     (let* ((p1 (run-program prg1 args1 :output :stream :input inp))
> 	   (p2 (run-program prg2 args2 :output :stream :input
> (external-process-output-stream p1))))
>
>       (close (external-process-output-stream p1))
>       (when (external-process-error-stream p1)
> 	  (close (external-process-error-stream p1)))
>       (when (external-process-input-stream p1)
> 	(close (external-process-input-stream p1)))
>
>       (let ((res (read-all (external-process-output-stream p2))))
> 	(close (external-process-output-stream p2))
>       (when (external-process-error-stream p2)
> 	  (close (external-process-error-stream p2)))
>       (when (external-process-input-stream p2)
> 	(close (external-process-input-stream p2)))
>
>       res))))
> ======================
>
> when I call it in the Terminal like this (actually discarding the output):
>
> (dotimes (i 1000) (pipe "/bin/ls" '() "/bin/cat" '() ""))
>
> I get:
> > Error in process listener(1): Too many open files
> > While executing: "Unknown"
> > Type :POP to abort.
> Type :? for other options.
>
> Do you have any idea, which streams I forgot to close?
>
> I'm running openmcl 1.0 on OS X 10.4.6
>
> Thanks for any help!
>
> Stefan Mandl
> _______________________________________________
> Openmcl-devel mailing list
> Openmcl-devel at clozure.com
> http://clozure.com/mailman/listinfo/openmcl-devel
>
>