[Openmcl-devel] process-input-wait question

[Gary Byers]

On Thu, 13 Jul 2006, Erik Pearson wrote:

> I've been looking at the implementation code for process-input-wait
> and fd-input-available, and was  wondering if Gary or anyone could
> address a couple of questions?
>
> First, why does the underlying "select()" call in fd-input-available
> utilizes the exception argument (errfds in the lisp, but exceptfds in
> the man page for select).  The purpose of exceptfds is to detect, on
> sockets (for which I'm using this), the availability of out-of-band
> data. The naming of the lisp variable "errfds" might lead one to
> believe that the authors at some point believed it was an "exception"
> as in "error" rather than "exception"  as in "interesting" or
> "different" as it seems to mean in this context.
>
> Other than this not being documented -- is the inclusion of the
> exceptfds test desirable? I don't know if the openmcl code makes room
> for oob shenanigans, which I don't really understand. I don't think
> that oob is part of the http spec as far as my present application,
> so I'd rather be able to ignore it completely.
>

A lot of internet protocols (not sure about HTTP, but I suspect that
it also falls into this category) sort of inherit from the Telnet
protocol.  (In theory, this'd mean that you could try to do Telnet
option negotiation when talking to an FTP control connection or
maybe a web server; in practice, it kind of means "assume 7-bit
ASCII and CRLF-terminated lines, but be kind of flexible about that.")

The full-blown Telnet protocol does indeed make use of
urgent/out-of-band data, and I have seen it used in Telnet servers
and clients (typically to indicate a pending "interrupt process"
or something like that.)  10 years or so ago, it was possible to
blue-screen Windows NT by sending it urgent TCP data, and it was
probably likely that most of the urgent/out-of-band TCP packets
on the entire net were being sent by script-kiddies.

Fascinating as that might be, something makes me think that
the handling of "errfds/exceptfds" has something to do with vxWorks
(if it wasn't just a misreading of a man page); to actually read
urgent/out-of-band data, you'd have to jump through a lot of
additional not-to-stream-like hoops to actually read it.


> I've modified the code a bit to return whether this condition has
> been detected when process-input-wait returns, and sometimes it is
> and sometimes it isn't (this is on a connection from a web browser to
> a lisp web server). I'm not sure I have the desire to figure out why.


What you may see fairly often is the case where the #_select call
returns with #$EINTR; threads get interrupted (by the GC, among other
things) and I'm not sure if #_select gets restarted automatically

>
> Another process-input-wait question has to do with the initial call
> to fd-input-available with a ticks argument of 0, and the subsequent
> calling of fd-input-available with the timeout ticks, the testing of
> ticks, etc. This looks like a little bit of polling -- but isn't
> select with the timeout value supposed to work as expected -- to
> return as soon as the selected file descriptors are available, or the
> time has passed, and isn't that just what we want? Perhaps this is to
> work around some bugs, or fine points in select logic that I don't
> understand?

It may just be an attempt to avoid the call to GET-TICK-COUNT; in
some incarnations, GET-TICK-COUNT consed a small bignum.

You're certainly right, though, in observing that a lot of this
is leftover from the pre-native threads days. (PROCESS-INPUT-WAIT
would invoke the lisp scheduler if input wasn't immediately
available, and aside from avoiding GET-TICK-COUNT overhead the

        (when (fd-input-available-p fd 0)
 	 (return t))

may have been trying to stay out of that complexity.

PROCESS-INPUT-WAIT is pretty much an anachronism; the easiest way to
wait for input (if necessary) on an fd and then read it is to
do a blocking #_read from that fd.

Doing a blocking read (or a blocking anything) was pretty much
forbidden under the cooperative lisp scheduler, and that led to
things like PROCESS-INPUT-WAIT. It also led to things like
*MULTIPROCESSING-SOCKET-IO*, which (when true at socket-creation
time) causes newly-created sockets to be non-blocking.  There
may still be cases where non-blocking sockets would be useful,
but I'm not sure that those cases are all that frequent.

Most of what happened to the I/O code when native threads were
introduced is that timeouts were made infinite (I think that
all of the actual calls to PROCESS-INPUT-WAIT in that code
use a NULL timeout.)

 	(process-input-wait fd nil)
         (fd-read fd ...)	; e.g., (#_read fd ...)

doesn't poll or busy-wait or anything like that, but it doesn't
do anything useful that the #_read doesn't do, and it involves
some unnecessary overhead  (an extra syscall and some FF-CALLs
to zero out/set FD bits.)



>
> defun process-input-wait (fd &optional ticks)
>   "Wait until input is available on a given file-descriptor."
>   (let* ((wait-end (if ticks (+ (get-tick-count) ticks))))
>     (loop
>        (when (fd-input-available-p fd 0)
> 	 (return t))
>        (let* ((now (if ticks (get-tick-count))))
> 	 (if (and wait-end (>= now wait-end))
> 	     (return))
> 	 (fd-input-available-p fd (if ticks (- wait-end now)))))))
>
>
>  Is there any reason that this won't work?
>
> (defun process-input-wait (fd &optional ticks)
>   "Wait until input is available on a given file-descriptor."
>   (fd-input-available-p fd ticks))
>

Certainly a lot clearer.

IIRC, PROCESS-INPUT-WAIT was once something like:

(process-wait "waiting for input" #'(lambda () (fd-input-available-p fd 0)))

That eventually evolved into something that tried to interact with the
lisp scheduler a little better (if all threads were waiting for I/O,
the lisp scheduler could sleep in #_select on all of them.)

In its current use, it's basically just "block until input is available,
and do some of that in a funny way for historical reasons."


> Thanks,
> Erik.
> _______________________________________________
> Openmcl-devel mailing list
> Openmcl-devel at clozure.com
> http://clozure.com/mailman/listinfo/openmcl-devel
>
>