[Openmcl-devel] process-input-wait question

[Erik Pearson]

Hi Gary,

Thanks as always for the detailed explanation. I seem to have a bit  
of deja vu ... probably asked a similar question a year or two ago.

I need the timeout feature of fd-input-available, although plain ol'  
read-byte works fine otherwise. This is for  keepalive connections  
for a web server, where I need to kill off the connections for  
browsers that don't play nice and drop the connections themselves. A  
similar thing could be done with a thread that wakes up and polls  
open connections looking for old unused ones.

Thanks,
Erik.


On Jul 13, 2006, at 1:58 AM, Gary Byers wrote:

>
>
> On Thu, 13 Jul 2006, Erik Pearson wrote:
>
>> I've been looking at the implementation code for process-input-wait
>> and fd-input-available, and was  wondering if Gary or anyone could
>> address a couple of questions?
>>
>> First, why does the underlying "select()" call in fd-input-available
>> utilizes the exception argument (errfds in the lisp, but exceptfds in
>> the man page for select).  The purpose of exceptfds is to detect, on
>> sockets (for which I'm using this), the availability of out-of-band
>> data. The naming of the lisp variable "errfds" might lead one to
>> believe that the authors at some point believed it was an "exception"
>> as in "error" rather than "exception"  as in "interesting" or
>> "different" as it seems to mean in this context.
>>
>> Other than this not being documented -- is the inclusion of the
>> exceptfds test desirable? I don't know if the openmcl code makes room
>> for oob shenanigans, which I don't really understand. I don't think
>> that oob is part of the http spec as far as my present application,
>> so I'd rather be able to ignore it completely.
>>
>
> A lot of internet protocols (not sure about HTTP, but I suspect that
> it also falls into this category) sort of inherit from the Telnet
> protocol.  (In theory, this'd mean that you could try to do Telnet
> option negotiation when talking to an FTP control connection or
> maybe a web server; in practice, it kind of means "assume 7-bit
> ASCII and CRLF-terminated lines, but be kind of flexible about that.")
>
> The full-blown Telnet protocol does indeed make use of
> urgent/out-of-band data, and I have seen it used in Telnet servers
> and clients (typically to indicate a pending "interrupt process"
> or something like that.)  10 years or so ago, it was possible to
> blue-screen Windows NT by sending it urgent TCP data, and it was
> probably likely that most of the urgent/out-of-band TCP packets
> on the entire net were being sent by script-kiddies.
>
> Fascinating as that might be, something makes me think that
> the handling of "errfds/exceptfds" has something to do with vxWorks
> (if it wasn't just a misreading of a man page); to actually read
> urgent/out-of-band data, you'd have to jump through a lot of
> additional not-to-stream-like hoops to actually read it.
>
>
>> I've modified the code a bit to return whether this condition has
>> been detected when process-input-wait returns, and sometimes it is
>> and sometimes it isn't (this is on a connection from a web browser to
>> a lisp web server). I'm not sure I have the desire to figure out why.
>
>
> What you may see fairly often is the case where the #_select call
> returns with #$EINTR; threads get interrupted (by the GC, among other
> things) and I'm not sure if #_select gets restarted automatically
>
>>
>> Another process-input-wait question has to do with the initial call
>> to fd-input-available with a ticks argument of 0, and the subsequent
>> calling of fd-input-available with the timeout ticks, the testing of
>> ticks, etc. This looks like a little bit of polling -- but isn't
>> select with the timeout value supposed to work as expected -- to
>> return as soon as the selected file descriptors are available, or the
>> time has passed, and isn't that just what we want? Perhaps this is to
>> work around some bugs, or fine points in select logic that I don't
>> understand?
>
> It may just be an attempt to avoid the call to GET-TICK-COUNT; in
> some incarnations, GET-TICK-COUNT consed a small bignum.
>
> You're certainly right, though, in observing that a lot of this
> is leftover from the pre-native threads days. (PROCESS-INPUT-WAIT
> would invoke the lisp scheduler if input wasn't immediately
> available, and aside from avoiding GET-TICK-COUNT overhead the
>
>        (when (fd-input-available-p fd 0)
> 	 (return t))
>
> may have been trying to stay out of that complexity.
>
> PROCESS-INPUT-WAIT is pretty much an anachronism; the easiest way to
> wait for input (if necessary) on an fd and then read it is to
> do a blocking #_read from that fd.
>
> Doing a blocking read (or a blocking anything) was pretty much
> forbidden under the cooperative lisp scheduler, and that led to
> things like PROCESS-INPUT-WAIT. It also led to things like
> *MULTIPROCESSING-SOCKET-IO*, which (when true at socket-creation
> time) causes newly-created sockets to be non-blocking.  There
> may still be cases where non-blocking sockets would be useful,
> but I'm not sure that those cases are all that frequent.
>
> Most of what happened to the I/O code when native threads were
> introduced is that timeouts were made infinite (I think that
> all of the actual calls to PROCESS-INPUT-WAIT in that code
> use a NULL timeout.)
>
> 	(process-input-wait fd nil)
>         (fd-read fd ...)	; e.g., (#_read fd ...)
>
> doesn't poll or busy-wait or anything like that, but it doesn't
> do anything useful that the #_read doesn't do, and it involves
> some unnecessary overhead  (an extra syscall and some FF-CALLs
> to zero out/set FD bits.)
>
>
>
>>
>> defun process-input-wait (fd &optional ticks)
>>   "Wait until input is available on a given file-descriptor."
>>   (let* ((wait-end (if ticks (+ (get-tick-count) ticks))))
>>     (loop
>>        (when (fd-input-available-p fd 0)
>> 	 (return t))
>>        (let* ((now (if ticks (get-tick-count))))
>> 	 (if (and wait-end (>= now wait-end))
>> 	     (return))
>> 	 (fd-input-available-p fd (if ticks (- wait-end now)))))))
>>
>>
>>  Is there any reason that this won't work?
>>
>> (defun process-input-wait (fd &optional ticks)
>>   "Wait until input is available on a given file-descriptor."
>>   (fd-input-available-p fd ticks))
>>
>
> Certainly a lot clearer.
>
> IIRC, PROCESS-INPUT-WAIT was once something like:
>
> (process-wait "waiting for input" #'(lambda () (fd-input-available- 
> p fd 0)))
>
> That eventually evolved into something that tried to interact with the
> lisp scheduler a little better (if all threads were waiting for I/O,
> the lisp scheduler could sleep in #_select on all of them.)
>
> In its current use, it's basically just "block until input is  
> available,
> and do some of that in a funny way for historical reasons."
>
>
>> Thanks,
>> Erik.
>> _______________________________________________
>> Openmcl-devel mailing list
>> Openmcl-devel at clozure.com
>> http://clozure.com/mailman/listinfo/openmcl-devel
>>
>>