[Openmcl-devel] type specifier '(simple-vector n) in defmethod

[Ron Garret]

> On Jan 6, 2024, at 11:12 AM, Nicolas Martyanoff <nicolas at n16f.net> wrote:
> 
> Ron Garret <ron at flownet.com> writes:
> 
>>> On Jan 6, 2024, at 6:00 AM, Nicolas Martyanoff <nicolas at n16f.net> wrote:
>>> 
>>> In CL, this really does not map well to streams because streams are
>>> fundamentally blocking (see for example the semantic of READ-SEQUENCE).
>> 
>> That's not true. CL:READ-SEQUENCE is blocking, but LISTEN works on
>> binary streams so implementing a non-blocking version is an elementary
>> exercise.
> 
> A function similar to LISTEN would not help in any way.
> 
> OS primitives dealing with non-blocking IO are event based: you do not
> ask if you can read (or write) a file descriptor, you register it
> against a some kind of multiplexer (e.g. an epoll instance), then
> another primitive lets you ask this multiplexer which events were
> signaled for any registered file descriptor. You then dispatch these
> events manually which usually mean running the appropriate callback.
> 
> The approach is fundamentally top-down: you start from IO events, then
> dispatch. But a read event only means you can perform one single
> non-blocking read call. So you need to guarantee that any stream
> function reading your file descriptor will perform one read and one read
> only (if you do a second one, you'll potentially block with all the
> infortunate consequences for your event loop). You cannot guarantee that
> (heck you cannot even know if standard streams are buffered or not, and
> don't get me started on external formats), and various stream operations
> are fundamentally based on the idea of reading repeatedly
> (READ-SEQUENCE, READ-LINE, etc.). So you are forced to bypass streams,
> and use your FFI to call read() on your own and implement buffering
> yourself. Similar issue when writing.
> 
> I work around this problem by implementing my own TCP streams using the
> (non-standard) Gray stream API. In non-blocking mode, I signal a
> condition indicating that I could not complete the operation because
> doing so would require additional calls to read(). This way the IO mux
> knows which events must be watched for. But then we're back writing all
> parsers as operating on IO buffers instead of just reading streams, and
> you still end up juggling with multiple callbacks for processing which
> results in much more complex code.
> 
> These problems do not exist in Go or in Erlang: you can use a
> stream-like system and just read the file descriptor. The runtime
> handles IO in a non-blocking way and assigns another green thread for
> execution until the read call can continue; when it happens, it schedule
> the initial green thread for execution, and the read call returns. You
> simply cannot emulate that in CL unless your implementation provides the
> correct runtime support.

I think I need to clarify something.  Recall the statement I was responding to:

> In CL, this really does not map well to streams because streams are fundamentally blocking (see for example the semantic of READ-SEQUENCE).  [Emphasis added.]

where "this" is something along the lines of: writing performant I/O-bound code that has to handle a lot of asynchronous events, the sort of thing Go and Erlang handle well.

The thing I was taking issue with was not that "does not map well to streams" but rather the "because streams are fundamentally blocking."  It's not the claim I'm disputing, it's the explanation.  (Well, I actually dispute the claim too, but that's a different discussion.  But I don't dispute that Go and Erlang are better at "this" than (current implementations of) CL, so this is probably a moot point.)

The reason "this doesn't map well onto streams" is not because "streams are fundamentally blocking" in the sense that read-sequence is blocking.  It is possible to write a non-blocking version of read-sequence in standard CL.  It won't be particularly efficient, but you can do it, so there is nothing "fundamental" about the fact that read-sequence-no-hang is not part of standard Common Lisp.

The reason that CL is not as well suited as Go and Erlang to writing performant I/O-bound code that has to handle a lot of asynchronous events is because Go and Erlang were specifically designed to handle that problem and CL wasn't.  CL was designed long before large-scale web servers were a thing.  The fact that you can use the language to do this kind of work *at all* is remarkable, a testament to the foresight of its designers.  (Can you imagine writing a web server in Fortran or Cobol?)  Go and Erlang were designed from the ground up to address this specific problem, and so *of course* they are going to be better at it than any language that was not specifically designed for it.  But the fact that Go and Erlang are better at it has nothing to do with CL streams being "fundamentally blocking".

If you want Go/Erlang-style I/O in CL you can have it.  The solutions to the problem implemented in Go and Erlang can be implemented in CL.  This obviously requires language extensions because the standard base language doesn't even have threads/processes at all, but open-source implementations of CL are almost infinitely malleable.  Any feature that you find in any other language can be integrated into CL with varying degrees of effort.  The biggest challenge is that AFAIK no modern CL implementation uses green threads, so you'd need to implement those, and that's non-trivial at least if efficiency is a concern.  But it's not impossible.  Most CL implementations used green threads back in the days before pre-emptive multitaksing became widely available, so this is a well-worn path.  It's just a question of how much work you're willing to do (or how big a check you're willing to write).

rg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clozure.com/pipermail/openmcl-devel/attachments/20240106/f75c1ec6/attachment.htm>