[Openmcl-devel] Speed, compilers and multi-core processors

[mikel evins]

On May 19, 2009, at 1:12 PM, Dan Weinreb wrote:

>
>
> Alexander Repenning wrote:
>> not so fast ;-)
>>
>> The "how can we make use of multiple cores" is currently on the
>> the hottest funding topics supported by NSF, DOE, Microsoft, .....
>>
> For some applications, it's very easy to take advantage of all those
> cores.  The airline reservation system we're building at ITA
> has its "core" (business tier) model written in Common Lisp.
> We get lots and lots of requests coming in, which are distributed
> to a load balancer to a lot (maybe 20?) of boxes, each of
> which has 8 cores and 32GB of memory.  On each one,
> we run 8 CCL processes, and each one has a single
> request-handling thread.  Voila, all the cores are
> being used.
>
> But, of course, you're talking about situations where a single
> program wants to use lots of cores.
>
> The approach being taken by many new languages is
> (1) do as much as possible with no side effects, so
> that nobody is writing anything, and therefore
> no two threads have conflicts, and (2) when you do
> have two threads reading and/or writing the same
> memory, use transactions.
>
> Guy Steele said that it's quite amazing how many new
> languages work this way.  His own new language,
> Fortress is one of them, and he named several
> others.  Anders Hejlsberg, the C# inventor at
> Microsoft, is also talking about this approach (I've
> seen him discuss it in two videos that I found on
> the web).
>
> And you know what other language works this way?
> Clojure, a very promising new dialect of Lisp.

Yup. It works pretty well. I've used it a lot for about eight months.

You can't get something that works as well simply by adding Clojure's  
concurrency features to Common Lisp. Clojure *forbids* simple  
assignment; you can't do it. There are four data types that support  
assignment, and none of them supports simple assignment. All four use  
transactional assignment that is safe under concurrent use, but  
significantly more expensive than simple assignment. Clojure's basic  
data structures are also different from Common Lisp's, despite looking  
familiar. Its List, Vector, and Map types, for example, look like CL's  
lists, vectors, and hashtables, but are fundamentally different data  
structures under the covers, in order to support both immutability and  
efficient updates. (They are implemented using Bagwell's ideal hash  
tries, which provide an efficient way to implement "inserts" as newly- 
created data structures that refer to unaltered old ones).

Clojure forbids simple assignment because all it takes to destroy its  
safety guarantees under concurrency is a single SETQ in the wrong  
context. Presumably, this is why some of the approaches it uses, such  
as Software Transactional Memory, are widely used in languages like  
Haskell and not widely used in languages like C and Common Lisp:  
because all your safety guarantees go out the window in the presence  
of simple assignment. Clojure and Haskell solve that problem by making  
simple assignment impossible. (Actually, Clojure doesn't; because it  
runs on the JVM and provides a seamless integration with Java, you can  
still do simple assignment using the Java features. If you do that,  
though All Bets Are Off).

--me