[Openmcl-devel] http servers and threads under heavy load; was: Good Video

[R. Matthew Emerson]

On Oct 17, 2009, at 4:45 PM, Daniel Pezely wrote:

>
> Sometimes, however, OS-native threads are required for your particular
> needs.  In those cases, try lock-free, spin-free message lists within
> the pipeline.  It's no magic bullet, of course...  You'll want atomic-
> push and -pop operations (and ideally, atomic-rotatef-1), which in
> SBCL can be built upon #'sb-ext:compare-and-swap function.  Since CCL
> internally has #'ccl::atomic-incf and -decf, I'm sure that the other
> functions will follow soon or already exist by some other name.  I
> couldn't find any with a quick check of latest source in svn; anyone
> know?

There was a discussion on the #ccl irc channel on Freenode a couple of  
days ago about this.

The upshot of it is that there are several building blocks of this  
nature in ccl, but they're all undocumented and all somewhat  
idiosyncratic.

http://ccl.clozure.com/irc-logs/ccl/2009-10/ccl-2009.10.15.txt

01:52:37 <chandler> Is there an exposed CAS-style operation? (Forgive  
me if I'm missing something in the documentation.)
01:54:03 <gbyers> There are several, all funky and none (AFAIK)  
documented.  CLX wants a CONDITIONAL-STORE, and there's something  
that's slightly exposed for its benefit.
01:55:24 <sellout> chandler: Adlai was playing with a CAS earlier  
today, I think ...
01:55:48 <sellout> (for CCL)
01:56:16 <sykopomp> sellout: he's been working on it for a while. I  
don't know if he's gotten around to writing that latest version after  
he figured out what was wrong...
01:56:22 <chandler> Ah, I just found that in the logs.
01:56:42 <gbyers> The general case of CAS is weird for a few reasons  
(having to do with the GC, threads, and interrupts.)
01:56:58 <rme> Without special lisp runtime support, I don't think  
it's possible to do a CAS for lisp objects.
01:58:02 <chandler> Can you elucidate some of the reasoning behind that?
02:00:54 <gbyers> on x8664, "cmpxchg" uses %rax as an implicit  
operand: you generally load the contents of a memory location into  
%rax, then try to store a new value there, and the conditional store  
works if the location still contains the value in %rax.  If the GC  
runs between the load and the cmpxchg, the memory location's value may  
change but the value in %rax (presumed to be a non-lisp value) won't  
change.
02:01:24 <rme> In CCL, a gc can happen at any instruction boundary.   
CCL's gc thinks it always knows if a register contains a lisp  
object ... what he said.
02:04:44 <chandler> Ah. Having only dealt with this at a C level, I  
never noticed.
02:07:05 <chandler> Presumably some sort of thread state could be  
updated around the instruction, with a worst case that the old value  
is incorrectly marked as alive. (Or I suppose the GC could look back  
an instruction and handle cmpxchg cases specially.)
02:07:20 <gbyers> rme and I walk around saying "... but if a GC  
happens here, we're screwed."  It may be a curse.
02:07:32 <chandler> Heh.
02:07:34 <rme> For a good time (not) read pc_luser_xp() in ccl:lisp- 
kernel;x86-exceptions.c
02:10:34 <chandler> I'll have a look when I'm not IRCing from my phone.
02:11:00 <gbyers> We do the problematic cmpxchg's in a few known  
locations in the lisp kernel, and that function recognizes when a  
thread gets stopped in the middle of one of those sequences and backs  
up/moves forward as appropriate.
02:15:21 <gbyers> We also do the generational write-barrier stuff in  
software, so if a cmpxchg succeeds, it may have just stored a pointer  
to a young object in an old object.  We need to set a bit to indicate  
a possible intergenerational reference, and the store and the bit- 
setting have to appear to be atomic.