[Openmcl-devel] It's the world's most advanced (desktop) operating system!
Gary Byers
gb at clozure.com
Sun Sep 9 11:38:02 PDT 2012
I was always taught that if you can't say anything nice about an OS, it's
best not to say anything at all. It's certainly nice that the little test
case described here was easy to develop and seems to behave consistently,
and I'm sure that I could come up with other nice things to say about Mountain
Lion if I tried, though it may take a while.
Here's a little program that seems to demonstrate the "memory surge"
problem that Alex Repenning has described. It's been difficult to
reproduce that problem for many interested parties (apparently not for
Alex), though people have seen a similar problem with long delays on
thread creation under Mountain Lion (as described in
<http://trac.clozure.com/ccl/ticket/1005>. Those problems seemed to
only affect people running 64-bit CCL and Alex said that the problems
he's seen occur in both the 32-bit and 64-bit versions. I'm about as
sure as I can be that the problem that Alex has been seeing and ticket 1005
are the same bug and I think that I have a reliable way of reproducing it
(and a fix that I feel at least somewhat comfortable with.)
If anyone interested in trying this could do so, that'd be helpful. I
expect it to cause abnormal termination of CCL when run under Mountain
Lion and to not do so on earlier OSX relases, so it'd be interesting if
anyone gets unexpected results. I can't offer any reward for doing so
beyond the Schadenfreude that may come with the realization that Mountain
Lion's problems aren't limited to its near total lack of usability ...
To test:
0) Run a fairly recent 32-bit CCL; you don't need to run the IDE (and it
may be better to just run the command-line version to keep things simple.)
Running the command-line version under SLIME should also work, but if you
can stand to type directly to the REPL in a terminal window or Emacs shell
buffer that would also keep the number of variable factors to a minimum.
1) Load the attached file ("fill-lowmem.lisp"), then call the functions
that it defines. First:
a) Doing:
? (cl-user::fill-memory-holes #x0a000000)
That'll walk the address space of the process, finding unallocated memory
regions and mapping them as "in use, but neither readable nor writeable".
This process continues until the argument address (#x0a000000) is reached,
at which point the function returns the total of the sizes of the "holes"
that it's filled in. I generally get a result of around 32MB
b) One or more of the previously-allocated memory regions is managed by #_malloc,
and some of the memory in those regions is "free" as far as #_malloc is concerned.
The next step is to try to force #_malloc to see that memory as being in-use; this
can be achieved by doing:
? (cl-user::fill-malloc #x0a000000)
This repeatedly calls #_malloc with an argument that's the size of a data structure
created during thread creation, until #_malloc returns 10K consecutive pointers
whose addresses are > the argument (#x0a000000). That's supposed to ensure that
all "low" malloc-managed regions are full and that all subsequent calls to #_malloc
with this size argument will return addresses > #x0a000000. This function returns
the total size of all of the blocks of memory it allocates; I get a figure of around
8MB.
So, we've allocated a comparatively modest amount of memory (~40MB) and tried to
ensure that any calls to #_malloc that might occur during thread creation return
a pointer that's > #x0a000000.
Let's create some lisp threads, shall we ?
? (dotimes (i 100) (print i) (process-run-function "do you feel lucky?" (lambda ())))
I see a 0 printed, usually; then I see some indication that #_malloc has filled
all of the address space in the process and there isn't any more.
That may not be enough Schadenfreude for some of you. Here's more!
When CCL creates a lisp thread, it creates a data structure called a "Thread
Context Record" (TCR) which contains information about the thread. (On some
platforms, the TCR - like all thread-local variables - is created automatically;
Apple hasn't invented thread-local storage yet.) On OSX, we want the address of
a thread's TCR to serve double-duty as the "name" (32-bit identifier) of a Mach
message port that'll be used in the exception-handling mechanism. If 'tcr' is
a 32-bit pointer returned by #_malloc, then code (in the C function allocate_tcr(),
in ccl/lisp-kernel/thread-manager.c, IIRC) does something like:
(loop
(let* ((tcr (#_malloc ...))
(return-value (#_mach_port_allocate_name (#_mach_task_self)
#$MACH_PORT_RIGHT_RECEIVE
(%ptr-to-int tcr))))
(if (= return-value 0)
(return tcr)
(push tcr list-of-failed-candidates)))) ; will be freed later
If that (incomplete) pseudocode looks familiar, I've mentioned it far too often
in the last few weeks, both in discussing ticket 1005 and in discussing Alex's
"memory surge" problem. The expectation here is that the loop will almost always
exit on the first iteration, but that the call to #_mach_port_allocate_name may
fail occasionally if the address in question just happens to already be a randomly
assigned port name for some other Mach port.
On Mountain Lion, the call to #_mach_port_allocate_name will fail with error
code 3 (#$KERN_NO_SPACE) if the address is > ~#x09c00000 (which is what the
#x0a000000 in the stress-test above is all about.) If #_malloc always returns
such addresses, we just stay in that loop until we run out of memory; if it
could occasionally return an address that Mountain Lion likes (after returning
many addresses that Mount Lion doesn't like), thread creation could take a long
time and cause spikes in memory and CPU usage.
Schadenfreude is like a drug: one keeps wanting more and more. (Well,
that's true if the drug's worth what you paid for it, but that's
another story.) I haven't been too sure of the exact value at which
#_mach_port_allocate_name starts failing on Mountain Lion (it's
somewhere above #x09c00000 and #x0a000000), so I wrote a little
program to find that point via binary search. When it tried a certain
value (likely one near the failure point), CCL, the Emacs I was
running it under, and the shell I'd run Emacs from were all killed.
(I was logged in via SSH, and my window session on the machine - in
another room - was also terminated.) This seems to be 100% repeatable.
I'll leave the code as an exercise, but any sick twisted individual who
finds it amusing that The World's Most Advanced Operating System can be
DOS'ed by a 10-line CCL program is probably doubled over with laughter right
now.
Ahem. I was laughing -with- Mountain Lion, not at it. The rest of us
doubtless find this to be sobering news and may be wondering what can
be done so that we and our users aren't affected by this, and may want
a better answer than "hope that Mountain Lion goes away." What I can
do in the short term is to report the bug to Apple (I wasn't sure
whether this was a bug or simply an undocumented and ill-advised API
change, but API changes don't usually open the system to
denial-of-service attacks ...) and I can try to ensure that CCL avoids
the issue to the extent possible (though this may impose limits on the
number of concurrent threads that can be created; I'd rather that people
who try to create very large of threads see that fail for other reasons.)
I'll try to commit that to the trunk today.
We try to make a thread's TCR and alias for the Mach port on which the
thread's exception messages are received because it's necessary to
quickly map between the two (and the identity mapping is very quick.)
A better long-term workaround would be to use one or more explicit
mappings (some sort of hash-table mapping from port ID's to TCRs and
maybe in the other direction as well.) That's certainly doable, but
making it fast and thread-safe and reliable (and nothing can be faster,
safer, or more reliable than the identity mapping) is likely to require
some effort. If I knew for certain that the OS bug was going to be fixed
"soon", I'm not sure if that effort would be worthwhile.
The old saying - "Mach sucks, but no one understands how" - still elicits
a chuckle from me (though I can understand that it may have worn a bit thin
on many people.) Until a few years ago, I was fairly confident that there
were people at Apple who knew exactly how Mach sucked (had been making it
suck for years and were truly world-class experts on all of its quirks
and eccentricities.) I'm not confident that that's still true, and that's
disturbing.
Operating systems are very large and very complex, and this bug isn't the
worst OS bug that anyone's ever seen or will see. It's possible that this
bug could have been present during a long testing cycle and simply gone
undetected, but OSX releases don't go through long testing cycles anymore
(and Apple seems to be proud of this.) I'm sure that few programs besides
CCL call #_mach_port_allocate_name, but I suspect that the problems could
be a level or two below that.
A few paragraphs ago, I was doubled over with laughter and now I'm wallowing
in depression. I'm not really confident that things are going to get better
as far as OSX is concerned, but (at some level) I still wish that they would.
-------------- next part --------------
(in-package "CL-USER")
#-darwinx8632-target
(eval-when (:load-toplevel :execute)
(error "This code is darwin and x8632-specific"))
(defun fill-memory-holes (limit)
(rlet ((info :vm_region_basic_info_t)
(pnextaddr :vm_address_t)
(psize :vm_size_t)
(pinfo-size :mach_msg_type_number_t #$VM_REGION_BASIC_INFO_COUNT)
(pvm-object-name :mach_port_t))
(let* ((addr 0)
(total 0))
(loop
(when (> addr limit) (return total))
(setf (pref pnextaddr :vm_address_t) addr
(pref pinfo-size :mach_msg_type_number_t ) #$VM_REGION_BASIC_INFO_COUNT)
(#_vm_region (#_mach_task_self)
pnextaddr
psize
#$VM_REGION_BASIC_INFO
info
pinfo-size
pvm-object-name)
(let* ((gapsize (-(pref pnextaddr :vm_address_t) addr)))
(when (and (> addr 0) (> gapsize 0))
(incf total gapsize)
(#_mmap (%int-to-ptr addr) gapsize #$PROT_NONE (logior #$MAP_PRIVATE #$MAP_ANON) -1 0)))
(setq addr (+ (pref pnextaddr :vm_address_t) (pref psize :vm_size_t)))))))
(defun fill-malloc (limit)
(do* ((i 0 (1+ i))
(k 0))
((> k 10240) (* i x8632::tcr.size))
(if (> (%ptr-to-int (#_malloc x8632::tcr.size)) limit)
(incf k)
(setq k 0))))
More information about the Openmcl-devel
mailing list