<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>I am sorry if you feel I have wasted your or Matts time. I have always appreciated your support and beyond that provided some actual funding to Clozure Associates. Moreover, I also offered, in a couple of emails ago, potential financial support if you can help us with this problem. I still think this is a pretty general CCL issue which sooner or later would manifest itself in most gui-based CCL applications trying to run on Mountain Lion. </div><div><br></div><div>I do apologize for having created a confusion with a version number (this was the result of an unnoticed build error with the latest build) but are trying to help you. We have created smaller and smaller test cases to rule out any leftover side effects and have run them on a large number of machines and CCL versions. I am sure you have spent a good deal of time on this but I am also convinced we have spent even more time. You know, we are pretty frustrated here as well. We are sitting in the same boat of trying to make things work.</div><div><br></div><div>Looks like the main issue is that you either think this is not a general issue with CCL on Mountain Lion or simply hope Mountain Lion is going away. I am not a great fan of Mountain Lion myself. However, as tool builders I believe we have the responsibility to build tools for the platforms used by our users. I don't think hoping Mountain Lion will just go away is a very productive approach. Matthew's efforts are not wasted. I can crash CCL <span class="Apple-style-span" style="font-size: 12px; ">1.9-dev-r15450-trunk 32 and 64 bit with and without his mods (he added gui:execute-in-gui). However, with gui:execute-in-gui I have to select files to make the crash happen. </span></div><div><span class="Apple-style-span" style="font-size: 12px; "><br></span></div><div>At any rate, I think CCL and Mountain Lion are not playing together well. What do you want me to do to get things fixed: mail big check, submit more test cases, try with more tools, send Apple motivational speaker...?</div><div><br></div><div>Alex</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>_________</div><div><br></div><div><div>;; CCL 1.9-dev-r15450-trunk (32/64) crash on Mountain Lion 10.8.1</div><div><br></div><div><br></div><div>(defun choose-file-dialog2 ()</div><div> ;; 100% kosher: retain, no use of depreciated calls</div><div> (let ((panel (#/retain (#/openPanel ns:ns-open-panel))))</div><div> (#/runModal panel)</div><div> (#/release panel)))</div><div><br></div><div><br></div><div>(defun THE-AMAZING-MEMORY-SURGE ()</div><div> (dotimes (i 100)</div><div> (gui:execute-in-gui #'(lambda () ;; if you use this then you will get spinning progress bar and folder content no show</div><div> (choose-file-dialog2)))</div><div> (ccl::process-run-function "pretent to load project" #'(lambda ()))))</div><div><br></div><div><br></div><div>;; this will pop up a file chooser for a number of time. Select a file and watch the Activity Monitor. </div><div>;; Set view > update frequency in Actvity Monitor to fast (0.5s) for best results</div><div>;; Watch out for Clozure CL % CPU and Real Mem</div><div>;; for some time Real Mem will go up gradually (memory leak) then at some unpredicatable time it will SURGE to GIGABITES of memory </div><div>;; and ultimatley crash CCL</div><div>;; with with-autorelease-pool may crash quite quickly with a Unhandled exception 10, comment out if needed</div><div><br></div><div>; (the-amazing-memory-surge)</div><div><br></div></div><div><br></div><div>___________________</div><div><br></div><br><div><div>On Aug 30, 2012, at 8:45 AM, Gary Byers wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div><br><br>On Wed, 29 Aug 2012, Alexander Repenning wrote:<br><br><blockquote type="cite">On Aug 29, 2012, at 6:00 PM, R. Matthew Emerson wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> On Aug 29, 2012, at 7:04 PM, Alexander Repenning<br></blockquote><blockquote type="cite"> <<a href="mailto:Alexander.Repenning@colorado.edu">Alexander.Repenning@colorado.edu</a>> wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> I think I have here a pretty Kosher (uses retain, does not<br></blockquote><blockquote type="cite"> use depreciated functions) version of the dialog + memory<br></blockquote><blockquote type="cite"> surge problem. This version of?choose-file-dialog is<br></blockquote><blockquote type="cite"> completely stripped of any non essential activity. It does<br></blockquote><blockquote type="cite"> not even return the path, i.e., there is no practical<br></blockquote><blockquote type="cite"> value to this function.<br></blockquote><blockquote type="cite">Please have a go and see if you can or cannot experience that<br></blockquote><blockquote type="cite">Memory surge phenomenon. Please follow the instructions closely.<br></blockquote><blockquote type="cite">Otherwise you may miss the issues at sometimes can be kind<br></blockquote><blockquote type="cite">subtle.?<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> ;; CCL 1.8.1 64 (Mac App store) crash on Mountain Lion<br></blockquote><blockquote type="cite"> 10.8.1<br></blockquote><blockquote type="cite">The version of CCL in the Mac App Store still contains a bug in it<br></blockquote><blockquote type="cite">that Mountain Lion triggers. ?I am pretty sure that your test case is<br></blockquote><blockquote type="cite">running into that bug.<br></blockquote><blockquote type="cite"><br></blockquote><br>At the very least, the way that you presented your test case triggers<br>a known bug in that version of CCL and says nothing about whether or<br>not some other bug remains.<br><br><blockquote type="cite">I think we are going around in circles.<br></blockquote><br>I would have used a harsher term for the above, but OK: continually<br>using a 64-bit version of CCL that's known to have a bug which has<br>similar symptoms is "going around in circles." You may actually<br>be going through some careful and controlled testing procedure and<br>just spacing out and clouding the issue like this when you report<br>your findings, but this wastes time and makes it harder than it<br>should be to take you seriously. (This is not the first time that<br>you've done this.)<br><br><blockquote type="cite">While that bug does sound similar in<br></blockquote><blockquote type="cite">spirit I am quite sure its not the one because:<br></blockquote><blockquote type="cite">1) the error also manifests itself in the 32 bit version of CCL<br></blockquote><br>Matt wouldn't have wasted his time if the message he replied to had<br>said so clearly.<br><br>I'm honestly not trying to be dismissive or sarcastic here. This<br>stuff is complicated, and it's important to be as precise as possible<br>when discussing it (much more precise than one might be in casual<br>conversation.) That may take extra effort, but the alternative seems<br>unacceptable to me.<br><br><br><blockquote type="cite">2) We tried the most recent version of CCL?1.9-dev-r15450-trunk?<br></blockquote><blockquote type="cite">(DarwinX8632)! and the 64 bit version. Both the test case and the full app<br></blockquote><blockquote type="cite">crashed with the same Memory surge.<br></blockquote><blockquote type="cite">?<br></blockquote><blockquote type="cite">When I try your modified version (which we actually did try before as well)<br></blockquote><blockquote type="cite">things APPEAR to be better as long as you just dismiss the dialog with ESC.<br></blockquote><blockquote type="cite">However, the reason for that appear to be that then the file choose dialog<br></blockquote><blockquote type="cite">goes into this super slow, spinning indeterminate progress indicator mode<br></blockquote><blockquote type="cite">where it does not list contents of folder. That is interesting. However, if<br></blockquote><blockquote type="cite">you actually try to select a file, instead of pressing ESC, then there is a<br></blockquote><blockquote type="cite">good chance it will crash even faster than before. I never made it beyond<br></blockquote><blockquote type="cite">the first attempt. Can you confirm?<br></blockquote><blockquote type="cite"><br></blockquote><br>I've seen a spinning progress indicator (something that would have been a<br>beachball cursor a few OS revisions ago, and generally indicates that progress<br>isn't being made) appear in the lower left corner of the open panel. I wasn't<br>paying close attention to when this did and did not appear, but my impression<br>is that that there was some correlation between that cursor spinning around<br>and some kinds of misbehavior (e.g., "empty" or largely empty panel views that<br>shouldn't be empty.)<br><br>I don't think that I've seen this since trying to use a CHOOSE-FILE-DIALOG<br>implementation that (at a minimum) retained the panel before it was used<br>and released it afterwards. I'm 100% sure that I haven't seen excessive<br>memory or CPU utilization, but I've only seen that once (and only while<br>running your application.)<br><br><br><blockquote type="cite">3) An even simpler test just starting 1000 processes, one after the other,<br></blockquote><blockquote type="cite">does not exhibit the problem (32 bit).<br></blockquote><br>Note that a thread/process in 32-bit CCL needs about 2.5MB of foreign memory<br>just for its stacks; it also uses other finite resources (semaphores, message<br>ports, etc.) You can't have 1000 runnable threads in 32-bit CCL because the<br>~2.5GB of foreign memory isn't available; the only way that a loop that calls<br>PROCESS-RUN-FUNCTION 1000 times can run to completion is some older threads<br>exit before newer ones are created, and the only way that happens is if those<br>threads run to completion (and the more threads are created and competing for<br>CPU time and other resources, the less deterministic that is.)<br><br>In practice, I'd expect something like:<br><br>(dotimes (i 1000) (process-run-function "nothing" (lambda ())))<br><br>to use a lot of CPU but probably not exhaust virtual memory (simply because<br>all of the CPU contention keeps the thread running that loop from running<br>very often.) This isn't guaranteed, and it's such a ridiculous thing to do<br>that I'd find it difficult to get too worked up about things if it didn't.<br><br>If we put a delay in that loop:<br><br>(dotimes (i 1000)<br> (choose-file-dialog)<br> (process-run-function "nothing" (lambda ())))<br><br>we're probably effectively serializing thread creation, but we're also affecting<br>the environment in which CHOOSE-FILE-DIALOG runs (competing for some of the<br>same OS resources that the thread is competing for.) I'd expect that to work<br>as well, but of course it's also a ridiculous thing to do and it's hard to care<br>too much about whether it does or not.<br><br>Also ridiculous (in different ways) is:<br><br>(dotimes (i 1000)<br> (choose-file-dialog))<br><br>where CHOOSE-FILE-DIALOG retains the open-panel appropriately. I haven't<br>seen that fail, but I haven't counted to 1000 either.<br><br>Apple distributes a command-line tool called "heap", which will scan the<br>(malloc'ed) heap zones of a specified process and (among other things)<br>identify the number and sizes of the ObjC instances (the program calls them<br>"classes" ...) it finds there. The ObjC heap shouldn't change significantly<br>on each iteration and (when not answering email or trying to do work that<br>we actually get paid for) I've been trying to verify that it doesn't.<br><br><blockquote type="cite">In summary I think we still don't know what is going on but I don't think it<br></blockquote><blockquote type="cite">is connected to that specific malloc issue you mention.<br></blockquote><br>No one else thinks so either (at least not directly), but if you say<br>"running a test in a version of CCL affected by that issue says something<br>useful" a lot of time gets wasted by Matt or me saying "no it doesn't" and<br>then by you saying "and by 'version of CCL affected by that issue', I mean<br>'other versions'". I am emphatically in favor of streamlining this process.<br><br><blockquote type="cite">Need to run but I do remember reading that in sandbox mode (we are NOT<br></blockquote><blockquote type="cite">running in sandbox or or are we?) file choosers are NOT running in the main<br></blockquote><blockquote type="cite">thread but in some other special thread.?<br></blockquote><br>They actually run in another OS-level process, which means that the internals<br>of NSOpenPanel are probably a lot different than they used to be (and may<br>change again if and when the whole "sandboxing" issue goes away.)<br><br><blockquote type="cite">hmmm....<br></blockquote><blockquote type="cite">puzzled, ?Alex?<br></blockquote><blockquote type="cite"><br></blockquote><br>I'm a little less puzzled than I had been. The implementation of<br>CHOOSE-FILE-DIALOG (actually of GUI::COCOA-CHOOSE-FILE-DIALOG) was clearly<br>wrong and could result in methods being invoked on freed objects (which<br>in turn could cause malloc heap corruption that snowballs into further<br>malloc heap corruption ...) Modifying the state of a freed object could<br>involve modifying free memory (usually relatively harmless) or modifying<br>the state of some other object that's subsequently allocated at the freed<br>address (usually relatively harmful.) I don't know for sure that any of<br>this leads to the memory/CPU problems that you've seen, but it's certainly<br>plausible that it could. I'm fairly certain that it could lead to the<br>cosmetic problems (apparently empty directories and other display problems).<br><br>The problem described in ticket 1005 caused almost exactly the same symptoms.<br>Recall that that involved allocating a per-thread data structure whose address<br>could be used to identify a Mach message port; traditionally (I don't know<br>if this has changed), a Mach message port identifier has been a 32-bit value.<br>So, the code that allocated that data structure was:<br><br>(let* ((free-later ())<br> (p nil))<br> (loop<br> (setq p (#_malloc few-hundred-bytes))<br> (if (and (is-32-bit-pointer p)<br> (can-be-used-as-mach-port p)) ; not coincidentally also Mach port name<br> (return)<br> (push p free-later)))<br> ;; Free anything that failed the test above<br> (dolist (bad free-later) (#_free bad)))<br><br>That was changed: especially on Mountain Lion, #_malloc would often return<br>pointers for which IS-32-BIT-POINTER wasn't true, so we use alternatives to<br>#_malloc and #_free on 64-bit platforms (and IS-32-BIT-POINTER is therefore<br>always true.) The second test - that the pointer's address can be used as<br>a Mach port name - is intended to catch conflicts with the (often essentially<br>random) set of existing Mach port names. The second test is implemented by<br>trying to use P as a Mach port name and seeing if that fails; it could fail<br>if P was coincidentally already a Mach port name, but might (I'd have to RTFM)<br>also fail for other reasons (like "Mach is too damned busy now; try later." Yes,<br>it's 2012.) If the loop above was run while Mach was too damned busy, we'd keep<br>_mallocing until we were out of memory, and the one time that I was able to<br>get your application to fail malloc's heap was full. (I don't know that this<br>is what's happening, but it may be worth exploring.)<br><br>To state the obvious: Matt and I (and anyone else at Clozure who could<br>look at this) have other work that customers are paying us to do and<br>that other work has to take priority over this; if you think<br>otherwise, you are quite simply wrong and that isn't open to<br>discussion (with me, my partners, or anyone else *). If you're just<br>getting someone's spare time to look at this, that seems like an even<br>stronger reason to not waste that time with things like "running this<br>test in a version of CCL known to exhibit these symptoms for other<br>reasons exhibits these symptoms."<br><br>------<br>[*] I used to deal with a company whose slogan was "We cheat the other<br>guy, and pass the savings on to you!". Sadly, they aren't in business<br>anymore ...<br><br><blockquote type="cite">Gary mentioned this bug in a previous message:<br></blockquote><blockquote type="cite">- there's a known bug in 64-bit CCL on OSX that can cause lisp thread<br></blockquote><blockquote type="cite">creation<br></blockquote><blockquote type="cite">??to go into a horrible CPU-burning/memory-thrashing state. ?I think<br></blockquote><blockquote type="cite">that that<br></blockquote><blockquote type="cite">??bug's been present for a long time (since PPC64 days), but it's<br></blockquote><blockquote type="cite">apparently<br></blockquote><blockquote type="cite">??much easier to trigger on 10.8 (and/or recent versions of CCL) than<br></blockquote><blockquote type="cite">it has been.<br></blockquote><blockquote type="cite">??The problem ultimately has to do with whether or not #_malloc<br></blockquote><blockquote type="cite">(actually #_calloc)<br></blockquote><blockquote type="cite">??returns a 64-bit pointer whose high 32 bits are 0 and there can be<br></blockquote><blockquote type="cite">many factors<br></blockquote><blockquote type="cite">??that affect that (many of them subtle), and the fix is to stop<br></blockquote><blockquote type="cite">assuming that<br></blockquote><blockquote type="cite">??it does and allocate such pointers ourselves.<br></blockquote><blockquote type="cite">??That's been fixed (in the trunk for a few weeks and in the 1.8 tree<br></blockquote><blockquote type="cite">??for a few days) in svn; the symptoms happen to be very similar to<br></blockquote><blockquote type="cite">??what people have reported seeing with CHOOSE-FILE-DIALOG, but the<br></blockquote><blockquote type="cite">??CHOOSE-FILE-DIALOG problems seem to occur for at least some people<br></blockquote><blockquote type="cite">??in 32-bit CCL (which was never affected by this thread-creation<br></blockquote><blockquote type="cite">? problem) and in freshly-updated 64-bit versions.<br></blockquote><blockquote type="cite">The fix for this bug is not yet in the Mac App Store version of CCL.<br></blockquote><blockquote type="cite">?I'll try to update the Mac App Store version soon, but in the<br></blockquote><blockquote type="cite">meantime, please try using up-to-date CCL obtained via Subversion<br></blockquote><blockquote type="cite">(either trunk or 1.8).<br></blockquote><blockquote type="cite">I modified your test case to make the call to the open panel take<br></blockquote><blockquote type="cite">place in the main thread. ?It seemed to work as expected for me in an<br></blockquote><blockquote type="cite">up-to-date trunk CCL.<br></blockquote><blockquote type="cite">;; modified to use gui:execute-in-gui<br></blockquote><blockquote type="cite">(defun THE-AMAZING-MEMORY-SURGE ()<br></blockquote><blockquote type="cite">? (dotimes (i 100)<br></blockquote><blockquote type="cite">? ? (gui:execute-in-gui #'(lambda ()<br></blockquote><blockquote type="cite">? ? ? ? ? ? ? ? ? ? ? ? ? ? (choose-file-dialog2)))<br></blockquote><blockquote type="cite">? ? (ccl::process-run-function "pretent to load project" ?#'(lambda<br></blockquote><blockquote type="cite">()))))<br></blockquote><blockquote type="cite">(defun choose-file-dialog2 ()<br></blockquote><blockquote type="cite">? ;; 100% kosher: retain, no use of depreciated calls<br></blockquote><blockquote type="cite">? (let ((panel (#/retain (#/openPanel ns:ns-open-panel))))<br></blockquote><blockquote type="cite">? ? (#/runModal panel)<br></blockquote><blockquote type="cite">? ? (#/release panel)))<br></blockquote><blockquote type="cite">(defun THE-AMAZING-MEMORY-SURGE ()<br></blockquote><blockquote type="cite">? (dotimes (i 100)<br></blockquote><blockquote type="cite">? ? (ccl::with-autorelease-pool<br></blockquote><blockquote type="cite">? ? ? ? (choose-file-dialog2)<br></blockquote><blockquote type="cite">? ? ? (ccl::process-run-function "pretent to load project"<br></blockquote><blockquote type="cite">?#'(lambda () )))))<br></blockquote><blockquote type="cite">;; this will pop up a file chooser for a number of times. Each<br></blockquote><blockquote type="cite">time just press ESC and watch the Activity Monitor.?<br></blockquote><blockquote type="cite">;; Set view > update frequency in Actvity Monitor to very often<br></blockquote><blockquote type="cite">(0.5s) for best results<br></blockquote><blockquote type="cite">;; Watch out for Clozure CL % CPU and Real Mem<br></blockquote><blockquote type="cite">;; for some time Real Mem will go up gradually (memory leak)<br></blockquote><blockquote type="cite">then at some unpredicatable time it will SURGE to GIGABITES of<br></blockquote><blockquote type="cite">memory?and ultimately crash CCL<br></blockquote><blockquote type="cite">;; with with-autorelease-pool CCL may crash quite quickly with a<br></blockquote><blockquote type="cite">Unhandled exception 10, comment out if needed<br></blockquote><blockquote type="cite">; (the-amazing-memory-surge)<br></blockquote><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">Openmcl-devel mailing list<br></blockquote><blockquote type="cite"><a href="mailto:Openmcl-devel@clozure.com">Openmcl-devel@clozure.com</a><br></blockquote><blockquote type="cite"><a href="http://clozure.com/mailman/listinfo/openmcl-devel">http://clozure.com/mailman/listinfo/openmcl-devel</a><br></blockquote><blockquote type="cite">Prof. Alexander Repenning<br></blockquote><blockquote type="cite">University of Colorado<br></blockquote><blockquote type="cite">Computer Science Department<br></blockquote><blockquote type="cite">Boulder, CO 80309-430<br></blockquote><blockquote type="cite">vCard: <a href="http://www.cs.colorado.edu/~ralex/AlexanderRepenning.vcf">http://www.cs.colorado.edu/~ralex/AlexanderRepenning.vcf</a><br></blockquote><blockquote type="cite"><br></blockquote></div></blockquote></div><br><div>
<span class="Apple-style-span" style="font-size: 12px; "><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Helvetica" size="3" style="font: 12.0px Helvetica">Prof. Alexander Repenning</font></p><p style="margin: 0.0px 0.0px 0.0px 0.0px"><br class="khtml-block-placeholder"></p><p style="margin: 0.0px 0.0px 0.0px 0.0px">University of Colorado</p><p style="margin: 0.0px 0.0px 0.0px 0.0px">Computer Science Department</p><p style="margin: 0.0px 0.0px 0.0px 0.0px">Boulder, CO 80309-430</p><p style="margin: 0.0px 0.0px 0.0px 0.0px"><br class="khtml-block-placeholder"></p><p style="margin: 0.0px 0.0px 0.0px 0.0px"><font face="Helvetica" size="3" style="font: 12.0px Helvetica">vCard: <a href="http://www.cs.colorado.edu/~ralex/AlexanderRepenning.vcf">http://www.cs.colorado.edu/~ralex/AlexanderRepenning.vcf</a></font></p><br class="Apple-interchange-newline"></span></span></span>
</div>
<br></body></html>