[Openmcl-devel] Modal dialog problems with CCL 1.9 32/64 on Mountain Lion

[Gary Byers]

On Thu, 6 Sep 2012, Alexander Repenning wrote:

>
> On Sep 5, 2012, at 7:49 PM, Gary Byers wrote:
>
>> Just to clarify (I saw Alex's message indicating that our emails crossed):
>>
>> There are/have been several ways in the CCL IDE code base of forcing
>> something to execute in the main thread.  All of them ultimately call
>> either CCL::QUEUE-FOR-EVENT-PROCESS (which simply arranges that the
>> event process will run a specified function "soon") or
>> CCL::CALL-IN-EVENT-PROCESS (which waits until the event process runs
>> the function and returns the value(s) it returns.)  These functions
>> now default to using #/performSelectorOnMainThread... in preference to
>> using libdispatch.
>>
>> User code that uses things like GUI:EXECUTE-IN-GUI shouldn't have to
>> change.
>>
>
>
> We have done some testing. So far it seems as if this is holding up much better. Thank you! Opening up projects will no longer result in the memory surge.
>
> However, opening up a number of projects and then clicking controls will still result in the memory surge after a while.
>
> Just to be sure: you did not actually change GUI:EXECUTE-IN-GUI (that one already appeared to have used #/performSelectorOnMainThread previously) but did change CCL::QUEUE-FOR-EVENT-PROCESS and CCL::CALL-IN-EVENT-PROCESS , correct?

In the sources that we distribute (at least in the trunk and I don't think that
1.8 differs in this regard), GUI:EXECUTE-IN-GUI is defined in 
"ccl:cocoa-ide;cocoa-utils.lisp";

(defun execute-in-gui (thunk &key context)
   (declare (ignore context))
   (ccl::call-in-event-process thunk))

So yes, it's correct that I didn't change this function in my recent
commit, but it isn't correct that GUI:EXECUTE-IN-GUI previously called
#/performSelectorOnMainThread...  Nothing in the code that we distribute
called #/performSelectorOnMainThread... in (at least) the last few CCL releases;
my recent commit contains the only call to that method in the current trunk

In the past, we had used a few different low-level mechanisms to force the
event thread to run some code.  In the more recent past (sometime in the last
year or so), Matt consolidated the code so that everything goes through
CCL::QUEUE-FOR-EVENT-PROCESS and CCL::CALL-IN-EVENT-PROCESS and made these
primitives use the dispatch queue.  Since we seem to have reason to question
this use of the dispach queue on Lion and Mountain Lion, I changed those
primitives to use a new mechanism based on #/performSelectorOnMainThread:
and added a little bit of code to support that.  (The primitives actually
decide whether or not to use the new or old mechanism based on the value
of a special variable, just in case the next OSX release ("Mangy Stray" or
whatever they call it) breaks something else.

>
> There is a lot of event handling code that includes Matt's in-main-thread:

I know that Matt doesn't drink or use recreational drugs, so I think
that we're likely talking about either a psychotic episode or the work
of Matt's Evil Twin here.

As bad as this code is, it ultimately uses GUI:EXECUTE-IN-GUI (and
therefore whatever low-level mechanism EXECUTE-IN-GUI uses to cause
code to run on the event thread.)  In the current trunk, that
mechanism is based on #/performSelectorOnMainThread...; in recent
versions, it was based on Apple's Grand Central Dispatch (libdispatch)
technology, and the experiments of last week indicated that the
libdispatch mechanism led to (at least) display problems when running
an NSOpenPanel and that this was true when all of the code involved
was written in ObjC, so I changed our primitives (including
GUI:EXECUTE-IN-GUI) to use #/performSelectorOnMainThread.

If you have code that calls libdispatch-based functions (via CCL::DISPATCH-SYNC,
CCL::DISPATCH-ASYNC, or the FFI), you should change this code.

AFAIK, the only people who have seen the memory problems that you describe are:
  - you/Mike
  - me, when running your application.

If this is not true - if anyone else reading this has seen those problems in
a CCL not affected by ticket:1005 - I'd appreciate knowing that.

In the days of Classic MacOS, people used INIT resources to customize OS behavior,
and there were elaborate programs ("INIT managers") that could be used to control
which INITs loaded (and the order in which they loaded) at system boot time. If
someone experienced problems that others didn't, the first step was to advise them
to disable their INITs and see if the problem persisted.

Thankfully, this nightmare ended with OSX; there was a system
extension framework (whose name I can't remember; it may have
contained the word "Ape") that was popular a few years ago and
(depending on who you talked/listened to) was or was not responsible
for a wide class of problems, but I think that that thing is largely
gone now.

Affecting things at the OS level usually requires a kernel extension (kext),
and virtualization systems (VMWare, Parallels, VirtualBox) need to hook
into the OS at a fairly low level.  I don't remember the details, but when
I tried to use Parallels a few years ago I found that it was changing OS
behavior in ways that I didn't like and that it was necessary to drive
a silver stake through its heart in order to get rid of these effects.
I believe that there's a way to boot OSX with third-party kexts disabled,
and if you can do that it'd be helpful to know if this affects the memory
issue.  (I don't have any specific reason to be suspicious of Parallels;
I'm just suggesting that it could assist us in our inquiries.)  If you
have a machine on which Mountian Lion is installed and Parallels isn't,
it might be worth testing on that machine if you haven't already done so.


The one time that I crashed your application (by opening and closing projects
about a dozen times) I got a crash log that showed that 1.3GB of the address
space was full of #_malloc'ed objects and the rest of the addess space was
full of other stuff; the log didn't tell us what was being #_malloc'ed.  One
time when I tried to measure it I saw that opening and closing a project seemed
to leak around 30K bytes of malloc'ed memory (most of which seemed to involve
ObjC objects not getting released); that'll eventually kill you, but it'll take
a while. I didn't try to measure whether or to what degree other resources (Mach
ports) might be leaking.  (You can watch the process with the 'top' program in
the shell and that'd tell you about Mach port utilization; there are some scenarios
where a the inability to create a Mach port can cause excessive CPU and memory
utilization in the same CCL code that's involved in ticket:1005.)

You've said that you've sometimes gotten Apple crash reports that didn't seem
to shed much light on the problem; if I saw them, I might (or might not) reach
a different conclusion.  If you have several and send them to me off-list, I'll
look at them (and may or may not see anything.)

If much of the above reads like a sequence of stabs in the dark, that's exactly
what it is.

If the IN-MAIN-THREAD code below is indeed attributable to Matt's Evil Twin,
I sincerely hope that he's been banished back to his own parallel dimension.
(I never remember how that stuff works.)



>
> (defmacro IN-MAIN-THREAD (() &body body)
>  (let ((thunk (gensym))
>        (done (gensym))
>        (result (gensym)))
>    `(let ((,done nil)
>           (,result nil))
>       (flet ((,thunk ()
>                (setq ,result (multiple-value-list (progn , at body))
>                      ,done t)))
>         (gui::execute-in-gui #',thunk)
>         (process-wait "Main thread" #'(lambda () ,done))
>         (values-list ,result)))))
>
>
>
> Could there be some bad interaction with the process-wait and the changed functions above?
>
> Why is process-wait used given that execute-in-gui calls #/performSelectorOnMainThread:   with :waitUntilDone:  set to true? I assume it was used to make sure that the call is synchronous, i.e., wait for the function to be called by execute-in-gui to finish.
>
> thanks,   Alex
>
>
>