[Openmcl-devel] mach ports leaking?

[Gary Byers]

I don't know what your program's doing, so I can't really guess why top
clams that it's using as many Mach ports as it is.

There are a relatively small number of things ... threads (and additional
things used by Mach per-thread exception handling), semaphores ... that
are themselves Mach ports.  The lisp obviously allocates these things;
as far as I know, portresources that're associated with threads and semaphores
are reliably reclaimed when those things become unreachable.

I'm sure that there are lots of ways in which ports are used in and allocated
by the Mach kernel (possibly by the memory system, I/O, other forms of IPC,
whatever ...).  I don't know exactly what it would mean for an application
to "leak" ports that it doesn't explicitly create.

Depending on how files are opened, they may hae locks associated with
them, as can hash tables and other lisp data structures.  (Locks in
Darwin have semaphores - and therefore Mach ports - associated with
them.)  All of these things should get GCed eventually, unless
something takes unusual steps to keep this from happening (e.g., (PUSH
(MAKE-HASH-TABLE) *BIG-LIST-OF-HASH-TABLES*))

I also don't know why your application is using > 90000 ports; a lisp
sitting idle seems to use ~200; the IDE seems to use >500 (very few of
which are explicityly created by the lisp.)  A utility that provides
some sort of support for a Logitech mouse seems to be using >30000,
FireFox is using about 300 at the moment ... so yes, 90000 seems a bit
high.

The little lisp function that's enclosed shoud return the same number
that top reports.  (The second return value is the number of "dead
port names", which I assume has something to do with Mach's way
of counting references and recycling the integers used to name ports.)

? (gc)
NIL
? (mach-port-count)
180
0
? (make-semaphore)
#<SEMAPHORE #x300053B3FD9D>
? (mach-port-count)
181
0
? ()
NIL
? ()
NIL
;;; Nothing - including the repl history variables - is referencing the
;;; semaphore, so the GC should free it.
? (gc)
NIL
? (mach-port-count)
180
0
;;; And it appears that it did.


I don't know.  I wouldn't want to say that there's no way in which CCL
could leak ports, but I don't know of any way in which it does.  I don't
know exactly how the lisp would create 90000 ports (short of creating
lots of threads/semaphores/locks and losing track of them), and I
certainly don't know how your code is doing that.

The enclosed function might help to narrow it down a bit.  Apple
alsp provides a more extensive sample program

<http://developer.apple.com/samplecode/MachPortDump/>

that might be useful.

I was unable to find anything in anything that I'd be suspicious of
(thread/semaphore creation/destruction) that suggests a leak.  If
you can identify what parts of your program are (apparently) creating
lots of ports and/or not releasing them and the lisp seems to be
implicated in that, I'd certainly be interested in trying to fix
it, but I don't have a good guess as to what that might be.

On Mon, 2 Feb 2009, David Reitter wrote:

> When running a CPU+memory intensive job, I notice that the number of mach 
> ports is fairly high for the ccl process - and it is increasing rapidly:
>
> PID COMMAND      %CPU   TIME   #TH #PRTS #MREGS RPRVT  RSHRD  RSIZE  VSIZE
> 30223 mdworker     0.0%  0:00.18   3    50     31  664K  2712K  2428K    34M
> 30214 bash         0.0%  0:00.01   1    14     18  228K   660K   896K    18M
> 30213 login        0.0%  0:00.02   1    17     55  284K   260K  1064K    19M
> 30211 dx86cl64    71.4%  3:45.21   3 92673+   327  226M+  188K   228M+  256M
>
>
> Is this a possible memory/thread leak?
> As little as I understand about the implementation, I doubt it's on the heap: 
> (ccl:gc) doesn't change #PRTS.
>
> I'm on Darwin64 (OSX 10.5.6), with CCL 1.2 and also current checkout.
>
>
-------------- next part --------------
(in-package "CL-USER")


;;; Return the total number of Mach ports in the current task and
;;; the number of "dead port names".
(defun mach-port-count ()
  (rlet ((names-count #>mach_msg_type_number_t 0)
         (types-count #>mach_msg_type_number_t 0)
         (pnames #>mach_port_name_array_t)
         (ptypes #>mach_port_type_array_t)
         (ptype #>mach_port_type_t))
    (let* ((task (#_mach_task_self))
           (err (#_mach_port_names task pnames names-count ptypes types-count)))
      (when (eql 0 err)
        (let* ((nnames (pref names-count #>mach_msg_type_number_t))
               (nnamebytes (* nnames (ccl::%foreign-type-or-record-size #>mach_port_name_t :bytes)))
               (ntypebytes (* nnames (ccl::%foreign-type-or-record-size #>mach_port_type_t :bytes)))
               (ndead 0)
               (types (pref ptypes #>mach_port_type_array_t)))
          (dotimes (i nnames)
            (let* ((type (paref types  #>mach_port_type_array_t i)))
              (when (logtest type #$MACH_PORT_TYPE_DEAD_NAME)
                (incf ndead))))
          (#_vm_deallocate task (%ptr-to-int (pref pnames #>mach_port_name_array_t)) nnamebytes)
          (#_vm_deallocate task (%ptr-to-int types)  ntypebytes)
          (values nnames ndead))))))