[Openmcl-devel] Cocoa and memory management

[Ron Garret]

Yes, the images are >4k.  (They're VGA 24-bit images delivered as jpegs.  The image quality is settable in the camera.  I currently have them set to be about 25k bytes.)

I tried the following experiments:

(loop (dispose-heap-ivector (make-heap-ivector 2048 'u8)))
(loop (dispose-heap-ivector (make-heap-ivector 8196 'u8)))

(multiple-value-setq (v1 p1) (make-heap-ivector 2048 'u8))
(multiple-value-setq (v2 p2) (make-heap-ivector 8196 'u8))

(loop (#/release (make-instance ns:ns-data :with-bytes p1 :length (length v1))))
(loop (#/release (make-instance ns:ns-data :with-bytes p2 :length (length v2))))

None of those leak.  However, when I do this:

(multiple-value-setq (v p) (grab-raw-image-data))
(setf d (make-instance ns:ns-data :with-bytes p :length (length v)))
(loop (#/release (make-instance ns:ns-image :with-data d)))

it leaks.  But when I stop the process and close the listener it was running in, then the memory is reclaimed.  It seems as if Cocoa is allocating something in the current autorelease pool when it creates an image, which makes a certain amount of sense if you think about it.  So I tested this theory by doing:

(OBJC:WITH-AUTORELEASE-POOL
  (loop (#/release (make-instance ns:ns-image :with-data d))))

When I aborted the process it took a long time to return to the toplevel, at which point the leaked memory was reclaimed.

So I think I know what's going on, but I still have no idea what to do about it.  But surely I'm not the first person to have this problem?

rg

On Jul 5, 2013, at 4:21 AM, Gary Byers wrote:

> I don't know, and may take way too long to say that ...
> 
> Do you know offhand if most of the images that you're reading are 4K bytes
> or larger ?  IIRC, #_malloc treats objects that're larger than the MMU page
> size (4K bytes on every platform that CCL runs on) differently than it treats
> smaller objects.  Smaller objects are allocated within largish regions of
> mapped pages; if we allocate a 3Kbyte object, free it, allocate it again,
> free it again ... we're just reusing some pages that #_malloc mapped into
> memory when the application started up.   Objects whose size is >= the MMU
> page size are allocated in their own (newly-mapped) set of pages.  Freeing
> large objects unmaps the pages they occupied, so if you did:
> 
> (dotimes (i n)
>  (#_free (#_malloc large-number))) ; ignore error checking
> 
> you could reasonably expect the process's memory usage to remain essentially
> constant.  I don't think that I've flamed Mach in public  in weeks (maybe
> longer), so I'll just say:
> 
>  1) I think that Mach invalidates that reasonable assumption (e.g., a process
>     that repeatedly maps and unmaps memory pages will see its memory utilization
>     grow rather than remain constant and this growth will continue until Mach
>     gets around to noticing that it's completely unnecessary.)
>  2) I've read a paper (published in the late 1980s) that indicates that this
>     behavior is intentional.  It makes no sense to me whatsoever and it's
>     somewhat hard to believe that it ever made sense to anyone.  (To be fair,
>     it's worth remembering that things were different 25 years ago; to be
>     equally fair, that observation suggests that things in Mach should have
>     adapted to the changing world more than they seem to have.)
> 
> One way of testing this theory is to change your code to reuse the same
> buffer (heap ivector) rather than allocating/deallocating it on every
> loop iteration.  (Obviously, for this to be viable you'd have to make
> the buffer "bigger than it'd ever need to be" or deal with the case where
> it isn't big enough, but you may be able to avoid those issues for the
> sake of this experiment.)
> 
> If this doesn't affect what top reports ... well, we'd know more than we do
> now and I'll have had the chance to say bad things about Mach without as
> much vitriol as is sometimes involved in that.
> 
> If this does affect things (and similar changes to CCL itself have been
> necessary in the past), we can all flame Mach in unison and with as much
> vitriol as we can.
> 
> 
> 
> On Thu, 4 Jul 2013, Ron Garret wrote:
> 
>> I'm writing some code that grabs a stream of images from a camera and displays them in a window.  The code looks roughly like this:
>> 
>> (loop
>> (multiple-value-bind (v p) (make-heap-ivector ...)
>>   (read-image-from-camera-into v)
>>   (let ((img (make-instance ns:ns-image :with-data
>>                (make-instance ns:ns-data :with-bytes p :length (length v)))))
>>     (display-image img)
>>     (dispose-heap-ivector v)
>>     (#/release img)
>>     )))
>> 
>> I'm trying to figure out if I have a memory leak because of not #/releasing the ns-data instance that I'm allocating, so I let this run for a while.  According to top, I seem to have a leak because the image size grows.  But when I stop the loop, close the listener it was running in, and call GC, the image size shrinks back down again.
>> 
>> It seems like there are two possibilities:
>> 
>> 1.  I'm allocating memory in the current listener's autorelease pool, and it's getting reclaimed when I close the listener.  Except I'm not calling #/autorelease anywhere.
>> 
>> 2.  TOP is lying, and I'm not really leaking at all, but CCL is playing some games with virtual memory that fools top into thinking the image is growing when it's really not.
>> 
>> Any hints from those wise in the ways of objc memory management would be much appreciated.
>> 
>> Thanks,
>> rg
>> 
>> _______________________________________________
>> Openmcl-devel mailing list
>> Openmcl-devel at clozure.com
>> http://clozure.com/mailman/listinfo/openmcl-devel
>> 
>>