[Openmcl-devel] memory

[Gary Byers]

Hmm.  As a first approximation:


? (defun consed-by (thunk)
   (let* ((before (ccl::total-bytes-allocated)))
     (funcall thunk) ; for effect, for now
     (- (ccl::total-bytes-allocated) before)))
CONSED-BY
? (consed-by (lambda () (cons nil nil)))
16              ; correct, on a 64-bit lisp.

? (consed-by (lambda () (make-instance 'two-way-stream :input-stream (two-way-stream-input-stream *terminal-io*) :output-stream (two-way-stream-output-stream *terminal-io*))))
1312

That number looks high; it includes not only the size of the stream we created,
but any incidental consing done during initialization (&rest args, etc) and may
also reflect CLOS caches warming up.  If I call it again, I get 80 (which suggests
that the higher figure likely came from cache effects.)

The more reliable/repeatable number (80, in this case) might include some incidental
consing of short-lived objects (&rest args, any consing done by initfunctions, etc)
so that number's generally going to be at least a little bigger than the "true"
size of the object.

CCL::TOTAL-BYTES-ALLOCATED looks at a per-thread 64-bit counter's value in the
calling thread.  The counter's value should never decrease (until it wraps around;
I don't remember whether it's signed or unsigned) and AFAIK it's maintained
accurately.  (Since it becomes a bignum pretty quickly on 32-bit CCLs, calling
CCL::TOTAL-BYTES-ALLOCATED can cause 8 or 16 bytes to be allocated ...)

So, yes: this may report some other incidental consing and may need to be
called more than once to avoid seeing GF cache-loading effects, but it's
probably going to give you a usably close approximation of an object's
total size at the time it's created.





On Tue, 13 Apr 2010, Ron Garret wrote:

> Another way to frame the question that might make it easier to answer to to pose it thusly: how many bytes will be allocated if I allocate an object of this type?
>
> I noodled around with trying to answer this question by wrapping e.g. (ccl::total-bytes-allocated) around a call to an allocation function but couldn't get it to work.  But maybe someone who actually knows how the internal bookkeeping works (hi Gary!) would have more success with this approach.
>
> rg
>
>
> On Apr 13, 2010, at 7:02 AM, Daniel Weinreb wrote:
>
>> Tim, Gary,
>>
>> Tim Bradshaw wrote:
>>>
>>> .. and that's more-or-less the problem I came up with.  An
>>> implementation could, I think, provide some kind of "map-function-over-
>>> all-the-objects-this-thing-refers-to" function.
>> I agree.  How about a generic function that takes
>> object X and tells you (1) the size of the X, including all
>> of the Lisp objects that make up X logically, and (2) all of
>> the objects X refers to?
>>
>> By "tells you", I mean it could either return these
>> things, or call a function on them.  For example,
>> it could take a function as an argument, call
>> that on all the referenced objects, and return
>> the size of this (logical) object.
>>
>> CCL could define methods for
>> that generic function on such types as hash tables
>> and closures. CCL knows all
>> the primitive-objects that the hash table is
>> made of and would not expose that to the caller.
>>
>> Also, the application would define its own such
>> generic methods. The method can make its
>> own decisions about which referenced object
>> to follow, and which not to follow, and which
>> count as part of the size and which don't.
>>
>>
>>>  But that has at least
>>> two problems: how does the user of that function know which of these
>>> objects are ones they already know about and which are genuinely new?
>>>
>> You mean, what if there is double-counting because
>> object A refers to object B by more than one path?
>>
>> Yes, that's a problem.  I'd say that the proposed generic
>> function would not worry about this; it would be the
>> caller's responsibility.  The caller might "know" that
>> there aren't any cases of this in his own class, for
>> example, or otherwise be able to avoid the general
>> duplicate-elimination problem by use of higher-level
>> knowledge about an object.  Otherwise, I think you
>> have to have an eq hash table to keep track of
>> which objects have been seen already, which of
>> course is cost in time and space.
>>
>> Suppose your data structure includes symbols,
>> and you use their property lists.  But you don't
>> use their values, and some might have values
>> because they were already interned and someone
>> else used them.  In that case, your method would
>> call the generic function but ignore the "what
>> does the object reference", and instead call
>> the generic function directly on the property list.
>>
>>> and, what about objects that something refers to which should be
>>> genuinely secret (perhaps they're not safe for user code to even refer
>>> to or something like that)?
>> The generic function hides things like how hash tables
>> are implemented, just as generic functions can hide
>> other things about the implementation. Is that
>> good enough?
>>
>> One potential problem here is that if your data structure
>> uses symbols, it might be particularly hard to decide
>> whether your data structure is "responsible" for the
>> symbol, or whether the symbol was already created
>> anyway.
>>
>>
>>> I suppose the first of these could be
>>> solved by user code doing an EQ check (which it will need to do anyway
>>> to avoid double-counting and loops).  The second I'm not sure about -
>>> you could include the really-secret stuff in the basic size of the
>>> object (what SIZEOF) returns, but what if some of that is shared?
>>>
>>> I don't think any of this is easy, at all.  It's mostly the same as
>>> the answer to "why isn't there a general object copier in Lisp?",
>>> except this is a bit harder.
>>> _______________________________________________
>>> Openmcl-devel mailing list
>>> Openmcl-devel at clozure.com
>>> http://clozure.com/mailman/listinfo/openmcl-devel
>>>
>> _______________________________________________
>> Openmcl-devel mailing list
>> Openmcl-devel at clozure.com
>> http://clozure.com/mailman/listinfo/openmcl-devel
>
> _______________________________________________
> Openmcl-devel mailing list
> Openmcl-devel at clozure.com
> http://clozure.com/mailman/listinfo/openmcl-devel
>
>