[Openmcl-devel] memory

[Gary Byers]

On Mon, 12 Apr 2010, Bill St. Clair wrote:

> It seems to me that what you need would be partially your own recursive
> data structure walker and partly a CCL-provided SIZEOF generic function.
> The SIZEOF generic function would take a single argument and return the
> number of bytes used by that object alone, but not anything it
> references. Your function would walk one of your data structures,
> calling sizeof on each Lisp object that you consider to be part of that
> one user-level data structure and summing all the results.
>
> I sent in my last reply the code for SIZEOF on a CCL standard CLOS
> instance. Adding a full-featured SIZEOF generic function to CCL would
> likely be useful, and is probably not very difficult. You could do it
> yourself with a little routing around in the CCL source code. Or maybe
> somebody else on the list would like to make a portable version.
>
> -Bill
>

I like that idea: you can say over and over again that the hard part of
this involves determining what to traverse and how to traverse it (in
bounded stack space and dealing with circularity and other kinds of
structure-sharing), but there's nothing like delegating that problem
to someone else to really drive that point home.


(I see memory as being a twisted maze of interconnected references,
all alike ... so I tend to see this problem as being much harder in
general than it may be in some specific cases.)

;;; Arbitrary, not-necessarily-useful example:
(defun total-size-of-list (l)
   (cond ((atom l) (ccl::sizeof l))
         (t (+ (ccl::sizeof l)
               (total-size-of-list (car l))
               (total-size-of-list (cdr l))))))

? (total-size-of-list '(a b c d))
320
;;; That's 4 SYMBOLs that happen to be distinct at 64 bytes each, + 4 unique
;;; CONSes at 16 bytes each.  Not a particularly useful thing to measure,
;;; but a consistent answer.  If it's not clear to anyone, it's harder to
;;; get a consistent answer for cases involving structure sharing/circularity:

? (total-size-of-list '#1=(a b c d . #1#))

- you'd generally need to keep track of which things have been visited already -
and it can be hard to recursively traverse an arbitrarily large complex data
structure in bounded stack space even when no structure-sharing/circularity
is involved.

;; If this doesn't stack-overflow, try a longer list ...
? (total-size-of-list (make-list (ash 1 20)))

If you don't need to handle these kinds of cases, then yes, the hard part is
knowing how to obtain the size of some primitive object.  If you do need more
generality than that, then CCL::SIZEOF is very much the easy part.  I -think-
that the enclosed definition is correct.
-------------- next part --------------
(in-package "CCL")

(defun sizeof (thing)
  ;; All memory-allocated objects in CCL are either CONSes or
  ;; "UVECTOR"s; a UVECTOR contains a header which describes the
  ;; object's primitive type (represented as an (UNSIGNED-BYTE 8) and
  ;; accessible via the function CCL::TYPECODE) and element-count
  ;; (accessible via the function CCL::UVSIZE.)  A function defined in
  ;; the compiler backend knows how to map from a typecode and
  ;; element-count to a size in octets.  UVECTORs are aligned on
  ;; doubleword boundaries and contain this extra header word, so the
  ;; "physical size" of the uvector is a bit bigger.  On x86-64,
  ;; SYMBOLs and FUNCTIONs have their own tag, but there's an
  ;; underlying UVECTOR.
  (cond ((null thing) 0)
        ((consp thing) #+64-bit-target 16 #+32-bit-target 8)
        #+x8664-target ((symbolp thing) (sizeof (%symptr->symvector thing)))
        #+x8664-target ((functionp thing) (sizeof (function-to-function-vector thing)))
        ((uvectorp thing)
         (let* ((typecode (typecode thing))
                (element-count (uvsize thing))
                (sizeof-content-in-octets
                 ;; Call the architecture-specific backend function.
                 (funcall (arch::target-array-data-size-function
                           (backend-target-arch *host-backend*))
                          typecode element-count)))
           (logandc2 (+ sizeof-content-in-octets
                           #+64-bit-target (+ 8 15)
                           #+32-bit-target (+ 4 7))
                     #+64-bit-target 15
                     #+32-bit-target 7)))
        (t 0)))