[Openmcl-devel] Sudden increment in executable image size.

Kaz Kylheku kaz at kylheku.com
Thu Nov 18 06:01:21 UTC 2010


On Wed, 17 Nov 2010 20:42:44 -0700 (MST), Gary Byers <gb at clozure.com>
wrote:
> The fact that the file size that you see and what I think that you should
> see differ by about 2X seems a little suspicious, as does the fact that
> minor tweaks seem to cause it to double again. 

Moreover, it compresses to almost exactly the same compressed size.

> (As if the image file
> is getting written twice.)  Before worrying about that, the first thing
> to do is to check to see whether ROOM and HEAP-UTILIZATION return figures
> that're approximately equal to the size of the .image file.  If they are,
> then you're saving 40/78 MB of live lisp data and the next step would be
> to figure out what that extra stuff is; if the figures differ significantly,
> then it seems likely that something else is going on in the process of 
> writing heap contents to disk.

I should mention that these are sizes with the prepended kernel.

Anyway, ccl has gone bonkers on me.

First, the size jumped to 87 megs.

Furthermore, if I make a small tweak to my "build.lisp" file which
controls
the fasl loading and dumping, ccl hangs while dumping the image.

If all I do is remove single (load "...") form from that file (which
wasn't
even there several days ago!). "ccl -l build.lisp" locks up after
writing
a bit over 17 megs. Put the (load "...") back and it finishes writing
the image (now 87 megs large). This is 100% reproducible so far.

The small file being loaded just contains two forms: an (in-package
...)
and a (defparameter ...) for a small datum. (The variable is referenced
in the program, but there is a DEFVAR for it too).

I tried re-installing CCL from the original ZIP I downloaded, but same
thing. I've blown off and re-built all the fasl's, rebooted the
machine. :)
Nothing.

I will try a different image.

Here is the heap utilization before dumping image.

(And, by the way, adding the (room) and (heap-utilization) calls
also causes the image dump to succeed! I don't have that (load ...)
form in the file, but it saves an image now.)

Why does 64% of my heap consist of 11 simple fixnum vectors???


                   Total Size             Free                 Used
Lisp Heap:       59244544 (57856K)    3276800 (3200K)    55967744
(54656K)
Stacks:           8265456 (8072K)     8262580 (8069K)        2876 (3K)
Static:          13586600 (13268K)          0 (0K)       13586600
(13268K)
807.500 MB reserved for heap expansion.
Object type                       Count     Logical size   Physical
size   % of Heap
                                             (in bytes)      (in bytes)
SIMPLE-FIXNUM-VECTOR                 11        44564480        44564568
    64.01%
FUNCTION                          16936         7808760         7910624
    11.36%
SIMPLE-BASE-STRING                62082         3930888         4302224
     6.18%
HASH-TABLE-VECTOR                   300         3862432         3864832
     5.55%
CONS                             214094         1712752         1712752
     2.46%
SYMBOL                            49378         1382584         1580096
     2.27%
SIMPLE-UNSIGNED-BYTE-VECTOR        6756         1400745         1443304
     2.07%
SIMPLE-UNSIGNED-LONG-VECTOR       11875          994484         1064360
     1.53%
SIMPLE-UNSIGNED-WORD-VECTOR        5921          787374          834744
     1.20%
SIMPLE-VECTOR                     19353          650644          771960
     1.11%
STRUCTURE                         18100          361320          501600
     0.72%
SLOT-VECTOR                        8155          341852          394632
     0.57%
SIMPLE-SIGNED-WORD-VECTOR             2          266480          266496
     0.38%
INTERNAL-STRUCTURE                 3896          108380          125248
     0.18%
STANDARD-INSTANCE                  7203           86436          115248
     0.17%
SIMPLE-BIT-VECTOR                   752           64714           70312
     0.10%
BIGNUM                             5936           36144           67688
     0.10%
LOCK                                367            8808           11744
     0.02%
MACPTR                              387            7588            9136
     0.01%
SHORT-FLOAT                         395            1580            3160
     0.00%
PACKAGE                              34            1088            1360
     0.00%
DOUBLE-FLOAT                         73             876            1168
     0.00%
BASIC-STREAM                         37             724             976
     0.00%
VALUE-CELL                           76             304             608
     0.00%
POPULATION                           22             268             360
     0.00%
DEAD-MACPTR                          17             204             272
     0.00%
POOL                                 23              92             184
     0.00%
ARRAY-HEADER                          5             140             160
     0.00%
RATIO                                 3              24              48
     0.00%
COMPLEX                               1               8              16
     0.00%
Total                            432190        68382173        69619880
   100.00%

This agrees closely with a dumped image.

A "blank" ccl looks like this:

? Object type                       Count     Logical size   Physical
size   % of Heap
                                             (in bytes)      (in bytes)
FUNCTION                          12751         5669212         5745968
    35.03%
SIMPLE-BASE-STRING                51320         3148000         3450680
    21.03%
CONS                             163944         1311552         1311552
     8.00%
SYMBOL                            40962         1146936         1310784
     7.99%
HASH-TABLE-VECTOR                    81         1064944         1065592
     6.50%
SIMPLE-UNSIGNED-LONG-VECTOR        9790          732972          790656
     4.82%
SIMPLE-UNSIGNED-WORD-VECTOR        5032          690942          731200
     4.46%
SIMPLE-VECTOR                     15297          513884          610160
     3.72%
STRUCTURE                         15694          322488          443616
     2.70%
SIMPLE-SIGNED-WORD-VECTOR             2          266480          266496
     1.62%
SLOT-VECTOR                        4929          212448          243856
     1.49%
SIMPLE-UNSIGNED-BYTE-VECTOR        4021          137644          161208
     0.98%
INTERNAL-STRUCTURE                 2676           69960           81640
     0.50%
STANDARD-INSTANCE                  4464           53568           71424
     0.44%
BIGNUM                             5296           30912           58424
     0.36%
SIMPLE-BIT-VECTOR                   545           44823           48856
     0.30%
LOCK                                129            3096            4128
     0.03%
MACPTR                              148            2816            3408
     0.02%
SHORT-FLOAT                         164             656            1312
     0.01%
DOUBLE-FLOAT                         73             876            1168
     0.01%
BASIC-STREAM                         36             696             944
     0.01%
PACKAGE                              15             480             600
     0.00%
POPULATION                           22             268             360
     0.00%
DEAD-MACPTR                          17             204             272
     0.00%
POOL                                 23              92             184
     0.00%
VALUE-CELL                           18              72             144
     0.00%
COMPLEX                               1               8              16
     0.00%
Total                            337450        15426029        16404648
   100.00%
? 

The image it dumps for me, without :prepend-kernel, is 16347152 bytes.

I instrumented the loading of my fasls. In ASDF, I did this:


(defmethod perform ((o load-op) (c cl-source-file))
  (mapcar #'(lambda (file)
              (load file)
              (format t "loaded ~a~%" file)
              (room))
          (input-files o c)))


It turns out that the heap size suddenly jumps after the very last file
is loaded.
It's kind of the central file of my program where most of the action
is.

;;
;; second-to-last file:
;;

loaded src/kdict.wx32fsl
Approximately 7,798,784 bytes of memory can be allocated
before the next full GC is triggered.

                   Total Size             Free                 Used
Lisp Heap:       19529728 (19072K)    7798784 (7616K)    11730944
(11456K)
Stacks:           8265456 (8072K)     8262024 (8068K)        3432 (3K)
Static:          13586600 (13268K)          0 (0K)       13586600
(13268K)
845.375 MB reserved for heap expansion.


;;
;; now watch: jump from 19072K to 57856K
;;

loaded src/mikan.wx32fsl
Approximately 3,276,800 bytes of memory can be allocated
before the next full GC is triggered.

                   Total Size             Free                 Used
Lisp Heap:       59244544 (57856K)    3276800 (3200K)    55967744
(54656K)
Stacks:           8265456 (8072K)     8262024 (8068K)        3432 (3K)
Static:          13586600 (13268K)          0 (0K)       13586600
(13268K)
807.500 MB reserved for heap expansion.
Approximately 3,276,800 bytes of memory can be allocated
before the next full GC is triggered.


What the heck have I done in there???

The software runs fine, otherwise. Solid, passes tests.






More information about the Openmcl-devel mailing list