[Openmcl-devel] Sudden increment in executable image size.
Kaz Kylheku
kaz at kylheku.com
Wed Nov 17 22:01:21 PST 2010
On Wed, 17 Nov 2010 20:42:44 -0700 (MST), Gary Byers <gb at clozure.com>
wrote:
> The fact that the file size that you see and what I think that you should
> see differ by about 2X seems a little suspicious, as does the fact that
> minor tweaks seem to cause it to double again.
Moreover, it compresses to almost exactly the same compressed size.
> (As if the image file
> is getting written twice.) Before worrying about that, the first thing
> to do is to check to see whether ROOM and HEAP-UTILIZATION return figures
> that're approximately equal to the size of the .image file. If they are,
> then you're saving 40/78 MB of live lisp data and the next step would be
> to figure out what that extra stuff is; if the figures differ significantly,
> then it seems likely that something else is going on in the process of
> writing heap contents to disk.
I should mention that these are sizes with the prepended kernel.
Anyway, ccl has gone bonkers on me.
First, the size jumped to 87 megs.
Furthermore, if I make a small tweak to my "build.lisp" file which
controls
the fasl loading and dumping, ccl hangs while dumping the image.
If all I do is remove single (load "...") form from that file (which
wasn't
even there several days ago!). "ccl -l build.lisp" locks up after
writing
a bit over 17 megs. Put the (load "...") back and it finishes writing
the image (now 87 megs large). This is 100% reproducible so far.
The small file being loaded just contains two forms: an (in-package
...)
and a (defparameter ...) for a small datum. (The variable is referenced
in the program, but there is a DEFVAR for it too).
I tried re-installing CCL from the original ZIP I downloaded, but same
thing. I've blown off and re-built all the fasl's, rebooted the
machine. :)
Nothing.
I will try a different image.
Here is the heap utilization before dumping image.
(And, by the way, adding the (room) and (heap-utilization) calls
also causes the image dump to succeed! I don't have that (load ...)
form in the file, but it saves an image now.)
Why does 64% of my heap consist of 11 simple fixnum vectors???
Total Size Free Used
Lisp Heap: 59244544 (57856K) 3276800 (3200K) 55967744
(54656K)
Stacks: 8265456 (8072K) 8262580 (8069K) 2876 (3K)
Static: 13586600 (13268K) 0 (0K) 13586600
(13268K)
807.500 MB reserved for heap expansion.
Object type Count Logical size Physical
size % of Heap
(in bytes) (in bytes)
SIMPLE-FIXNUM-VECTOR 11 44564480 44564568
64.01%
FUNCTION 16936 7808760 7910624
11.36%
SIMPLE-BASE-STRING 62082 3930888 4302224
6.18%
HASH-TABLE-VECTOR 300 3862432 3864832
5.55%
CONS 214094 1712752 1712752
2.46%
SYMBOL 49378 1382584 1580096
2.27%
SIMPLE-UNSIGNED-BYTE-VECTOR 6756 1400745 1443304
2.07%
SIMPLE-UNSIGNED-LONG-VECTOR 11875 994484 1064360
1.53%
SIMPLE-UNSIGNED-WORD-VECTOR 5921 787374 834744
1.20%
SIMPLE-VECTOR 19353 650644 771960
1.11%
STRUCTURE 18100 361320 501600
0.72%
SLOT-VECTOR 8155 341852 394632
0.57%
SIMPLE-SIGNED-WORD-VECTOR 2 266480 266496
0.38%
INTERNAL-STRUCTURE 3896 108380 125248
0.18%
STANDARD-INSTANCE 7203 86436 115248
0.17%
SIMPLE-BIT-VECTOR 752 64714 70312
0.10%
BIGNUM 5936 36144 67688
0.10%
LOCK 367 8808 11744
0.02%
MACPTR 387 7588 9136
0.01%
SHORT-FLOAT 395 1580 3160
0.00%
PACKAGE 34 1088 1360
0.00%
DOUBLE-FLOAT 73 876 1168
0.00%
BASIC-STREAM 37 724 976
0.00%
VALUE-CELL 76 304 608
0.00%
POPULATION 22 268 360
0.00%
DEAD-MACPTR 17 204 272
0.00%
POOL 23 92 184
0.00%
ARRAY-HEADER 5 140 160
0.00%
RATIO 3 24 48
0.00%
COMPLEX 1 8 16
0.00%
Total 432190 68382173 69619880
100.00%
This agrees closely with a dumped image.
A "blank" ccl looks like this:
? Object type Count Logical size Physical
size % of Heap
(in bytes) (in bytes)
FUNCTION 12751 5669212 5745968
35.03%
SIMPLE-BASE-STRING 51320 3148000 3450680
21.03%
CONS 163944 1311552 1311552
8.00%
SYMBOL 40962 1146936 1310784
7.99%
HASH-TABLE-VECTOR 81 1064944 1065592
6.50%
SIMPLE-UNSIGNED-LONG-VECTOR 9790 732972 790656
4.82%
SIMPLE-UNSIGNED-WORD-VECTOR 5032 690942 731200
4.46%
SIMPLE-VECTOR 15297 513884 610160
3.72%
STRUCTURE 15694 322488 443616
2.70%
SIMPLE-SIGNED-WORD-VECTOR 2 266480 266496
1.62%
SLOT-VECTOR 4929 212448 243856
1.49%
SIMPLE-UNSIGNED-BYTE-VECTOR 4021 137644 161208
0.98%
INTERNAL-STRUCTURE 2676 69960 81640
0.50%
STANDARD-INSTANCE 4464 53568 71424
0.44%
BIGNUM 5296 30912 58424
0.36%
SIMPLE-BIT-VECTOR 545 44823 48856
0.30%
LOCK 129 3096 4128
0.03%
MACPTR 148 2816 3408
0.02%
SHORT-FLOAT 164 656 1312
0.01%
DOUBLE-FLOAT 73 876 1168
0.01%
BASIC-STREAM 36 696 944
0.01%
PACKAGE 15 480 600
0.00%
POPULATION 22 268 360
0.00%
DEAD-MACPTR 17 204 272
0.00%
POOL 23 92 184
0.00%
VALUE-CELL 18 72 144
0.00%
COMPLEX 1 8 16
0.00%
Total 337450 15426029 16404648
100.00%
?
The image it dumps for me, without :prepend-kernel, is 16347152 bytes.
I instrumented the loading of my fasls. In ASDF, I did this:
(defmethod perform ((o load-op) (c cl-source-file))
(mapcar #'(lambda (file)
(load file)
(format t "loaded ~a~%" file)
(room))
(input-files o c)))
It turns out that the heap size suddenly jumps after the very last file
is loaded.
It's kind of the central file of my program where most of the action
is.
;;
;; second-to-last file:
;;
loaded src/kdict.wx32fsl
Approximately 7,798,784 bytes of memory can be allocated
before the next full GC is triggered.
Total Size Free Used
Lisp Heap: 19529728 (19072K) 7798784 (7616K) 11730944
(11456K)
Stacks: 8265456 (8072K) 8262024 (8068K) 3432 (3K)
Static: 13586600 (13268K) 0 (0K) 13586600
(13268K)
845.375 MB reserved for heap expansion.
;;
;; now watch: jump from 19072K to 57856K
;;
loaded src/mikan.wx32fsl
Approximately 3,276,800 bytes of memory can be allocated
before the next full GC is triggered.
Total Size Free Used
Lisp Heap: 59244544 (57856K) 3276800 (3200K) 55967744
(54656K)
Stacks: 8265456 (8072K) 8262024 (8068K) 3432 (3K)
Static: 13586600 (13268K) 0 (0K) 13586600
(13268K)
807.500 MB reserved for heap expansion.
Approximately 3,276,800 bytes of memory can be allocated
before the next full GC is triggered.
What the heck have I done in there???
The software runs fine, otherwise. Solid, passes tests.
More information about the Openmcl-devel
mailing list