[Openmcl-devel] Undocumented heap size limit?

[Gary Byers]

GC time's generally proportional to the amount of live data.  In CCL's
GC, there are some additional data structures - including bitvectors -
whose size is a small percentage of the total heap size, so the cost
is somehing like:

   (+ (* N size-of-live-object)
      (* small-percentage-of-N small-percentage-of-heap-size))

Until the heap gets very large, "a small percentage of it" usually
isn't very significant; a few percent of "many GB" is "many 10s of
MB", and of course "10MB here, 10MB there, pretty soon you're talking
about real memory."  For the sake of argument, let's say that GC
time is essentially/generally proportional to the amount of live data.

If you're saying that you're seeing that the GC concludes that all but
~40MB of 1.6GB is live/non-garbage stuff ... well, I don't know if that's
a correct conclusion or not, but the size of the heap shouldn't really
factor into it and I'm not aware of any way in which it does.  We have
users who run images larger than 100GB (on machines with more than 100GB
of RAM and several times that in swap space.)  GC in that environment is
expensive, but as far as I know it's productive (e.g., 10s of GB get
retained and 10s of GB get freed.)

If GC was productive with the smaller heap size but ceased to be productive
with a larger heap size, my inclination is to say that something else
changed.  I don't have a good idea of what that'd be.


On Mon, 26 Jul 2010, Bill St. Clair wrote:

> We've got a live server that spends its time pulling a bunch of XML
> feeds and parsing them into an in-memory "database". It also uses
> Weblocks to provide a web site on the information from the feeds. It
> runs nicely for periods of at least a few days, but still has a few
> memory leaks that require it to be occasionally restarted.
>
> We have it set up with the default GC thresholds. It's 64-bit, so it
> reserves 512 gigs of virtual memory from the OS. It fairly quickly fills
> up to about 160 megs of lisp heap, at which point it does a full GC
> about every 40 seconds, taking 1/3 to 2/3 of a second for that GC.
>
> This morning we discovered it with a heap size of about 1.6 gigs. It was
> spending most of its time in the GC, taking 6 to 12 seconds in each full
> GC to recover about 40 megs.
>
> Are we running up against some undocumented size limit, or is it more
> likely that a full GC in that big a heap just takes that long, so we've
> got to increase our ephemeral generation sizes (we're currently using
> the defaults) and/or eliminate our memory leaks and make the code cons
> less if we're not going to spiral into that black hole?
>
> -Bill
> _______________________________________________
> Openmcl-devel mailing list
> Openmcl-devel at clozure.com
> http://clozure.com/mailman/listinfo/openmcl-devel
>
>