[Openmcl-devel] Decreasing the size of the reserved heap

[Gary Byers]

The -R option controls how much address space is reserved for CCL; tools like
'top' report this as if it was interesting, and it usually isn't.

On most 64-bit platforms, "reserving" 512GB of contiguous address space
costs essentially the same as reserving 1GB.  That "reserved address space"
isn't (yet) readable or writable; there's no physical of virtual memory
associated with it.  (I think that at least some versions of Windows
will create page-table entries for reserved pages.)  The effect of
reserving the address space is to make it difficult for random C code
to use address space that CCL's heap may eventually want to use.

512GB seems like a lot today; most of us would have difficulty actually
using that.  10 years from now (when we all have 1TB of some sort of
memory in our wristwatches) it may not seem that way.

For the last few years, the story's been a little more complicated
than that: of that 512GB of reserved address space, a fixed amount
(I think that it's indeed 128GB) is reserved for "static" things (things
whose address won't change during a session, unlike the "dynamic" things
that the GC may always be shuffling around.)  The -R option basically
controls how much address space is reserved on top of that 129GB.

(There are good reasons for making this 128GB a fixed limit; I don't
remember all of them at the moment, and they aren't any more interesting
than the rest of this.)  It is also true that using the -R option to change
the size of the reserved region can confuse ROOM; we have an open ticket
that complains about that and someday that'll probably be fixed.)

Unless there's some other reason to cause tools like 'top' to print
smaller numbers (e.g., to keep people from sending emails that say
"OMG!  Top says that I'm using 512GB of virtual memory!  That must
be why OSX is filling my disk with swap files!" [It isn't ...]), there's
little reason that I can think of for changing the default on 64-bit
platforms.

On 32-bit machines, there's a whole lot less address space available; typically,
the OS makes somewhere between 1 and 3 GB available to a userspace process,
and CCL usually tries to reserve a fairly large chunk of what's available
(leaving a smaller chunk for foreign/C code.)  In fairly rare cases, foreign
code that tries to allocate memory (#_malloc, #_mmap, etc.) can have those
attempts fail - not because there isn't enough physical/virtual memory
available but because there isn't enough address space to put it in.)  The
-R option was mostly intended to help deal with this case.

Some early Linux x8664 kernels had difficulty reserving 512GB; from what
I could tell (in emailing people using such machines) this was one of several
problems those kernels had.  It's not 2004 anymore (or so I've heard.)

On Wed, 8 Jan 2014, David L. Rager wrote:

> Greetings,
>
> For bizarre reasons, I'm trying to get CCL to start with a lower
> amount of allocated virtual memory.  But it seems that CCL is mostly
> ignoring my request (according to top, CCL is still allocating 128g of
> memory on Linux kernel 2.6.32-358.18.1.el6.x86_64).  I've spent a lot
> of time searching for the cause, but it's time to ask whether there's
> anything wrong with my command:
>
>   scripts/ccl64 -R 4000000000
>
> I know that the topic "Heap Allocation" (found at
> http://ccl.clozure.com/manual/chapter17.3.html) uses the word "try" to
> describe the attempt at only reserving small amounts of heap space.
> But this is enough of a difficulty for my situation that I wanted to
> ask anyway.  Is there something else I can try?  Is there a value in
> the CCL source code that I can edit manually and then recompile?
>
> Thanks,
> David
> _______________________________________________
> Openmcl-devel mailing list
> Openmcl-devel at clozure.com
> http://clozure.com/mailman/listinfo/openmcl-devel
>
>