[Openmcl-devel] Decreasing the size of the reserved heap

[David L. Rager]

Hi Gary,

Indeed, I agree that worrying about 'top' output is a silly thing to
do.  That being said, sometimes clusters (indeed, one that I use but
do not have the rights to administer) use the amount of memory
allocated as an indicator of how large the heap might be expected to
grow.  If the cluster uses such a heuristic, it could prohibit any
additional jobs from running on the machine that might soon have a
process that will consume 512GB of memory.  Thus, all the CPUs and
most of the memory on that machine are being reserved for CCL, when,
in fact, CCL is only using one core and a couple GB of memory.

I admit that this is my difficulty, and not Clozure's.  Furthermore, I
tend to agree that Virtual Memory should be cheap to allocate and not
cause problems.  This being said, if there are some constants in the
source code that I could set to decrease the 512/128GB threshold, to,
say, 16GB, that would be helpful.

Thanks,
David



On Wed, Jan 8, 2014 at 1:30 PM, Gary Byers <gb at clozure.com> wrote:
> The -R option controls how much address space is reserved for CCL; tools
> like
> 'top' report this as if it was interesting, and it usually isn't.
>
> On most 64-bit platforms, "reserving" 512GB of contiguous address space
> costs essentially the same as reserving 1GB.  That "reserved address space"
> isn't (yet) readable or writable; there's no physical of virtual memory
> associated with it.  (I think that at least some versions of Windows
> will create page-table entries for reserved pages.)  The effect of
> reserving the address space is to make it difficult for random C code
> to use address space that CCL's heap may eventually want to use.
>
> 512GB seems like a lot today; most of us would have difficulty actually
> using that.  10 years from now (when we all have 1TB of some sort of
> memory in our wristwatches) it may not seem that way.
>
> For the last few years, the story's been a little more complicated
> than that: of that 512GB of reserved address space, a fixed amount
> (I think that it's indeed 128GB) is reserved for "static" things (things
> whose address won't change during a session, unlike the "dynamic" things
> that the GC may always be shuffling around.)  The -R option basically
> controls how much address space is reserved on top of that 129GB.
>
> (There are good reasons for making this 128GB a fixed limit; I don't
> remember all of them at the moment, and they aren't any more interesting
> than the rest of this.)  It is also true that using the -R option to change
> the size of the reserved region can confuse ROOM; we have an open ticket
> that complains about that and someday that'll probably be fixed.)
>
> Unless there's some other reason to cause tools like 'top' to print
> smaller numbers (e.g., to keep people from sending emails that say
> "OMG!  Top says that I'm using 512GB of virtual memory!  That must
> be why OSX is filling my disk with swap files!" [It isn't ...]), there's
> little reason that I can think of for changing the default on 64-bit
> platforms.
>
> On 32-bit machines, there's a whole lot less address space available;
> typically,
> the OS makes somewhere between 1 and 3 GB available to a userspace process,
> and CCL usually tries to reserve a fairly large chunk of what's available
> (leaving a smaller chunk for foreign/C code.)  In fairly rare cases, foreign
> code that tries to allocate memory (#_malloc, #_mmap, etc.) can have those
> attempts fail - not because there isn't enough physical/virtual memory
> available but because there isn't enough address space to put it in.)  The
> -R option was mostly intended to help deal with this case.
>
> Some early Linux x8664 kernels had difficulty reserving 512GB; from what
> I could tell (in emailing people using such machines) this was one of
> several
> problems those kernels had.  It's not 2004 anymore (or so I've heard.)
>
>
> On Wed, 8 Jan 2014, David L. Rager wrote:
>
>> Greetings,
>>
>> For bizarre reasons, I'm trying to get CCL to start with a lower
>> amount of allocated virtual memory.  But it seems that CCL is mostly
>> ignoring my request (according to top, CCL is still allocating 128g of
>> memory on Linux kernel 2.6.32-358.18.1.el6.x86_64).  I've spent a lot
>> of time searching for the cause, but it's time to ask whether there's
>> anything wrong with my command:
>>
>>   scripts/ccl64 -R 4000000000
>>
>> I know that the topic "Heap Allocation" (found at
>> http://ccl.clozure.com/manual/chapter17.3.html) uses the word "try" to
>> describe the attempt at only reserving small amounts of heap space.
>> But this is enough of a difficulty for my situation that I wanted to
>> ask anyway.  Is there something else I can try?  Is there a value in
>> the CCL source code that I can edit manually and then recompile?
>>
>> Thanks,
>> David
>> _______________________________________________
>> Openmcl-devel mailing list
>> Openmcl-devel at clozure.com
>> http://clozure.com/mailman/listinfo/openmcl-devel
>>
>>
>