[Openmcl-devel] Decreasing the size of the reserved heap

[Gary Byers]

Gee, I don't remember ... I'd have to LOOK AT THE SOURCE, wouldn't I ?

(Yes, I realize that it's easier to do that if you're familiar with the
code than if you aren't, but there's not -that- much code involved and
it's hard to believe that it's -that- difficult ...)

Assuming that anyone interested in how CCL reserves memory for its
heap has found the function ReserveMemoryForHeap() in lisp-kernel/memory.c,
the (partial) answer to Tim's question is that it calls VirtualAlloc()
(on Windows) or mmap() (on Unix) with appropriate options.

Another part of the answer involves considering the difference between
"allocating memory that you don't access" and "allocating memory that
you can't access."

As I think/hope that I mentioned the other day, Windows will allegedly
allocate some kernel data structures (page table entries) whose size is
proportional to the size of region being mapped.  Linux claims to do
something similar unless the MAP_NORESERVE option is specified.

Aside from this case (PTEs), essentially no resorces (real or virtual
memory) are committed in order to describe this reserved memory region
at this time.  Very soon after this region is created, an image file
is read or mapped into that memory region and different subregions
(with different kinds of access permissions) have different costs
associated with them.  What those costs are and when they're paid can
get complicated fast and "overcommit" policies can make that even
more complicated, but until we're asking for resources to be committed
the fact that they may be speculatively committed or lazily committed
or otherwise overcommitted isn't really relevant.



On Thu, 9 Jan 2014, Tim Bradshaw wrote:

> On 8 Jan 2014, at 19:30, Gary Byers wrote:
>
>> On most 64-bit platforms, "reserving" 512GB of contiguous address space
>> costs essentially the same as reserving 1GB.  That "reserved address space"
>> isn't (yet) readable or writable; there's no physical of virtual memory
>> associated with it.
>
> How does CCL do this reservation?  I understand that allocating memory which you don't touch is really free, but there is usually a question of OS book-keeping, which can matter.  In particular are you assuming that the OS allows overcommit, or doing some other kind of reservation (I think even platforms which don't allow overcommit often have mechanisms of doing soft allocations of some kind which avoid the nasties associated with overcommit)?
>
> Sorry if this is an idiot question!
>
>