[Openmcl-devel] Decreasing the size of the reserved heap

[David L. Rager]

Hi Gary,

Thanks very much for the script!  It seems to be working.

For anyone else using the script, I'll note that there's a typo in the
attachment that's easily fixed, and as Gary hinted in his explanation,
that starting CCL with >128GB of memory seems to be important (when I
started it with 4GB, I got lots of errors, regardless of the value I
gave to unmap-reserved-static-space).  To use the script I added it to
my ~/ccl-init.lisp and placed a call of
(ccl::unmap-reserved-static-space (expt 2 17)) at the end of that
file.  Also, I used a -R setting of 138G.  This yielded an image with
a VM footprint of around 12GB.

Thanks,
David

On Mon, Jan 13, 2014 at 10:07 PM, Gary Byers <gb at clozure.com> wrote:
> OK.  We discussed this briefly in IRC last week and I said that I'd look
> into an approach to getting the cluster and top to see less "virtual memory"
> being as being "in use" and get back to you this weekend.  (So, I took
> a long weekend ...)
>
> First of all, I memntioned that there was an apparent bug that caused
> ROOM to get confused when the -R option was used to reserve less than
> 128Gb; the code that handles that isn't really sure if that means "in
> addition to the 128GB reserved for static things" or "a very small
> amount, which is treated as "just a little more than 128GB, since
> we sort of need to reserve at least 128GB".  In addition, the space
> that's "reserved for heap expansion" includes both space for allocated
> data and space for data structurs that're used to keep track of that
> data.  (A bitvector that the GC uses to keep track of live objects
> needs 1 bit for every 128 bits of data; if we're dealing with hundreds
> of GB of data, we need to reserve a few GB of the total reservation
> for those auxiliary data structures.
>
> Starting CCL with
>
> $ ccl64 [other options[ -R 133gb
>
> seems to leave a bit over 800Mb to dynamically allocate things in
> and creates auxiliary data structures to suppport all of the static
> and dynamic data that could exist.
>
> The "static data" in question (what some of that 128GB is used for)
> includes data and code that's been "purified" by SAVE-APPLICATION
> (where "purified" in this context means "moved out of the GC's way")
> and room for "static conses".  As you (David) and some other people
> know, "static conses" (CONS cells whose address is guaranteed to
> never change) are used by some versions of ACL2 to support a function
> memoization scheme; they aren't generally interesting or useful otherwise.
>
> In most applications, most of that 128GB of address space that's
> reserved for static conses and purified code/data isn't used; if we
> unmap the address range that's reserved for the growth of those
> static regions, stupid things like whatever's managing your cluster
> and things that try to monitor use of "virtual memory" will think
> that we're using less virtual memory.   (We could also unmap some of
> the pages that we reserved for those "auxiliary data structures"; that's
> a few GB which may or may not matter to the stupid thing that's managing
> your cluster, but this whole message thread has gone on too long already
> and that's left as an exercise.
>
> It's worth remembering that most of the reason that we're reserving
> ranges of address space in the first place is to keep other (foriegn)
> code from using addresses that we think CCL might want to use (in the
> near or distant future.)  Those sorts of conflicts are much more likely
> to happen in a 32-bit environment than in a 64-bit one (less address
> space = greater chance of conflict), but the chance is still non-zero.
> (A bug that kept a Webkit demo from working reliably for a few OSX
> releases seems to have been caused by a Webkit Javascript JIT compiler
> wanting to use addresses that were already allocated by CCL).  If we
> unmap some of the pages we've reserved, we may not be able to safely
> map them again.
>
> So, to unmap some of those reserved pages and take our chances:  first,
> we need a little functon to find a CCL kernel data structure that
> descibes memory areas:
>
>
> (in-package "CCL")
>
> (defun find-area (code)
>   (let* ((p (%null-ptr)))
>     (%setf-macptr-to-object p (%get-kernel-global all-areas))
>     (if (eql (%get-object p target::area.code) code)
>       p
>       (do* ((q (%get-ptr p target::area.succ) (%get-ptr q
> target::area.succ)))
>            ((eql p q))
>         (when (eql (%get-object q target::area.code) code)
>           (return q))))))
>
> We can use that function to find the end of the memory region that was
> purified by SAVE-APPLICATION:
>
> (defun purified-end-address ()
>   (let* ((a (find-area area-managed-static)))
>     (if a
>       (%get-ptr a target::area.high)
>       (error "can't find the end of purified space"))))
>
>
>
> We can find the address of the start of the dynamic heap by several
> means; this way is fairly simple:
>
> (defun dynamic-heap-start-address ()
>   (%int-to-ptr (ash (%get-kernel-global heap-start) target::fixnumshift)))
>
> but (as they often do ...) static conses confuse things.  We allocate static
> conses in chunks of 32K each, and each static cons cell is 16 bytes in size,
> so the chuunks are (* 32868 16) = 512KB eack.  Whenever a chunk is
> allocated,
> the value returned by DYNAMIC-HEAP-START-ADDRESS will decrease by 512KB.
>
> (defun room-for-n-static-conses (n)
>   (* (logandc2 (+ 32767 n) 32767) target::cons.size))
>
> CCL::RESERVED-STATIC-CONSES returns the number of static conses that have
> been reserved since the lisp was created; it'll always be a multiple of
> 32768, so if we want to be be able to safely allocate N static conses
> (including whatever's been allocated) and don't want to risk address
> conflicts (or any other priblems that I'm not thinking of) we need
> to allow (DYNAMIC-HEAP-START-ADDRESS) to be lowered down to a target
> address:
>
> (defun target-dynamic-heap-start-address (nstatic-conses)
>   (%inc-ptr (dynamic-heap-start-address)
>      (- (room-for-n-static-conses (- nstatic-conses
> (reserved-static-conses))))))
>
>
> So, if we promise not to save an image or use more static conses than we say
> we will, we don't really need to keep the reserved memory between
> (PURIFIED-END-ADDRESS) and (TARGET-DYNAMIC-HEAP-START-ADDREESS N) and can
> ask the OS to make the intervening pages "free" rather than "mapped but
> not accessible".
>
> (defun unmap-reserved-static-space (nstatic-conses)
>   (let* ((start (purified-end-address))
>          (end (target-dynamic-heap-start-address nstatic-conses))
>          (nbytes (- (%ptr-to-int end) (%ptr-to-int start))))
>     (#_munmap start nbytes)))
>
> There's more stuff (a few GB of it) in bitmaps and "auxiliary data
> structures"
> that we could unmap, but it's probably a bit harder to find.  (You know what
> they say about the first 127GB being the easiest.)
>
>
>
> On Wed, 8 Jan 2014, David L. Rager wrote:
>
>> Hi Gary,
>>
>> Indeed, I agree that worrying about 'top' output is a silly thing to
>> do.  That being said, sometimes clusters (indeed, one that I use but
>> do not have the rights to administer) use the amount of memory
>> allocated as an indicator of how large the heap might be expected to
>> grow.  If the cluster uses such a heuristic, it could prohibit any
>> additional jobs from running on the machine that might soon have a
>> process that will consume 512GB of memory.  Thus, all the CPUs and
>> most of the memory on that machine are being reserved for CCL, when,
>> in fact, CCL is only using one core and a couple GB of memory.
>>
>> I admit that this is my difficulty, and not Clozure's.  Furthermore, I
>> tend to agree that Virtual Memory should be cheap to allocate and not
>> cause problems.  This being said, if there are some constants in the
>> source code that I could set to decrease the 512/128GB threshold, to,
>> say, 16GB, that would be helpful.
>>
>> Thanks,
>> David
>>
>>
>>
>> On Wed, Jan 8, 2014 at 1:30 PM, Gary Byers <gb at clozure.com> wrote:
>>>
>>> The -R option controls how much address space is reserved for CCL; tools
>>> like
>>> 'top' report this as if it was interesting, and it usually isn't.
>>>
>>> On most 64-bit platforms, "reserving" 512GB of contiguous address space
>>> costs essentially the same as reserving 1GB.  That "reserved address
>>> space"
>>> isn't (yet) readable or writable; there's no physical of virtual memory
>>> associated with it.  (I think that at least some versions of Windows
>>> will create page-table entries for reserved pages.)  The effect of
>>> reserving the address space is to make it difficult for random C code
>>> to use address space that CCL's heap may eventually want to use.
>>>
>>> 512GB seems like a lot today; most of us would have difficulty actually
>>> using that.  10 years from now (when we all have 1TB of some sort of
>>> memory in our wristwatches) it may not seem that way.
>>>
>>> For the last few years, the story's been a little more complicated
>>> than that: of that 512GB of reserved address space, a fixed amount
>>> (I think that it's indeed 128GB) is reserved for "static" things (things
>>> whose address won't change during a session, unlike the "dynamic" things
>>> that the GC may always be shuffling around.)  The -R option basically
>>> controls how much address space is reserved on top of that 129GB.
>>>
>>> (There are good reasons for making this 128GB a fixed limit; I don't
>>> remember all of them at the moment, and they aren't any more interesting
>>> than the rest of this.)  It is also true that using the -R option to
>>> change
>>> the size of the reserved region can confuse ROOM; we have an open ticket
>>> that complains about that and someday that'll probably be fixed.)
>>>
>>> Unless there's some other reason to cause tools like 'top' to print
>>> smaller numbers (e.g., to keep people from sending emails that say
>>> "OMG!  Top says that I'm using 512GB of virtual memory!  That must
>>> be why OSX is filling my disk with swap files!" [It isn't ...]), there's
>>> little reason that I can think of for changing the default on 64-bit
>>> platforms.
>>>
>>> On 32-bit machines, there's a whole lot less address space available;
>>> typically,
>>> the OS makes somewhere between 1 and 3 GB available to a userspace
>>> process,
>>> and CCL usually tries to reserve a fairly large chunk of what's available
>>> (leaving a smaller chunk for foreign/C code.)  In fairly rare cases,
>>> foreign
>>> code that tries to allocate memory (#_malloc, #_mmap, etc.) can have
>>> those
>>> attempts fail - not because there isn't enough physical/virtual memory
>>> available but because there isn't enough address space to put it in.)
>>> The
>>> -R option was mostly intended to help deal with this case.
>>>
>>> Some early Linux x8664 kernels had difficulty reserving 512GB; from what
>>> I could tell (in emailing people using such machines) this was one of
>>> several
>>> problems those kernels had.  It's not 2004 anymore (or so I've heard.)
>>>
>>>
>>> On Wed, 8 Jan 2014, David L. Rager wrote:
>>>
>>>> Greetings,
>>>>
>>>> For bizarre reasons, I'm trying to get CCL to start with a lower
>>>> amount of allocated virtual memory.  But it seems that CCL is mostly
>>>> ignoring my request (according to top, CCL is still allocating 128g of
>>>> memory on Linux kernel 2.6.32-358.18.1.el6.x86_64).  I've spent a lot
>>>> of time searching for the cause, but it's time to ask whether there's
>>>> anything wrong with my command:
>>>>
>>>>   scripts/ccl64 -R 4000000000
>>>>
>>>> I know that the topic "Heap Allocation" (found at
>>>> http://ccl.clozure.com/manual/chapter17.3.html) uses the word "try" to
>>>> describe the attempt at only reserving small amounts of heap space.
>>>> But this is enough of a difficulty for my situation that I wanted to
>>>> ask anyway.  Is there something else I can try?  Is there a value in
>>>> the CCL source code that I can edit manually and then recompile?
>>>>
>>>> Thanks,
>>>> David
>>>> _______________________________________________
>>>> Openmcl-devel mailing list
>>>> Openmcl-devel at clozure.com
>>>> http://clozure.com/mailman/listinfo/openmcl-devel
>>>>
>>>>
>>>
>>
>