[Openmcl-devel] stack size

[Gary Byers]

In MCL, stacks were allocated in segments/chunks of (IIRC) 32K bytes each.
When a segment overflowed, a new segment was allocated and added to a linked
list of segments, the top frame (or two) was copied from the old segment to
the new, the appropriate stack pointer was adjusted, and control resumed.
(Most of what I remember about it is that it was more complicated than I
remember.)  When the total size of all segments in a stack exceeded some
settable threshold, a continuable error was signaled and the user had the
option of continuing with a larger threshold.  (I think that's the "nice
global variable" that David was asking about.)

This had some nice properties (most notably -from the user's point of
view- that there wasn't a fixed/hard limit on stack size), but there
were some significant costs associated with it as well.  The code that
handled transitions between stack segments depended on some
write-protected guard pages being written to, so the compiler had to
generate code which wrote to stack frames even in cases where this
wouldn't have otherwise been necessary (and that cost something).  Code
with "spiky" stack utilization tended to perform very poorly (Ackermann's
function was an extreme example of this, but there were also real-world
cases that spent a lot of time handling the faults that occured during segment
transitions), and (IIRC) an older Linux threads library kept thread-local
data at the bottom of a thread's stack and couldn't find that data if the
stack pointer wasn't pointing into the original segment.

All of these factors made it seem that segmented stacks were ultimately a
bad idea.  CCL's stacks are large(ish) and contiguous and their usable size
is fixed; there's a relatively small amount of stack memory between the
hard limit and the point at which an overflow condition is detected and
signaled (that's supposed to be enough to let you poke around in backtrace
and try to determine why the overflow occurred.)  If you try to recurse
deeply from a break loop after an overflow is signaled, the process will
just be terminated.

The "largish" stack sizes (around 1MB in 32-bit CCL and 2MB in 64-bit versions)
allow for some fairly deep recursion and seem adequate for most purposes (at
least I rarely hear people howling that they aren't), but if you really need
more you have to ensure that the stacks your code is running on are larger.
If that code is running on a thread that you create, see the documentation
of MAKE-PROCESS (and therefore of PROCESS-RUN-FUNCTION) for options that can
be used to control this and note that the options default to the values of
some unexported variables.

If your code is running in the initial listener thread (e.g., the REPL), a
command-line argument gives at least coarse-grained control over the size
of that thread's stacks.

[~] gb at abq> ccl64
Welcome to Clozure Common Lisp Version 1.9-dev-r15384M-trunk  (FreebsdX8664)!
? (room)
Approximately 33,292,288 bytes of memory can be allocated 
before the next full GC is triggered.

                    Total Size             Free                 Used
Lisp Heap:       40632320 (39680K)   33292288 (32512K)    7340032 (7168K)
Stacks:          10652256 (10403K)   10646880 (10397K)       5376 (5K)
Static:          24467056 (23894K)          0 (0K)       24467056 (23894K)
376793.250 MB reserved for heap expansion.
NIL
? (quit)
[~] gb at abq> ccl64 -Z 16M
Welcome to Clozure Common Lisp Version 1.9-dev-r15384M-trunk  (FreebsdX8664)!
? (room)
Approximately 33,292,288 bytes of memory can be allocated 
before the next full GC is triggered.

                    Total Size             Free                 Used
Lisp Heap:       40632320 (39680K)   33292288 (32512K)    7340032 (7168K)
Stacks:          47352400 (46243K)   47347024 (46237K)       5376 (5K)
Static:          24467056 (23894K)          0 (0K)       24467056 (23894K)
376793.250 MB reserved for heap expansion.
NIL
?

Note how the -Z option (which can also be specified as --thread-stack-size)
affects the total stack size that ROOM reports.

On Tue, 24 Jul 2012, David Cope wrote:

> 
> Is there a variable available to increase stack size? I'm running a fairly
> small program with few conses and while it's intensive, I find I'm running
> out of room with no apparent reason. 'm trouble shooting but if I can't find
> a good out, I thought a nice global variable like MCL used to have might
> save me.?
> 
> Thanks,
> 
> Dave C
> 
>