[Openmcl-devel] RUN-PROGRAM and memory overcommit

[R. Matthew Emerson]

On Feb 6, 2014, at 6:36 PM, Ron Garret <ron at flownet.com> wrote:

> The other day I got an out-of-memory error on a Linux machine when calling RUN-PROGRAM from a CCL process that was consuming about 75% of the available RAM.  The machine does not have a swap partition, and /proc/sys/vm/overcommit_memory was set to 0.
> 
> According to:
> 
> http://stackoverflow.com/questions/12924048/fork-memory-allocation-behavior
> 
> this error could be avoided by spawning the child process with vfork instead of fork.
> 
> Before submitting a ticket I thought I would ask: is there a reason that run-program uses fork instead of vfork?

Speaking generally, vfork() and threads don't get along.  For that matter, fork() and threads don't get along that well either. But in the vfork() case they really don't get along.

http://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439.html

I've wondered whether using posix_spawn() would be a better alternative, but there are apparently ways to lose there as well.

http://trac.clozure.com/ccl/ticket/687

In the really-far-out department, I've also thought that it might be possible to have a small run-program-helper process that the lisp kernel would start up before mapping the heap image.  There would be a lot of details to work out, but the idea is that the lisp could talk to the run-program-helper via some form of IPC and have it do the fork/exec dance on its behalf.  Since the helper would be tiny, having fork() duplicate its address space would be no big deal.

Someone (Stelian Ionescu?) might have already done something like this;  I remember talking to him about it at some lisp conference or other.