[Openmcl-devel] sharing data between fortran and OpenMCL

[Gary Byers]

On Fri, 1 Oct 2004, Cyrus Harmon wrote:

> I suppose this might be a bit offtopic, but here goes anyway. So why
> does data get moved around by the GC? I'm sure this must have been a
> frequent topic for discussion on c.l.l. long ago, but I don't know the
> answer. I can see the arguments for the benefit of heap compaction a la
> the old MacOS memory management model, but what is it about GC that
> necessitates moving data? In the C world, one just mallocs and frees
> memory and lets the OS sort it out. Long-lived blocks stick around, and
> others don't but the data doesn't move. I can see certain applications
> (DB, file system, e.g.) where one would like to manage memory in a
> compact, fixed-size manner and provide for a nice compact memory
> layout, but with modern VMs, is this really such an issue for
> allocating/releasing LISP data?
>
> Thanks,
>
> Cyrus
>

Is locality of reference a good or bad thing ?  If it's a good thing,
what does the OS do to improve it for a typical C program ?

How good is the OS at sorting out the kinds of memory allocation that
lisp programs do ?

How do the memory allocation behaviors of a typical C program and
a typical lisp program compare ?

If a C-oriented memory-management system decided that relocation was
a good thing (offered benefits that outweighed its costs), how would
it implement such a scheme ?

If the GC is free to relocate objects, one arguable benefit is that
free memory is contiguous (and allocation is simple.)  If the GC wants
to reclaim gargage (make it available to the allocator) most reasonable
schemes for doing that would involve traversing free blocks and identifying
them as free (putting them on a free list or something similar); this
may mean that the GC has to do work proportional to heap size instead
of doing work proportional to the amount of live data, and the difference
may be substantial.

Back to adressing questions with questions: do C programs that
efficiently handle large amounts of short-lived and long-lived data
(GCC, for example) just use malloc and let the OS sort things out, or
do they use domain-specific memory allocation strategies that have
evolved and matured over a long period of time.

Someone famous (Knuth ?) tried to describe fragmentation by asking the
reader to envision a lunch counter where customers arrive individually
or in pairs and leave indivdually or in pairs, in more or less random
fashion.  How long does it take before a pair of arriving customers
can't be seated next to each other ?

It's possibly tempting to suggest that The Modern Lunch Counter of
Today solves this problem by having more seats.  (I've also seen
suggestions that the pattern of arriving customers isn't really
random, and that paying attention to that pattern and seating
customers carefully can minimize the problem.)  Making the lunch
counter bigger may delay the catastrophic effects of fragmentation,
but may also introduce other problems (the waiters/waitresses/VM system
have to walk farther to take and fill orders, coffee cups don't
get refilled as often, tips are lousy, the swap file gets bigger.)

There are other approaches to this problem, but I think that shoving
the customers around every now and then solves more problems than it
creates (and that the interesting questions revolve around how to do
the shoving.)  It does mean that it's meaningless for a Lisp customer
to tell a C customer (or anyone else) that it's in seat #17, since
the bouncer (GC) is more concerned about keeping the coffee cups full,
keeping the customers and wait staff happy, and not having to turn
away paying customers when a good hard shove would fix the problem.

If there are circumstances where it's really important that the address
of a lisp object remain fixed, I'd agree that there should be ways
of ensuring that.  MAKE-HEAP-IVECTOR sort of goes halfway: you get
a lisp object that's told to sit in the kitchen (where the bouncer
doesn't go), but you have to worry about throwing that customer
out at closing time and not throwing them out beforehand.  The other
option (I promise that this'll be the last strained analogy in this
message) would be to install a VIP seating area, where customers
who paid a little more for the privilege could be guaranteed to
keep their preferered seat until ... well, at least until all of
the other customers had forgotten about them and the bouncer had
made a special trip to escort them out.

I hope that it's clear that lisp memory management issues are best
visualized in terms of noirish Edward Hopper paintings ...