[Openmcl-devel] sharing data between fortran and OpenMCL

[Cyrus Harmon]

Thanks for the informative and entertaining insight into this 
non-trivial topic.

Ok, so now we're probably officially way off-topic but in the hopes of 
convincing Gary that VIP seating for my matrices is a good thing (tm) I 
will continue off-topic for a moment.

> Is locality of reference a good or bad thing ?  If it's a good thing,
> what does the OS do to improve it for a typical C program ?

I'm hard pressed to think of examples where it is a bad thing.

> How good is the OS at sorting out the kinds of memory allocation that
> lisp programs do ?

Clearly the answer is dependent on the answer to the question "what 
kinds of memory allocation do lisp programs do?" Are lisp programs 
really so different than programs written in C, Perl, Python, R, 
etc...? Should the way the memory management system works be a function 
of the language or what the program is doing or both?

> How do the memory allocation behaviors of a typical C program and
> a typical lisp program compare ?

Is this a homework question? Lot's more temporary stuff on the stack in 
the typical LISP program? More middlin' size blocks of memory to keep 
track of typed-values. These are just guesses and I'm sure there are 
better answers. But is there really a "typical" lisp program? Is this 
what we want to optimize for?

> If a C-oriented memory-management system decided that relocation was
> a good thing (offered benefits that outweighed its costs), how would
> it implement such a scheme ?

Like OpenMCL does today? The problem arises when you mix cross the 
boundary between stuff that wants regular memory addresses (the OS, 
other libraries, etc...) and stuff that thinks it can move memory 
around. This always seemed like one of the big PITA's in the old MacOS 
(Carbon too?).

> If the GC is free to relocate objects, one arguable benefit is that
> free memory is contiguous (and allocation is simple.)  If the GC wants
> to reclaim gargage (make it available to the allocator) most reasonable
> schemes for doing that would involve traversing free blocks and 
> identifying
> them as free (putting them on a free list or something similar); this
> may mean that the GC has to do work proportional to heap size instead
> of doing work proportional to the amount of live data, and the 
> difference
> may be substantial.
>
> Back to adressing questions with questions: do C programs that
> efficiently handle large amounts of short-lived and long-lived data
> (GCC, for example) just use malloc and let the OS sort things out, or
> do they use domain-specific memory allocation strategies that have
> evolved and matured over a long period of time.

Sounds like a good case where one might actual gather real data rather 
than just speculating ;->

> Someone famous (Knuth ?) tried to describe fragmentation by asking the
> reader to envision a lunch counter where customers arrive individually
> or in pairs and leave indivdually or in pairs, in more or less random
> fashion.  How long does it take before a pair of arriving customers
> can't be seated next to each other ?
>
> It's possibly tempting to suggest that The Modern Lunch Counter of
> Today solves this problem by having more seats.  (I've also seen
> suggestions that the pattern of arriving customers isn't really
> random, and that paying attention to that pattern and seating
> customers carefully can minimize the problem.)  Making the lunch
> counter bigger may delay the catastrophic effects of fragmentation,
> but may also introduce other problems (the waiters/waitresses/VM system
> have to walk farther to take and fill orders, coffee cups don't
> get refilled as often, tips are lousy, the swap file gets bigger.)

Ahh! but the swap file provides a way to map a fragmented to heap to a 
compact on-disk representation, at least in theory. The coffee counter 
on the other hand...
>
> There are other approaches to this problem, but I think that shoving
> the customers around every now and then solves more problems than it
> creates (and that the interesting questions revolve around how to do
> the shoving.)  It does mean that it's meaningless for a Lisp customer
> to tell a C customer (or anyone else) that it's in seat #17, since
> the bouncer (GC) is more concerned about keeping the coffee cups full,
> keeping the customers and wait staff happy, and not having to turn
> away paying customers when a good hard shove would fix the problem.

Interesting.
>
> If there are circumstances where it's really important that the address
> of a lisp object remain fixed, I'd agree that there should be ways
> of ensuring that.  MAKE-HEAP-IVECTOR sort of goes halfway: you get
> a lisp object that's told to sit in the kitchen (where the bouncer
> doesn't go), but you have to worry about throwing that customer
> out at closing time and not throwing them out beforehand.  The other
> option (I promise that this'll be the last strained analogy in this
> message) would be to install a VIP seating area, where customers
> who paid a little more for the privilege could be guaranteed to
> keep their preferered seat until ... well, at least until all of
> the other customers had forgotten about them and the bouncer had
> made a special trip to escort them out.

I like the VIP lounge idea, but I would like it even better if there 
were a way to take mark somebody already at the counter as a VIP or 
friend of the owner who wouldn't get moved. One thing I don't like 
about the heap-ivector thing is that I (think I) am limited to what 
kinds of objects I can allocate this way. I would prefer it if I could 
take any lisp object and mark it is unrelocateable.
>
> I hope that it's clear that lisp memory management issues are best
> visualized in terms of noirish Edward Hopper paintings ...

Better than a Pollock I suppose.