[Openmcl-devel] Fwd: Garbage collector - ccl:terminate (memory stuff)

[Grégory Vanuxem]

---------- Forwarded message ---------
De : Grégory Vanuxem <g.vanuxem at gmail.com>
Date: sam. 4 mai 2024 à 17:24
Subject: Re: [Openmcl-devel] Garbage collector - ccl:terminate (memory stuff)
To: Tim McNerney <mc at media.mit.edu>


Hi Tim,


Le mer. 1 mai 2024 à 16:52, Tim McNerney <mc at media.mit.edu> a écrit :
>
> Apologies for only half understanding this conversation. Are you trying to make sure C data structures are reclaimed in a hybrid C/Lisp application?


Please, no apologies. Moreover my issue is not easy to explain for me.

It's not that, I only need to dereference an entry (a Julia object) in
a hashtable in Julia. It is called via a C wrapper: FriCAS -> CCL ->
julia_ccl.lisp -> julia_wrapper.c -> libjulia.so ->
delete!(myhashtable, myindex). After, the Julia GC can do its job with
this no longer necessary object.

So, in FriCAS, when an object is no longer necessary FriCAS does its
job (hey CCL! you can reclaim your memory about this object when you
want, I trashed or lost it, I no longer use it, feel free! ;-))

In CL I only keep a reference to this Julia object in a very small CL
class instance that contains the Julia type and the Julia hashtable
reference key.

After, the CCL GC does its job, but **my problem here**, see this
discussion before, CCL:terminate is never called. Since it's in FriCAS
and saved CCL based images are produced during the FriCAS build it is
not simple to me. In fact I do not understand why terminate is never
called even if the terminate-when-unrecheable stuff is done, but
apparently it uses the original "(terminate (***) nil))". My last
e-mail was about that and I have definitely not found a way to really
use the terminate mechanism to let _me_ delete my hashtable entry
referenced by my key which is stored in the CL class (JLREF in
substance).

Hope that is more clear. Dubious though.

- Greg


>
>
> --Tim
>
> > On May 1, 2024, at 08:16, Grégory Vanuxem <g.vanuxem at gmail.com> wrote:
> >
> > Hello Peter,
> >
> >> Le dim. 28 avr. 2024 à 11:04, peter <p2.edoc at gmail.com> a écrit :
> >>
> >> Hi Greg,
> >>
> >> I'm struggling to get my head around quite what you are trying to do.
> >> But something that came to mind was from lisp machine lisp (Zetalisp, Genera, etc).
> >> There we could partition and allocate an area of memory, so we could use this area for our code, then GC it entirely in an instant. I have no idea whether such an approach is viable in CCL or SBCL, or whether that would be of use for you.
> >>
> >> My guess is that you are monitoring your RAM usage while your system is running, and seeing memory fill up. Effectively suffering a "memory leak".
> >
> > No, I have just added in the C wrapper to Julia a printf help::
> > ================================================
> > void jl_delete_wrapped_index(const char *index)
> > {
> >    jl_value_t *res;
> >    char* nstr = "";
> >
> >    if (!strcmp(nstr, index)){
> >        jl_printf(jl_stderr_stream(), "ERROR: delete(index): empty index\n");
> >        return;
> >    }
> >    //printf("Freeing... %s\n", index);
> >    res = jl_call2(del, refs, jl_cstr_to_string(index));
> >    //printf("%s freed\n", index);
> > =================================================
> >
> > At the CL level, the jlref class only holds the Julia specialized data
> > type as a string, and an index that refers to an element in a Julia
> > hash table. For Float64 or ComplexF64 (double-float-s) that is not a
> > problem I can call BLAS/LAPACK subroutines "directly" or special
> > functions applied on scalars so I do not need this hashtable but for
> > more elaborated data types I use the scheme used in the Julia
> > documentation (https://docs.julialang.org/en/v1/manual/embedding/),
> > that is, store the variable in an indexed hashtable.
> >
> >
> >>
> >> My thinking in general runs to just ensuring that there are no references left. I once was building a huge evolving tree structure and found myself throttled by lack of memory. Eventually I found just one entity in the whole still being referenced. Unfortunately that item also was circuitously connected to a great proportion of the remaining redundant material. My fault for keeping redundant pointers.
> >
> > Yes, clearly. I have some bad memories with trees ;)
> >
> > But, in fact I will have to later do some checks at the Julia level
> > for forgotten variables. For now, sometimes, I just print the hash
> > table after freeing all referenced FriCAS variables.
> >
> > In  mycase, I only need to store an index '(random
> > most-positive-fixnum)', so in fact I am playing if I can say that,
> > with two GCs. And when Clozure CL via FriCAS performs its job, what
> > I'm trying to do is delete the Julia hashtable entry to let the Julia
> > GC do its job.
> >
> >>
> >> I expect the above may be not much use to you, but it is interesting trying to tune in to your issues.
> >
> > Thanks for your response, my problem was probably not very well
> > worded. I am still struggling with this but I take a pause with this.
> > The only thing I have found interesting as of now seems to be the
> > mac-ui/ and objc-bridge/ code in the Clozure CL source tree. It has an
> > allusion to keep a ccl:terminate method after a save-application,
> > which is my case. I will look at this more deeply.
> >
>