[Openmcl-devel] Garbage collector - ccl:terminate (memory stuff)

[Grégory Vanuxem]

Hello,

Many thanks for your response, and sorry for the delay, I am still
struggling with this.

Le sam. 20 avr. 2024 à 19:55, Ron Garret <ron at flownet.com> a écrit :
>
>
> > On Apr 20, 2024, at 5:12 AM, Grégory Vanuxem <g.vanuxem at gmail.com> wrote:
> >
> > Hello,
> >
> > I have long-standing issues with Clozure CL and its garbage collector
> > (GC). A few questions to ask if you can give me some hints about them.
> >
> > 1)
> > Is it possible to force a garbage collection in a simple manner? I do
> > not want to use this regularly, I just want to temporarily check its
> > work. That is I need to know when and if it really garbage collects
> > unreferenced personal variables(s).
>
> Call (GC).  Is that simple enough?  :-)

Yes, but I was not sure :-) In fact my main problem is using the class
mentioned before in FriCAS. The FriCAS interpreter runs in a package
"BOOT" which uses the "FRICAS-LISP" package which does not use CCL, so
I use ccl::gc to call it. I guess CL-USER uses CCL since (gc) is
callable directly.

>
> > 2)
> > The garbage collector reclaims memory in another thread than the
> > "main" thread? Am I right?
>
> Yes.

Good to know. I am writing a small interface to some specific
operations in Julia and calling Julia operations via "libjulia.so" is
not compatible with multiple external threads. So the GC thread(s) can
not directly (in fact via a C wrapper) call Julia operation. I
encountered this also with SBCL but SBCL has a different "terminate"
mechanism.

>
> > It is very difficult from my point of view to know in real time what it is doing.
>
> Yes, that's a feature.  The whole point of GC is so you don't have to worry about memory management.  The flip side of not worrying is not knowing.

Yes.

> > 3)
> > I need to know via the Clozure CL GC when memory will be reclaimed,
> > more precisely, if a variable is no longer referenced, and will be
> > GC-ed, I need to do other stuff on it but I see nothing happening
> > apparently. My code, copied/pasted from the documentation, seems not
> > effective.
>
> I'm not sure if this was just careless wording or a reflection of a real misunderstanding, but the phrase "a variable is no longer referenced" is non-sensical.  Variables are not referenced, *objects* are referenced *by* variables (and other things).  To be strictly correct, objects are referenced by variable *bindings*.

Careless wording, C language in mind when I wrote my mail.

> So, for example:
>
> (setf x (list 1 2 3))
>
> The call to LIST creates three objects, all of which are cons cells.  All are reachable through the value binding of the symbol X.  If I then do:
>
> (setf y (cons 4 x))
>
> I have now created a fourth object, another cons cell.  If I now do:
>
> (setf x nil)
>
> nothing happens with regards to GC because all of the objects I created above are still reachable via the value binding of the symbol Y.
>
> If I now do:
>
> (setf y nil)
>
> you would thing that all four of the cons cells I created become unreachable, and will be collected the next time GC runs.  However, there is a gotcha here: there are three standard variables in CL called *, ** and *** which keep track of the three previously returned values in the REPL.  So to actually make the four cons cells unreachable I have to perform three more interactions in the listener that do not reference these three variables, like this:
>
> ? (defclass foo () ())
> #<STANDARD-CLASS FOO>
> ? (let ((s *terminal-io*)) (defmethod ccl:terminate ((thing foo)) (print 'seeya s)))
> #<STANDARD-METHOD TERMINATE (FOO)>
> ? (make-instance 'foo)
> #<FOO #x30200277EDFD>
> ? (ccl:terminate-when-unreachable *)
> TERMINATE
> ? (gc)
> NIL
> ? 1
> 1
> ? 2
> 2
> ? 3
> 3
> ? (gc)
> NIL
> ?
> SEEYA
>
> Note that *standard-output* and *terminal-io* will by default send output to the altconsole when ccl:terminate runs.
>

Very interesting in terms of simplicity, clearly, thanks, but I can
not reproduce it in the BOOT package of FriCAS. I tried several
things, like using the CL-USER or the CCL packages but without
success. I even tried to directly define my 'jlref' class in the CCL
package but without success neither. In fact I am stuck here. May be
related to the Lisp reader?

For the remainder, the ccl:terminate method is never called, and there
is no error so I presume it is still defined to return 'nil' directly,
the Clozure CL source code, without executing my own code. Now it is:

(in-package "BOOT")

(defclass jlref ()
    ((id  :reader jlrefId   :initarg :id)
    (type :accessor jlrefType :initarg :type))
    (:default-initargs :id nil :type nil))

(defmethod print-object((obj jlref) stream)
    (print-unreadable-object (obj stream :type t :identity t)
        (princ (concatenate 'string " " (jlrefType obj)) stream)
        (princ (concatenate 'string " " (jlrefId obj)) stream)))

(defun |make_jlref| (str)
    (let* ((index (write-to-string (random most-positive-fixnum)))
            (id (|jl_setindex_wrap_eval_string| index str)))
        (if (not (equal id "")) ; unless str code is wrong
            (let ((ret (make-instance 'jlref :id id
                    :type (|jl_string_eval_string|
                        (concatenate 'string
"string(typeof(getindex(refs,\"" id "\")))")))))
                    #+:sbcl (sb-ext:finalize ret (lambda ()
                        (sb-concurrency:enqueue index *jqueue*))) ret)
            (error "Invalid Julia command"))))


#+:openmcl
(progn
(export (import (find-symbol "TERMINATE" 'ccl)))
(export (import (find-symbol "TERMINATE-WHEN-UNREACHABLE" 'ccl)))

(defmethod initialize-instance ((obj jlref) &rest initargs)
    (declare (ignore initargs))
    (call-next-method)
    (ccl:terminate-when-unreachable obj))

(defmethod ccl:terminate ((obj jlref))
    (print 'ABC *terminal-io*)
    (with-slots (id) obj
        (when id (queues:qpush *jqueue* id))))
)

As the code shows I also tried to import and/or also export
ccl:terminate and terminate-when-unreachable and the only thing that
seems to work is the terminate-when-unreachable since when tracing it,
it effectively returns "CCL:TERMINATE". I even tried
terminate-when-unreachable with two args, undocumented as far as I
know, that is, giving it my own function/method which does the
necessary job before the GC actions.

I wonder if someone encountered this, using CCL:terminate in a
different package and how she/he succeeded.

As a last note, there is a strange "routine" in the FRICAS-LISP
package (BOOT, the interpreter and algebra libraries use it) that I do
not really understand, wouldn't it be related? I can eventually "play"
with this piece of code (some code parts are cutted below):

(make-package "FRICAS-LISP"
     :use (list (or (find-package "COMMON-LISP")
                    "LISP")))

Here is the piece of code I do not totally understand and if it has
effect on CCL symbols that prevents me to define the ccl:terminate or
ccl::terminate method on jlref object:

;;; We use uninterned symbols because at this point we do not
;;; want to add symbols to FRICAS-LISP
(let ((#1=#:ls nil))
    (do-symbols (#2=#:el "FRICAS-LISP") (setf #1# (cons #2# #1#)))
    (mapcar (lambda (#3=#:x) (export (list #3#))) #1#)
)

And later:

;;; Main FriCAS package.  The interpreter and the algebra are run
;;; after switching to the boot package (in-package "BOOT") so any
;;; symbol that the interpreter or algebra uses has to appear here.
(make-package "BOOT" :use '("FRICAS-LISP"))

All the best,

- Greg