[Openmcl-devel] Shark

[Gary Byers]

On Sun, 10 Apr 2005, Dan Knapp wrote:

>
>    Another anomaly is that Shark doesn't appear to understand Lisp
> function
> calls; so instead of a call tree, you get a flat list.  I gather this
> is because
> the actual call is done by funcall or some other system-wide trampoline?
> Unfortunately there may be no way to instruct Shark about that.

In OpenMCL, lisp functions pass arguments in and return values in a
completely different set of registers/stack locations than do C
functions.  Arguments that get passed on the stack get passed on a
different stack.

>
>    It also sometimes shows the caller as a child of the callee, which
> may be
> due to closures appearing to be part of the caller?

OpenMCL does use the same stack as C for the return address, however,
C and lisp code differ in whether the stack frame holding a given
return address is built by the caller or the callee.  A non-leaf C
function does something like the following on entry:

    (save-return-address-in-frame-built-by-caller)
    (build-frame-for-callees-arguments-and-return-address)

and on exit:

    (discard-frame-built-for-callees)
    (recover-return-address-and-jump-to-it)

In OpenMCL, the prologue is usually something like:

    (build-frame-and-store-return-address-in-it)

and the epilogue

    (recover-return-address-from-top-stack-frame-and-discard-top-frame-then-jump-to-return-address)	; whew!

I imagine that this difference is enough to confuse tools like Shark.


>
> > Agreed.  It'd be good to have something that worked with Shark (there
> > are ways of controlling Shark remotely, e.g.:
> >
> >    (unwind-protect
> >     (progn
> >      (start-shark-profiling-session)
> >      (long-computation))
> >     (stop-shark-profiling-session))
> >
> > which might be a little better than trying to press Control-Escape
> > quickly
> > enough), and there are also ways of accessing the kernel profiling
> > facility directly.
>
>    Does this really matter?  The process should be mostly sleeping when
> it's not
> executing user code, so that shouldn't get charged against it, right?
> ... But I'll
> look into how to do that.
>
>    It's almost certainly better to use Shark than to use the kernel
> profiler
> directly.

The advantages of using Shark are significant (the UI, its ability
to say interesting things about foreign code ...)

The disadvantages include those that you noted above (it doesn't
understand lisp calling conventions or register usage.)

>
>    Here's the symbols that turned up with identical addresses:
>
> common-lisp_symbol-macrolet
> 0x02000424
> 0x02000448
> }
> {
> common-lisp_throw
> 0x02000424
> 0x02000448
> }
>
>    Also at this address are:
>
> common-lisp_locally
> common-lisp_catch
> common-lisp_load-time-value
> common-lisp_multiple-value-call
> common-lisp_return-from
> common-lisp_let
> common-lisp_go
> common-lisp_macrolet
> common-lisp_multiple-value-prog1
> common-lisp_let*
> common-lisp_if
> common-lisp_setq
> common-lisp_flet
> common-lisp_eval-when
> common-lisp_the
> common-lisp_unwind-protect
> common-lisp_progn
> common-lisp_labels
> common-lisp_progv
> common-lisp_function
> common-lisp_quote
> common-lisp_tagbody
> common-lisp_block
>
> ... these are all builtins, so there would be no mystery if it were
> just these.  But there's
> user functions that have the same address, too:
>

They're all special operators.  FBOUNDP is supposed to return non-NIL
if its argument has any sort of "functional" interpretation.  In OpenMCL,
FBOUNDP returns "whatever's in the symbol's function cell", which may or
may not be a function (but will generally look enough like one to fool
any code trying to something like FUNCALL.)

? (fboundp 'if)
#(#<CODE-VECTOR  #x2000436> IF)
? (fboundp 'progn)
#(#<CODE-VECTOR  #x2000436> PROGN)
? (funcall 'if t t nil)
> Error in process listener(1): Special operator or global macro-function IF can't be FUNCALLed or APPLYed
? (fboundp 'or)
#(#<CODE-VECTOR  #x2000436> #<Compiled-function OR Macroexpander #x62470CE>)
? (funcall 'or t t nil)
> Error in process listener(1): Special operator or global macro-function OR can't be FUNCALLed or APPLYed

Special operators and macros all contain little function-like vectors
in their function cells.  These vectors all contain the same CODE-VECTOR
in their 0th element, and the job of that code vector is to signal an
error (because special operators and macros should never be "called" at
runtime in the same sense that regular functions can be.)

We should pretty much ignore that code vector.

> {
> parser__nodes-to-shift
> 0x02003F64
> 0x02003F80
> }
> {
> parser__nodes-to-reduce
> 0x02003F64
> 0x02003F80

I would guess that these are generic functions.

> }
> [30 more symbols in that package, including all the ones I actually
> want to profile]
> common-lisp_open-stream-p
> common-lisp_make-instances-obsolete
> common-lisp_stream
> common-lisp_simple-condition-format-arguments
> common-lisp_streamp
> common-lisp_arithmetic-error-operands
> common-lisp_unbound-slot-instance
> common-lisp_package-error-package
> common-lisp_echo-stream-output-stream
> common-lisp_debug
> common-lisp_input-stream-p
> common-lisp_type-error-datum
> common-lisp_two-way-stream-input-stream
> common-lisp_output-stream-p
> common-lisp_stream-error-stream
> common-lisp_class-name
> common-lisp_echo-stream-input-stream
> common-lisp_print-not-readable-object
> common-lisp_simple-condition-format-control
> common-lisp_arithmetic-error-operation
> common-lisp_type-error-expected-type
> common-lisp_stream-element-type
> common-lisp_two-way-stream-output-stream
> common-lisp_function-keywords
> common-lisp_cell-error-name
> common-lisp_method-qualifiers
> common-lisp_broadcast-stream-streams
> common-lisp_stream-external-format
> common-lisp_interactive-stream-p
> common-lisp_concatenated-stream-streams
> common-lisp_file-error-pathname
> common-lisp_synonym-stream-symbol
>

And these certainly are.

Generic functions also share the same code vector (there may be a few
different variants, but there's a pretty small set of them.)

When it's called, a generic function computes the effective method
(hopefully, it gets a cache hit and doesn't re-compute things that
it's already computed) and calls that method.  The generic function's
code basically just ensures that any incoming arguments that were
passed in registers on a stack and calls some functions in the guts
of CLOS (the "discriminating code").  We -might- want to profile
this little bit of glue code (I might ...), but we're probably
more interested in the time spent in the GF's methods than the
time spent in the GF itself (which is basically just boilerplate
code.)

To make a long story short, I think that we want to restructure
things a little:


(defun C-friendly-function-name (string package &optional flags)
  "Shark may or may not like lisp function names.  Mangle them
   here.  FLAGs might tell us how to do so."
   (replace ...))

(defun print-shark-spatch-record (symbol &optional (stream t))
  "May print more than one record per symbol"
  (flet ((emit-spatch-record (fn string)
          "FN is presumed to be interesting, and has a relatively
           unique code vector")
          (let* ((code-vector (ccl:uvref fn 0))
                 (start-address ...)
                 (end-address ...))
             (format stream "..." ...)))
   (when (and (fboundp symbol) (not (special-operator-p symbol)))
     (let* ((fn (or (macro-function symbol) (fdefinition symbol)))
            (name (symbol-name symbol))
            (package ...))
        (if (typep fn 'generic-function)
          (dolist (method (generic-function-methods fn))
            (let* ((method-function (method-function method)))
              (emit-spatch-record
                method-function
               (c-friendly-function-name (format nil "~s" method)
                                          package
                                         :method))))
           ;; Non-generic function
           (emit-spatch-record fn (c-friendly-function-name name)))))))

That's still kind of an "artist's conception", but I think that
it'll remove most of the duplicate code-vectors (and introduce
code-vectors for methods ...)