[Openmcl-devel] hemlock and tags files

[Gary Byers]

The fact that OpenMCL already records the source file associated with
most defining forms (and uses this information so that Meta-. works in
ILisp/SLIME) strongly suggests one implementation strategy.

(Yes, I realize that the information parsed into a tags file is
potentially different from the information that the lisp already
maintains.  No, I don't see a reason to maintain two sets of source
file info.)

Other people might prefer a tags file scheme, or having the choice
somehow (I'm not sure how well the two schemes would coexist, but
it might be possible to make them do so.)


On Thu, 7 Oct 2004, alex crain wrote:

> I'm getting ready to work on the source tagging features in hemlock. My
> immediate goals are to
> be able to click on a function and get either source code or
> documentation for the function.
>
> Documentation is moving along nicely and now I'm thinking about source
> code indexing.
>
> I want to model tagging after the emacs implementation, with the one
> added feature that I want
> to be able to do find-tag on some text and have hemlock produce
> pointers to all the source
> that defines that tag, maybe using a drawer or something to show the
> options. I get annoyed when
> I do find-tag and get referred to a piece of commented out code.

There is (of course) a tradeoff or two here.  If you maintain exact
source-file/byte position information for each definition, then yes,
that's helpful and the editor should be able to find that definition
with a high degree of reliability.

As soon as the file has been edited with an external editor, that
breaks down (at least a bit), and even if you only consider cases
where the file's only edited in Hemlock, you have to keep those
byte positions updated (perhaps by having translated them to marks
a bit earlier.)  In some cases, this may degenerate into a heuristic
search.
v
The other case - where you just try to associate the file with
the definition, without trying to maintain accurate position information -
sort of uses heuristic search as a starting point.  It avoids the
synchronization issues entirely, but how robust it is depends on how
good the heuristics that guide the search are.  This approach works
well some of the time and very poorly some of the time; the heuristic
search sort of needs to be a full-blown lisp parser to do a good job
of avoiding commented-out/conditionalized-out definitions.

I think that people who've thought about this before noticed that
a lisp-aware parser was already a part of a lisp-aware editor, and
that it was possible to "sectionize" (I -think- that that was the
term; it might have been "sectionalize") a buffer into regions or
region-like things.  If a buffer contains:

;;; -*- Mode: Lisp ... *-*

(defun fact (n)
  (if (zerop n)
    1
    (* n (fact (1- n)))))

;; I used to be able to do "(defun fib (" with my eyes closed.
(defun fib (m n)
  ;; whatever the definition of FIB is ..
  )

then the buffer would contain at least 2 sections.   Finding the
definition of FIB in that buffer doesn't need to start at the beginning
of the file and start searching.(and doesn't need to get confused
by the string "(defun fib" in a comment.).

This is a little tricker than it may sound: real buffers (while people
are editing in them) contain partial, syntactically invalid forms,
duplicate definitions, conditionalization, etc.  I think that a lot
of things (syntax coloring/highlighting, "list definitions", "track
changed definitions", ...) as well as meta-. would be easier to do
if the buffer had been sectionized (or maybe sectionalized) and if
editing changes maintained that information.

The other classic example is:

;;; -*- Mode: Lisp; Package: Foo -*-

(in-package "FOO")

(defun ...)
....

(in-package "BAR")
(defvar ...)

For many buffers, the notion that there's a "buffer package" is adequate;
for some, that isn't possible or practical.  There may be trickier
cases, but you'd sort of like the editor to behave reasonably in this
example (and cause the DEFVAR to be read in the BAR package.)


>
> Anyway, is there a way that I could implement this that would really
> piss people off?
> If so I'll avoid that one :-)
>
> More importantly, I'm fishing for ideas on the best way to do it. I'd
> like to utilized multiple
> tags tables, maybe on a per package basis, both to keep the namespace
> clean and
> to avoid regenerating tables for source that is largely static, like
> the OpenMCL core.

It's an oversimplification, but I think that any solution (external
tags files or using what's already there) eventually degenerates into
searching a buffer (with or without a strong hint about where to start
that search.)  I think that the important thing is to make that search
smarter.

>
> Anybody have any ideas one the best way to keep track of the files? One
> way might
> be to put a link to a table as meta-data in the source file and have
> the table updated
> whenever the file is saved. You could then have some concept of a
> project manager
> that would maintain preferences for groups of files and their
> associated indexes.

As files are loaded, the lisp records the source file associated with
each top-level definition.  The source-file information for everything
defined in OpenMCL is fairly complete.

The argument in favor of tags files is that they provide a mechanism
to record source file information about files that have not (yet)
been compiled.  The arguments against them is that they introduce
a different class of (and more) synchronization issues.

I've never missed the ability to navigate around (via meta-.) in the
sources of a system that I haven't loaded, but this may be more an
issue of what I'm used to and what my expectations are than a matter
of what The Right Thing is.  I would say that I think people are
more likely to be interested in the sources of code that they've
loaded and of OpenMCL itself than they would be of arbitrary other
systems, and that it's probably true that code you're running/working
on is more likely to be interesting than code that's in the filesystem
somewhere and has been indexed somehow.

>
> :alex
>
> _______________________________________________
> Openmcl-devel mailing list
> Openmcl-devel at clozure.com
> http://clozure.com/mailman/listinfo/openmcl-devel
>
>