[Openmcl-devel] documentation syntax

[Eli Barzilay]

Ron Garret <ron at flownet.com> writes:

> I think we may be talking past each other.

Sounds likely.


> Let me see if I can get us onto the same page.  Let me know if you
> agree with the following, and if not, where you begin to disagree:
>
> 1.  There is a distinction between S-expressions and the textual
>     representations of S-expressions.  The READ and PRINT functions
>     convert back and forth between S-expressions and their textual
>     representations.
>
> 2.  The READ and PRINT functions are modifiable, so it is possible
>     for the same S-expression to have different textual
>     representations (and vice versa, though that would probably be
>     bad engineering practice.  i.e. we should avoid syntactic
>     conventions that allows the same text to map onto multiple
>     S-expressions.)
>
> 3.  There is a canonical textual representation of S-expressions
>     defined by the Common Lisp Standard.
>
> 4.  Another possible textual representation for S-expressions is one
>     that follows the syntactic conventions of HTML/XML/SGML, to wit:
>
> <tag attr=val ...> content ... </tag>    ==> ((tag attr val ...) content ...)
>
>     with the optional simplification to (tag content ...) in the
>     no-attributes case.
>
> 5.  A third possible textual representation of S-expressions is
>     Scribble syntax.

Yes to all of the above.


> 6.  Many other major and minor variations on the theme are possible.
>     The use of balanced quotes is a minor variation that can be
>     optionally introduced as an element of many different surface
>     syntaxes for S-expressions.

Yes -- it is minor in both the implementation effort, and in the
gained utility.


> 7.  The task we have before us is to design a system that allows
>     collaborative development of documentation.

Yes, that was my understanding.  (To clarify my earlier disclaimer:
I'm a PLT user (and developer), I came in because Scribble was
mentioned, and I believe it has a huge potential for in-language
documentation systems for all languages in the extended lisp family.)


> 8.  There is general agreement that a Wiki is a good structure to
>     support such development.

I don't agree with this.  Wikis are good for getting collaboration by
virtue of being open to the public.  The various forms of markdowns
and/or limited HTML make them convenient for editing random pieces of
text, but they're not good when you want an organized and coherent
effort such as a manual.  For all of these attempts that I've seen one
of two things happened: either the result failed at being a good
high-quality document, or the initially loosely defined markdown
language grew up to be *much* heavier -- usually in a bad way.

Wikipedia is a good example: it has a markdown language that was
initially very lightweight; but if you look at the source of
practically any page, the amount of markup is practically the same as
any XML/HTML/Docbook/Latex document.  The lightweightedness feature of
the language is now completely burried below mountains of ad-hoc
notations, templates, and conventions, to the point of having bots
crawl pages and fix some mistakes.

In *my* opinion, this is a bad result.  The language is certainly
expressive enough (that's obvious: if it isn't fit for whatever, it's
immediately extended), but the cost is lack of consistency (making
copy/paste the only practical way to edit for most people).  Even
sticking to some XML format would have been better, since you'd no
longer need to learn the many ways content can be specified.

This is my subjective opinion, and you can disagree with that -- but
as a lisper I hope that you see how this corresponds to infix vs sexpr
syntax.  Having a uniform concrete syntax (= the reader/writer level)
is a huge benefit -- coming up with some random looping macro is easy
not only because you have macro, but because you're completely free of
the low-level decisions of the concrete syntax.  Because of this, I
believe that uniformity is a huge benefit and should be kept as a
matter of near axiom.

If you agree with this, then the question is what should this uniform
syntax be.  On one hand you have XML, with its usual bag of problems
and mis-features.  As I've seen in this thread, this (in combination
with docbook), got things to such a point where editing documents is
painful.  (And when this happens it becomes clear that something is
very wrong.)  On the other hand you have sexprs -- which are even more
uniform than XML, and have an additional benefit of being familiar for
the target crowd in this case.  But -- as Brian started this subthread
with -- they don't fit well when you need to write lots of free-form
text, and they're even worse when this text has lots of code.  This is
where the tradeoff game between convenience and uniformity kicks in,
and I believe that the scribble syntax is a good point wrt free-form
text in the same way sexprs are good for code.


> 9.  Wikis traditionally consist of two major components: a revision
>     control system, and a renderer that translates content from a
>     syntax designed to be easy to edit into one that can be
>     displayed by a browser.  Typically the editing syntax looks
>     something like Markdown, and the rendered syntax is HTML, though
>     there is no inherent reason why this should be so.  A Wiki could
>     just as well render to PDF, or Microsoft Word, or ascii art.

(Addressed above.)


> 10.  Traditionally, a given wiki is strongly bound to a particular
>      editing syntax.  Unlike the previous point, there *is* an
>      inherent reason why this is so: it is because the translation
>      from editing syntax to display syntax is done in one step by a
>      particular rendering engine that is part of the wiki, and so
>      pages must be edited in the syntax supported by the particular
>      rendering engine that is part of the wiki.

Yes, but you don't have to view it this way.  The HTML that modern
wikis produce is highly uniform, and much closer to a semantic
description.  You can therefore view the the rendered HTML as a piace
of uniform data that represents the text -- with an external CSS that
dictates the actual typesetting.  If you squint hard enough (= use an
xml->sexpr parser), this is already the sexprs that you talk about
below.


> 11.  Some wikis, Trac in particular, have rendering engines that can
>      be extended by the user.
>
> Are you with me so far?  If so, here is the key point:
>
> When I say, "Let's use S-expressions" I do not mean "Let's use the
> canonical textual representation of S-expressions (or some variant
> thereof) as the editing syntax of our wiki."  What I mean is: "Let's
> take advantage of the distinction between S-expressions and their
> textual representations."  If we do that, we can use a regular Wiki
> for rendering and revision control, but we don't need to agree on a
> surface syntax.  All we need to agree on is a structure for the
> underlying S-expressions (and we would need to agree on something
> analogous to that no matter we do if we're going to produce
> something coherent).

...right.  And (I assume) that the trac customizability is where this
can be specified.  In other words, I think that this already there, or
close enough to it, and the double-layered approach is (or should be)
obvious for anyone who uses lisp.

But the input syntax matters -- the cannonical sexpr that is used in
the intermediate level is not equivalent to the input, in the same way
that sexprs are not equivalent to actual code.  Limiting the input to
wiki-markdown is a mistake IMO, as I described above.  Saying that
this is only for now, and the system would allow alternative concrete
syntaxes for actual editing is an illusion that is pretty much
guaranteed to never materialize.


> Put this another way: what I'm suggesting at root is a change to the
> fundamental structure of a traditional wiki: make the rendering
> process a two-stage process instead of the one-stage process that it
> currently is.  Stage one is to go from (possible multiple) surface
> syntax to S-expressions, and stage 2 is to go from S-expressions to
> (again, possibly multiple) target syntax.  The primary advantages
> would be 1) we don't have to agree on a single editing syntax and 2)
> we can use the power of Lisp to process the intermediate
> S-expressions to, for example, do error checking, or expand macros.

(It's amusing how I replied to this paragraph before I read this
one...)


> All the issues of balances quotes, escaping and unescaping, and
> whatnot, are all issues of surface syntax design.  They are
> important issues, but they are dominated by the fact that if we do
> what I'm suggesting we don't have to reach consensus on them in
> order to make progress.

In practice, I think that you're heavily underestimating designing a
capable practical syntax, and I think that you're also too optimistic
about the chances of getting more than a single input syntax
implemented.

But since I'm ultimately an outsider, the only thing I could offer is
insight about the issues in designing such a system (and in using it
to migrate and produce a huge amount of documentation), and you can
easily dismiss my opinions as irrelevant.

-- 
          ((lambda (x) (x x)) (lambda (x) (x x)))          Eli Barzilay:
                    http://barzilay.org/                   Maze is Life!