[Openmcl-devel] Another Directory Bug

[Gary Byers]

On Sun, 4 Apr 2010, wws2 new wrote:

>
> Gary,
>

> As always your comments help me to understand what choices and
> problems arise when designing these functions.

Huh.  And here I thought I was explaining why your claim that
there was a bug in DIRECTORY was ill-informed, and explaining
how to get the behavior you seem to want.

> From Terminal if I type " ls -al /Volumes/" I get a list of volumes
> including the startup volumes, with some sort of pointer to "/"
> which I takes means that it resolves to the same thing.

Do you think that it might be worth trying to understand what the pointer to "/"
actually means ?  (This information is written down and freely and widely
accessible; it's not some sort of secret.)

Hint: it's a symbolic link.

> At any rate, I get a list of the names of all "partitions" which is what I want.  The same applies if I go to the finder, or to disk-utility.
>

In the shell, if I do:

$ touch /Volumes/foo
$ mkdir /Volumes/bar
$ ln -s path-which-may-or-may-not-exist /Volumes/baz

have I mounted volumes named "foo", "bar", or "baz" ?

Of course not.

If I do:

$ mv /Volumes/Macintosh\ HD/ /Volumes/Other_Name

have I changed the name of a volume, or merely changed the name of its
mount point?  (Um, the latter.)

Whatever mechanisms the Finder and Disk Utility use to enumerate
mounted volume names (FSGetVolumeInfo, something in the
DiskArbitration framework, some other public or private API), they
pretty much have to involve something other than a directory traversal
of /Volumes; since volumes can be mounted and unmounted while any kind
of "volume enumeration" is taking place (news flash: more than one program
can run concurrently), any kind of reliable volume enumeration is more
complicated than that.

It's not unreasonable for you to decide to ignore these issues and 
try to obtain an approximate enumeration by traversing the /Volumes
directory.  It is unreasonable for you to try to do this without
understanding what that directory actually contains and without
listening when someone explains this to you.

> In CCL, I cannot duplicate that, ie I cannot find a way to list all
> partition names, and I think it should as it is aware of the name of
> the startup partition.  (directory "/Volumes/*" :directories t
> :files t :follow-links t) Gives me a list which now includes #P"/".
> (directory "/*" :directories t :files nil :follow-links t) Lists all
> directories on the startup.  The list is the same if I include :all
> t.  However, the following shows that at some level CCL knows the
> name of the startup.  (directory "/Volumes/Startup/*" :directories t
> :files t :follow-links t).

I already explained how to get the behavior that you expect.  I'll try
one last time, but no more.

Do you want to return the TRUENAME of the link to "/" in #p"/Volumes",
or do you want to return the link itself (whose filename matches the
name of the root volume) ?  (In other words, do you want to :FOLLOW-LINKS
or not ?  T or NIL ?)

All of your complaints above show that :FOLLOW-LINKS T does not result
in the behavior that you expect.  Doing the same thing over and over
and expecting different results is one definition of ... not understanding
a concept that requires very little effort to understand.

If you want the name of this link (rather than its TRUENAME) to be returned,
you want to say :FOLLOW-LINKS NIL.

> How do I get the name of the Startup partition?  I would like
> directory to provide it (at least with some set of parameters) so
> that I can use it in a less Unix, more Macintosh end-user way.  At
> the same time, with some other set of parameters, I would like it to
> give me everything in a Unix way.  In that way we have both.

As I said in my earlier message and just spent way too much time repeating,
DIRECTORY does provide a mechanism that allows the suppression of link resolution,
and whether you believe it or not or understand it or not, that's the isssue here.

What the Finder and other programs do to enumerate volumes has nothing to do
with traversing the /Volumes directory.  This is also true whether you believe
it or understand it or not.  (It is empirically, verifiably true.)

That said, I'm generally sympathetic to the idea of supporting a "mac like"
view of the system, but there are some issues and practical considerations.

The practical consideration first: it's not impossible to implement,
but there are technical reasons that make it pretty highly
impractical.  (Those reasons have to do with the frameworks that
provide other views of the filesystem and how those frameworks behave
with respect to "bundled" .apps and command-line applications.  The
short version of all of this is the mechanism that CCL uses to morph
from a command-line application into a bundled GUI .app wouldn't work
if the frameworks in question were always loaded by the command-line
program.  That'd be a fairly significant amount of functionality to
give up; unless one was willing to do so, we'd be talking about
"mac-like" extensions to DIRECTORY and other filesystem functions
that're only in effect in the IDE.  That might not be entirely
unreasonable, but before considering that question, it's probably
wise to ask what a "mac-like view of the filesystem is."

Someone who's been trapped in The Iron Lung Time Warp That Is MCL
for the last decade or so would probably identify a "mac-like view
of the filesystem" differently from The Rest Of Us.  To its credit,
MCL continued to provide support for things like HFS aliases and
HFS pathname syntax; until fairly recently (I don't remember when
exactly), it tried to maintain compatibility with OS9.  I was fairly
critical of some of Alice's decisions and of the pace at which MCL
moved forward, but the remaining MCL user community looked (from
the outside) to be extraordinarily change-resistant and not too
technically competent, and Alice had to be sensitive to those concerns.
That's an understandable and arguably correct decision, but I don't
think that characterizing MCL as having been stuck in a time warp
for a decade or more is unfair.  I absolutely, positively don't want
to hear someone say that about CCL 5-10 years down the road, and that
means that any "mac-centric filesystem layer" pretty much has to be
Cocoa-based, and if that means that means that people have to listen
and learn and abandon some knowledge that they acquired long ago, so
be it.

Look! No link-following!

? (#/contentsOfDirectoryAtPath:error: (#/defaultManager ns:ns-file-manager) #@"/Volumes" +null-ptr+)
#<NS-MUTABLE-ARRAY (
     HD2,
     "Macintosh HD"
) (#x15482A70)>

I think that we can safely conclude that #/contentsOfDirectoryAtPath:error:
isn't ANSI CL compliant.


> The way the name of the Startup partition is hidden at the moment,
> gets onto the problem of symbolic links, which is causing me other
> problems.  So far I cannot with my limited low-level understanding
> find a way of testing whether a path is an alias or not, or how to
> resolve it, something that MCL eventually provided.  My application
> uses that functionality a lot in order to traverse and manipulate
> directories of data, and avoid operating on things (aliases) that
> would fail or muck things up if attempted.  For example I can open
> an alias file named "a" to a folder, which somehow should fail...  >

For all intents and purposes, legacy HFS aliases (which are largely
similar in function to symbolic links but are very different in
implementation) are only visible to Carbon programs.  (They're
implemented in the subset of Carbon that hasn't been deprecated for
the last few years.)  As of 10.6, Cocoa gained the ability to determine
whether or not a path referred to an alias, and it's had the ability
to control whether "open" and "save" panels show aliases or their targets
for a while.  Predicting the future course of OSX isn't a good bet, but
I think that if the legacy alias technology was part of the future (or
even much of the present) it'd be better supported in Cocoa.

Symbolic links are part of the filesystem (not part of some framework)
and are therefore supported at the BSD layer and in Cocoa (and on other
Unix platforms and even in modern versions of Windows); aliases are
implemented in terms of resource forks and file types and other HFS[+]-
specific features.  (Features that themselves are rarely used in OSX
these days, outside of places like MCL.)

If you really need some of the extra functionality that aliases provide
and that isn't present in symbolic links - and believe that OSX will continue
to provide support for them in at least Carbon indefinitely - then it would
make sense to continue to use them and complain about the lack of support
for them in CCL and Cocoa.

If that doesn't seem like a very good tradeoff ... well, I'd certainly
agree with that.  (The additional functionality - notably, better
ability to fix broken links when the target moves, the ability to
request that offline volumes be mounted when an alias refers to
something on such a volume - just doesn't seem worth it.)

Other people might evaluate these tradeoffs differently.  I think that it's
fair to say that many people have been aware of these issues for some of all
of the last 10+ years; if you're just discovering them, that may mean that
MCL did a good job of keeping them hidden from you.  That doesn't mean
that

> ? (with-open-file (ert (probe-file (full-pathname "root:a"))) (print ert))
>
> #<BASIC-FILE-CHARACTER-INPUT-STREAM ("/Volumes/Arbeit/Allegro/a"/8 ISO-8859-1) #x300045CF725D>
> #<BASIC-FILE-CHARACTER-INPUT-STREAM ("/Volumes/Arbeit/Allegro/a"/:closed #x300045CF725D>
> ?
>
> I take your point that Unix has blurred the distinction between directories and files, but there should be a way of hiding that (logically they are equivalent) from some users.  I don't mind knowing about this blurred world, or working with it some, but at the end of the day my model of how my data and programs are spread across my partitions should match that of what I see and do as a user in the finder.
>
> Let me try to boil my posts and your reply into a suggestion and some requests.  I think it makes sense for  (directory.. ) to be capable of resolving aliases and getting past directories or filenames with "*" in them so that I can obtain a list of stuff on my partitions that correspond with what I see in the finder.  It can even throw in the ".DS_store" (IIRC it stores information about the file colors and positions users have applied in the finder...) and other hidden stuff as long as in each case there is a test function (like is-this-an-alias-p) that allows me to weed it out.  What I do at the moment, which works for files like ".DS_Store", is to always test the file-type, something I control for my own data-files but not in general of course.

Two things that you might find useful to keep in mind:

1) CCL runs on 5 different OSes; it's not the Mac-centric thing that MCL was
    and that you seem to be assuming it is.
2) There are lots of users whose needs and expectations are very different
    from yours.

OK, a third thing:

3) CCL is not currently a drop-in MCL replacement, intended to address the
    needs of non-technical users in ways that MCL did.  MCL was indeed very
    good at this, but it was a spectacular commercial failure: there just
    weren't enough such users and providing support to them was so resource-
    intensive that in any sane world Digitool would have folded long ago;
    Alice essentially didn't get paid for the last 10 years or so that she
    worked there.

    When former MCL users stagger out of  their timewarp and pronounce that
    they think that CCL should address their needs just like MCL did ... well,
    let's just say that that I don't remember that rosy Camelot with quite the
    same affection that these ... aliens and beggars do.  (For some reason,
    MCL refugees seem to think that they exist in large numbers; if this was
    true, things probably would have happened differently.)

> I would also like to get rid of the delete-duplicates and the
> alphabetical sorting inside of directory as that takes too long when
> manipulating hundreds of thousands of data-files.

DIRECTORY doesn't actually do any sorting, but I agree that DELETE-DUPLICATES
is too expensive in your case.

> Can we have a set of routines which tell me the file attributes like
> create-date, modify-date so I can sort them if I want to?  Apologies
> in advance if they already exist, but in general this layer seems
> missing at the moment.

FILE-WRITE-DATE is part of ANSI CL; CCL:FILE-CREATE-DATE is exported.  10 seconds
with APROPOS would have revealed this to you, as would common sense:

   1) It'd be pretty hard to write useful code without functions like this.
   2) People write useful code in CCL.
   3) As a user of a CL implementation, you unavoidably have -some- obligation
      to understand what's present in the language standard (FILE-WRITE-DATE.)
      You don't have an obligation to memorize the entire spec, but since the
      spec is freely available online, I think that you do have an obligation
      to know how to look things up yourself.

"Apologies in advance, but ..." may have meant something other than
"I'm too damned lazy or clueless to look this up for myself, so do it
for me.  For free", but I can't imagine what else that'd mean.

I've already wasted too much of my time here; please either make some sort of
effort or go somewhere else and waste someone else's time.