Locators

This chat discusses some problems with the current locator abstraction in libsystem.

It might actually be easy to implement a replacement for someone with a solid understanding of the issues.

15:01 < cgay> brucem: I'd prefer to fix join and make path-to-string use it.  Two birds...
15:02 < brucem> cgay: yeah, just not sure that's possible given the current mess.
15:03 < brucem> cgay: i'm loathe to make huge locator changes without housel.
15:10 -!- andreas3 [[email protected]] has quit [Remote host closed the connection]
15:11 -!- anannie [~chatzilla@unaffiliated/anannie] has quit [Quit: ChatZilla 0.9.90.1 [Firefox 27.0/20140130133824]]
15:13 -!- andreas1 [[email protected]] has joined #dylan
15:46 < prom> brucem: i'm all for fixing locators. the current ones don't deal right with lots of unix things.
15:46 < prom> brucem: ~ is badly implemented, path resolution functionality is limited...
15:46 < brucem> prom: i'm concerned that none of us really know them that well.
15:47 < prom> brucem: i sorta do. i read the docs many times and also the code. last time was when doing CLI stuff.
15:47 < prom> brucem: but really we just need to figure out something better.
15:50 < prom> something we can probably just kill: sources/system/locators/web-locators.dylan
15:51 < prom> related topics: content types (parsing and such, not a registry - at least for now).
15:55 < prom> bad things about locators: 1. incomplete unix support 2. platform-specific behaviour (file-system-separator - means you can't handle windows paths on unix - which is bad abstraction)
15:56 < prom> about 1: home directory addressing is incomplete, no support for realpath, no support for symlinks
15:56 < prom> 3. file system iteration is an art
15:57 < prom> 4. i would like to have watches (inotify on linux, libuv abstracts these things) <--- not an immediate issue
15:57 < prom> (just a use case for consideration)
15:58 < prom> "file-system-separator" also is a badie when you consider that URLs are simply more complex.
15:59 < brucem> prom: so who has something good enough to borrow from?
15:59 < prom> so are unix paths (~ and .. and .) as well as windows (\\ \ ... also special semantics for "." as in "\\.\COM19:")
16:00 < prom> brucem: java does it somewhat better. URLs and locators are entirely separate but can be converted... they don't do separator-independence terribly well but they only do that because their language can't do anything nice.
16:03 < prom> brucem: don't even think of looking at lisp. they do things better, AFAIK, but overly generic for historic reasons.
16:04 < prom> once upon a time in TOPS-20: PS:<root-directory>fnord.txt
16:04 < prom> same thing on VMS/OpenVMS: sys$sysdisk:[000000]fnord.txt;1
16:04 < prom> (note the version, TOPS-20 also has that like "fnord.txt.1")
16:05 < prom> and lisp does all that but it would just be ballast.
16:05 < prom> same as locators that try to deal with protocols and servers.
16:05 < prom> (windows is special here)
16:05 < prom> (same as nfs-like "host:/fnord/foo.txt" paths as genera has them)
16:05 < prom> but nobody really needs that.
16:07 < prom> a concrete problem i encountered in CLI: https://github.com/promovicz/dylan-cli/blob/master/cli/nodes.dylan#L338
16:07 < prom> locators also try to distinguish between files and directories before you can.
16:08 < prom> there is absolutely no way to tell if "foobar" is either.
16:08 < prom> its just a locator. that has nothing to do with the object type.
16:09 < prom> if you're lucky you can stat() the thing and then you know. but not before. locators instead act on incomplete information and/or force you to append "/" to things so it gets it.
16:10 < prom> also there is other ballast such as "file-property". something like that should sure exist but like that it can't work.
16:11 < prom> also when dealing with locators you also need to think about permissions. to my knowledge we don't support a create-time mode (third argument to open syscall).
16:12 < prom> at first we can probably skip user/group considerations, ACLs and extended attributes.
16:13 < prom> (user/group considerations meaning ownership and <posix-uid>/<posix-gid>)
16:13  * prom pastes this into a wiki page for later review.
16:15 < prom> i think i've noticed the important things. probably worst is distringuishing directory and file locators.
16:15 < prom> that causes havoc.
16:15 < prom> the rest is just cutting away stuff nobody ever used.
16:15 < prom> the IDE probably deals with some of these things in places. but that can quite possibly be supported by keeping the old stuff around.
16:16 < prom> not sure how many fixups it would need.
16:16 < cgay> prom: I was going to kill uri and use web-locators.  Is that a bad idea?
16:16 < cgay> use -> use/fix
16:17 < prom> cgay: yes and no. the intention is laudable. but i don't think the current locators abstract things well so having an independent URI/URL implementation is more useful.
16:17 < cgay> That said, I have never had any desire whatsoever to have a combined "locators" library in Python that deals with both pathnames and URIs.
16:18 < prom> cgay: but i would like to have a class hierarchy that contains both. a common superclass. i'd call it "file specifier" or something like that.
16:19 < cgay> prom: What about ONLY having URIs?  file:/// exists
16:19 < prom> cgay: i considered that but those don't support ~, relative paths and other such things.
16:19 < prom> cgay: however i think that it would be nice if you could always use a file url where you need a "file locator".
16:20 < prom> cgay: thus the common superclass.
16:20 < prom> cgay: maybe even a type union that allows for strings, but that really is debatable sugar.
16:20 < prom> (because it can lead to non-portability, java made that mistake)
16:20 < cgay> I fear the perfect may get in the way of the good enuf here, though.
16:21 < cgay> The uri library needs a lot of love still, too.
16:21 < prom> cgay: i don't think so. locators already overcomplicate things. really i want to get rid of most of it.
16:22 < prom> cgay: and fix some problems that we actually have. the CLI isn't the only place kludging around locators.
16:22 < cgay> prom: Well, you probably know my opinion of the locators library.
16:23 < prom> cgay: i would guess you don't like them. i liked them on first sight but learned to hate them when i tried to use them.
16:23 < prom> cgay: the directory/file thing really is the worst thing here. no point in being type-safe about the distinction if you can't make the distinction.
16:23 < cgay> +1
16:24 < cgay> I said essentially the same thing, but you put it much better.
16:24 < cgay> http://colabti.org/irclogger/irclogger_log/dylan?date=2013-08-21#l36
16:25 < prom> it took me a while to figure out. i think this is a case of somewhat dylan-typical "over-idealization of a prototype".

Some API sketching:

/**
 * Platform-independent filesystem path abstraction
 */

define abstract sealed primary class <path> (<object>)
end class;

define sealed primary class <posix-path> (<path>)
end class;

define sealed primary class <windows-path> (<path>)
end class;


/**
 * True if the path is independent of the working directory
 *
 * POSIX: path must start with "/" or a "~" construct
 * Windows: path must start with a drive letter
 */
define generic path-absolute?(path :: <path>)
  => (is? :: <boolean>);

/**
 * True if the path depends on the working directory
 *
 * Any path that is not absolute is relative.
 */
define generic path-relative?(path :: <path>)
  => (is? :: <boolean>);


/**
 * Converts the given path to a file: protocol URI
 */
define generic path-as-uri(path :: <path>)
  => (uri :: <uri>);

/**
 * Converts the given URI to a path
 *
 * The given URI must use the "file:" protocol.
 */
define generic uri-as-path(uri :: <uri>)
  => (path :: <path>);


/**
 * Resolves the given path using the working directory
 */
define generic resolve-path(path :: <path>)
  => (resolved :: <path>);

define generic canonicalize-path(path :: <path>)
 => (canonical :: <path>);


/**
 * Is the given file visible to the user?
 *
 * On POSIX files are hidden if their name starts with a ".", so we can decide without I/O.
 *
 * On Windows there is a flag for this in the filesystem, so this needs to stat the file.
 */
define generic file-hidden?(path :: <path>)
  => (is? :: <boolean>);

/**
 * Does the indicated object exist?
 */
define generic file-exists?(path :: <path>)
  => (is? :: <boolean>);

/**
 * True if the indicated object exists and is a device
 */
define generic file-device?(path :: <path>)
  => (is? :: <boolean>);

/**
 * True if the indicated object exists and is a directory
 */
define generic file-directory?(path :: <path>)
  => (is? :: <boolean>);

/**
 * True if the indicated object exists and is a file
 */
define generic file-file?(path :: <path>)
  => (is? :: <boolean>);

/**
 * True if the indicated object exists and is a link
 */
define generic file-link?(path :: <path>)
  => (is? :: <boolean>);


define generic create-directory(path :: <path>, #key recursive? :: <boolean> = #f)
  => ();