Need to define policy for hedging bets on the slippery reliance on underlying RawLocalFileSystem

[jayunit100]

We now have many instances of issues which can arise because the underlying RawLocalFileSystem version has semantics which are subtley different from HDFS.

Should we package a "stable" replication of RawLocalFileSystem implemented methods which copy over, from hadoop, the implementations that we know to be correct and specifically support those? With the various bugs out there around RawLocalFileSystem, it is a shame that we are so heavily dependenant on a classpath which we have no control over.

Examples: mkdirs , createNonRecursive, rename , and many other methods in RawLocalFileSystem are changing from version to version and it would be nice if we could have a stable library that we updated and included which has the latest and most reliable/correct semantics.

This is an open ended question - I'd like as many people as possible to comment on this and make an informed vote. and Ask more questions I havent fully explained the issue.

[childsb]

i realize its a PITA, but we should be opening JIRAs against the apache code w/attached fixes.

only with a JIRA in place would i consider 1-off code OK.

[anushshetty]

Wouldn't it bring more code management issues? By stable replication, do you mean using a stable branch, which also has the stable RawLocalFileSystem?