Mountain Lion issues

[WestonHanners]

After I set up my remote and init'd my local bitpocket, I issued a 'bitpocket sync'

I got these warnings

"# Saving current state and backing up files (if needed)
| Root dir: /Users/whanners/BitPocket
mkdir: illegal option -- -
usage: mkdir [-pv] [-m mode] directory ...
ls: .bitpocket/backups/2013-04-16.071137: No such file or directory
ls: .bitpocket/backups/2013-04-16.071137: No such file or directory
rmdir: .bitpocket/backups/2013-04-16.071137: No such file or directory"

apparently, it also deletes anything I put in my bitpocket folder now when it syncs. Am i missing something here?

[WestonHanners]

I fixed the illegal option error by changing line 123 to mkdir -p instead of mkdir --parents. But I am still not able to push changes up. it just deletes the local copy.

[WestonHanners]

Update, after some experimentation, if I copy files directly into the master folder, they get synced to the slaves, but if I copy files into the slaves, they just get deleted. It appears to only be syncing in one direction.

[torfason]

Hi, it seems that some of your commands must have different APIs than what I have seen on my machines. I suspect the issue is that you are not running the GNU coreutils versions of cp/mkdir/rmdir/comm/touch/cat/sleep, although I don't want to completely rule out version incompatibility with bash/rsync/ssh or (even less likely) with grep/sed/kill. All of these are used with various options and we don't have the luxury of having autoconf/automake to figure out which are which.

I do have access to a Mountain Lion machine, but it has most of the the relevant tools (including GNU coreutils) installed through macports. I'd very much like to figure this out, I hate the idea that there might be data loss resulting from this.

Your tests with mkdir certainly brought us in the right direction, but could you perhaps, for completeness sake, post the output of the following commands:

bash --version
rsync --version
ssh -V
grep --version
sed --version
kill --version

cp --version
mkdir --version
rmdir --version
comm --version
touch --version
cat --version
sleep --version

I'll then try to take a look at bitpocket behavior on mountain lion with a set of commands that match these.

[WestonHanners]

bash: GNU bash, version 3.2.48(1)-release (x86_64-apple-darwin12)
rsync: rsync version 2.6.9 protocol version 29
ssh: OpenSSH_5.9p1, OpenSSL 0.9.8r 8 Feb 2011
grep: grep (BSD grep) 2.5.1-FreeBSD

none of the other commands generate version output

I am running this command via zsh, could this cause issues?

[torfason]

It seems pretty clear that this is caused by BSD versions of core tools (mkdir/sed/...) not behaving in the same way as the gnu versions. mkdir --parent is one example. --version is another (so your commands do not generate version output, probably because they are BSD, not GNU).

Another example:

$ cat rsyncoutput.txt | grep "^-|^d" | /opt/local/libexec/gnubin/sed "s:^\S_\s_\S_\s_\S_\s_\S_\s_:/:"
/.
/bla
/rsyncoutput.txt
$ cat rsyncoutput.txt | grep "^-|^d" | /usr/bin/sed "s:^\S_\s_\S_\s_\S_\s_\S_\s_:/:"
/drwx------ 170 2013/04/16 20:24:56 .
/-rw------- 0 2013/04/16 20:21:49 bla
/-rw------- 23 2013/04/16 20:24:56 rsyncoutput.txt

The second version is definitely not going to work. Too bad.

[torfason]

SOLUTIONS:

Immediate: Current version does not work with BSD versions of coreutils, and will cause data loss. You could (on mac) install macports and use that to install GNU coreutils (making sure GNU versions come before BSD versions in $PATH). It is also possible that a previous commit, 9622eac, works better, since that did not rely as heavily on coreutils.

Short-term: Unfortunately I think the only very short-term fix that I'm likely to apply is to test for GNU versions of coreutils and stop execution if they are not present.

Medium-term: I would like to make this robust to BSD, but that will take some work. Can't promise anything about time. I would try to respond quickly to a patch or pull request, though.

[torfason]

Just pushed an update that prevents bitpocket from running if it seems that sed is not behaving the way it needs to behave. I think there are ways to bypass the sed commands that cause issues but that requires some testing. Kronusdark, would you be up for doing that if I push a fix? (I'd push it to https://github.com/torfason/bitpocket to verify, before pushing it to the master repo).

[WestonHanners]

Sure push away

Sent from my iPhone

On Apr 17, 2013, at 8:41 PM, torfason [email protected] wrote:

Just pushed an update that prevents bitpocket from running if it seems that sed is not behaving the way it needs to behave. I think there are ways to bypass the sed commands that cause issues but that requires some testing. Kronusdark, would you be up for doing that if I push a new version?

—
Reply to this email directly or view it on GitHub.

[ku1ik]

👍

[rizwank]

Any plans to make this BSD (and therefore Mac) friendly?

[16n]

Actually this is pretty easy to solve.
Just install the gnu coreutils with homebrew. Homebrew will install everything prefixed with a g.
So if you want the GNU sed command on mac, you just have to use gsed.

I'm now in a testing phase. I don't really know if i should make a pull request or a new fork. What do you think @sickill?

[torfason]

I did a some work on making bitpocket work without requiring gnu tools, but going that route seemed to need more time than I had (even though that would really be the best solution, since one of the goals of bitpocket is maximum compatibility). So that's why there was no push to the repo and why this issue is still open.

But yes, installing gnu coreutils (and sed/grep - note that sed/grep are not part of gnu coreutils) is a workaround for now.

daseinhorn: What would your fix consist of? Changes that test for the existence of gsed/g* and use it instead of the standard tools if they are available? I'd be all for a pull request if that is what you are thinking.

[thmzlt]

Latest OS X (Mavericks) doesn't have GNU coreutils. I had to install coreutils and gnu-sed via hombrew, and run bitpocket using:

PATH="/usr/local/opt/coreutils/libexec/gnubin:/usr/local/opt/gnu-sed/libexec/gnubin:$PATH" bitpocket

Also, checking for sed only still lets me run bitpocket with an incompatible version of mkdir (for when I installed gnu-sed, but not coreutils).

[ku1ik]

This issue clearly shows that bitpocket depends on many external utilities which happen to be highly incompatible across multiple platforms. I think it would be good to implement detection of utility version and/or platform. For example bitpocket script could define functions named awk, sed that would call the proper (g)awk, (g)sed with proper args. Detection could happen as the first thing in the script and these awk/sed function would take actual binary names and proper switch names from variables set in the detection phase.

[SgtPooki]

Installing coreutils, gnu-sed and gnu-grep resolved this issue for me. Handling this automatically for future users would resolve some pain points.

[sje30]

hi @SgtPooki
when you installed these three packages, did you install them such that e.g. the binaries were called "sed" rather than "gsed"?

[greezybacon]

The #61 added support for macOS and was tested against macOS Sierra. Anyone is free to try the new release and offer commentary.