sync to linux systems

[se7en-x230]

I just found your script and I love it.

What I would like to do is sync my ~/.config and ~/.local folder to my cloud service.
I would want to sync the files from 2 exactly identical systems to the "backup" in the cloud as I edit sometimes files on my desktop and sometimes on my laptop.

This way I would have 2 identical setups and a backup in the cloud.
How would you set Zaloha up for that?

1. I set up 2 scripts with different source/target directories. one for .config one for .local
2. I would also like to add the script to cron to run once a day which is not possible as it requires input.

Maybe you have an idea how to do that.

Thanks a lot

[Fitus]

Hello,

thanks for your interest.

Firstly: yes, the synchronizations of the two directories .config and .local should be two separate synchronizations.
In the following, let's concentrate on only one of them:

If I understand you correctly, you want one backup to be fed from two sources? Huuh.

Let's initially think about updates of existing files: On source1, you edit an existing FILE_A.
Synchronize source1 to backup. Should result in an UPDATE action on FILE_A.
Then synchronize source2 against the same backup with the --revUp option.
If FILE_A on source2 is old (unchanged), then Zaloha should offer a REV.UP action to reverse-synchronize
FILE_A from backup to source2.

But what if FILE_A on source2 has been edited too? Then you have a conflict that should be resolved manually.

Then, let's talk about new files: On source1, you create a new file FILE_B.
Synchronize source1 to backup. Should result in a NEW action on FILE_B.
Then synchronize source2 against the same backup with the --revNew option. The FILE_B on backup is now
newer then the last run of Zaloha between source2 and backup, also Zaloha should offer a REV.NEW action
to reverse-synchronize FILE_B from backup to source2.

What is essential: the two Zaloha instances source1/backup and source2/backup must have different metadata
directories, because each instance must have its own remembered "last run" time in form of the 999 file timestamp.
Use the --metaDir option to achieve that.

Last: automatic operations. Let me tell you that what you want to achieve is something complex that needs human oversight
and eventual conflict resolution. We have not yet talked about removals and so on. I advice you to stay with the manual confirmations of Zaloha actions.

Yes simple backup cases can be automated. There is a section Automatic operations in the docu.
It is based on the --noExec option which tells Zaloha to just prepare the scripts and exit (and ask nothing).
You are then responsible for doing necessary sanity checks and execute the prepared scripts.

Regards, F.

[se7en-x230]

Yes you did understand my situation correctly.
I fully understand that it makes more sense to NOT automatically sync the folder.

If I understand correctly that would mean I can run one instance with the default settings from both sides to since into the same cloud folder.
No need for special settings as the default checks timestamps anyway.

That is quite straightforward.
If I edit a config file on one side I run the sync and update the cloud folder.

If I would then want to sync from the cloud to the other system I would again call another instant with the correct parth from the cloud to the system.

A synced scrip folder would need to be made executable again as I guess the permission would be lost.

Am I correct?

Nice work.

[se7en-x230]

I have also a problem with my .local folder.
Cleaning ............................................. done.
Checking .............................................
Zaloha AWK: (330_source_clean.csv FNR:1463) Unexpected, column 5 of cleaned file is zero for a file
Zaloha2.sh: Error on line 1526
test-local 2.88s user 2.63s system 53% cpu 10.402 total

I can backup .local/bin/ but not .local/ as a full directory. It breaks on .local/share/

[Fitus]

Hi,

I would make both desktop2cloud and laptop2cloud setups symmetrical. On the cloud, you need two metadata directories (I explain later): .Zaloha_metadata_config_desktop2cloud and .Zaloha_metadata_config_laptop2cloud. It is better to keep them outside of the backup directory.

In the following, for simplicity, I assume that all relative paths are relative to your home directory, on desktop, laptop and cloud.

Also, on desktop:

# Establish the SSH master connection
ssh -nNf -o ControlMaster=yes -o ControlPath='~/.ssh/cm-%r@%h:%p' 'user@cloud'

# Run Zaloha2.sh
./Zaloha2.sh --sourceDir=".config"          \
             --backupDir=".config"          \
             --backupUserHost='user@cloud'  \
             --sshOptions='-o ControlMaster=no -o ControlPath=~/.ssh/cm-%r@%h:%p'     \
             --scpOptions='-o ControlMaster=no -o ControlPath=~/.ssh/cm-%r@%h:%p -T'  \
             --revNew --revUp               \
             --metaDir=".Zaloha_metadata_config_desktop2cloud"

# Terminate the SSH master connection
ssh -O exit -o ControlPath='~/.ssh/cm-%r@%h:%p' 'user@cloud'

And on laptop:

# Establish the SSH master connection
ssh -nNf -o ControlMaster=yes -o ControlPath='~/.ssh/cm-%r@%h:%p' 'user@cloud'

# Run Zaloha2.sh
./Zaloha2.sh --sourceDir=".config"          \
             --backupDir=".config"          \
             --backupUserHost='user@cloud'  \
             --sshOptions='-o ControlMaster=no -o ControlPath=~/.ssh/cm-%r@%h:%p'     \
             --scpOptions='-o ControlMaster=no -o ControlPath=~/.ssh/cm-%r@%h:%p -T'  \
             --revNew --revUp               \
             --metaDir=".Zaloha_metadata_config_laptop2cloud"

# Terminate the SSH master connection
ssh -O exit -o ControlPath='~/.ssh/cm-%r@%h:%p' 'user@cloud'

Now, why you need the two metadata directories (two 999 files, each in one directory):
Assume all is synchronized and all mtimes are "yesterday":

Today, you create FILE_B on the laptop, mtime = 10:00.
At 10:30, you run the laptop2cloud synchronization.
FILE_B does not exist in the cloud, also will be copied to there (NEW).
Now you have FILE_B with mtime 10:00 and the 999 file in .Zaloha_metadata_config_laptop2cloud
with mtime 10:30 on the cloud.

Please note that the 999 file in .Zaloha_metadata_config_desktop2cloud still has mtime "yesterday".

Now run the desktop2cloud sync. The mtime 10:30 of FILE_B will now be compared with the mtime "yesterday"
of the 999 file in .Zaloha_metadata_config_desktop2cloud, resulting in a REV.NEW,
which is what you need.

If it were compared with the mtime 10:30 of the 999 file in .Zaloha_metadata_config_laptop2cloud,
the result would be REMOVE, logically.

Therefore you need two metadata directories.

Back to your two sources one backup idea: Zaloha is not, per design, a bi-directional synchronizer.
Basically, sourceDir is still the master and backupDir is still the slave.
The only two "goodies" which go in the opposite direction are the optional REV.NEWs and REV.UPs.
Removals, directory reorganizations and other actions always go from sourceDir to backupDir, never in the opposite direction. Hence, your idea will not work in such situations.

Permissions: there are the options --pMode and --pRevMode for keeping modes during sync.
All is described in the docu BTW.

Please keep this issue open, as a use-case based documentation, essentially.

Regards, F.

[Fitus]

Regarding the AWKCHECKER error: Zaloha AWK: (330_source_clean.csv FNR:1463) Unexpected, column 5 of cleaned file is zero for a file:

Column 5 is the file's last modification time (mtime), seconds since 01/01/1970. Zaloha checks that for files, it is numeric and not zero. Zero is a suspicious value for a file, so Zaloha aborts if it finds such case.

Please go into your .Zaloha_metadata_temp directory, open file 330_source_clean.csv and go to line 1463. This should be the line with the fifth column zero. What is this file? If you run the "stat" command on that file, what does it say? Hmmm.

Regards, F.

[Fitus]

Regarding the AWKCHECKER error: I think I have it. It is indeed possible to produce files that have modification time zero (01/01/1970). Proof here:

$ touch -t 197001010100 testfile
$ find testfile -printf '%Ts\n'
0
$ rm testfile

So the check is too stringent. On the other hand I want to check that FIND indeed reports meaningful mtimes on files and not zeros. Let me think about it. I make a new commit then.

Thanks and regards, F.

[Fitus]

Hi,

regarding the AWKCHECKER error: The commit is there. Now Zaloha reports a warning instead of an error, and only in case when all files on a device are reported with zero modification times by FIND. Individual files with zero modification times will not trigger the warning.

Please re-try and let me know.

Regards, F.

[se7en-x230]

Just for references. This is the line that breaks the script on .local/share/
/// S f 27199 1595203799 ext4 2050 8783024 1 seven users 755 0 cargo///sregistry///ssrc///sgithub.com-1ecc6299db9ec823///sproc-macro2-1.0.19///ssrc///swrapper.rs///s /// ///

I will check your new commit now

[se7en-x230]

works now. Thanks