Synchronize 2 folders

[xyplorerköln]

Thank you for taking the time to review what I wrote. Much appreciated.

Well, that's why I said back on February 25 that keeping track of state is really essential to a true sync.

Here's why:

If you brute-force "Two-Way Sync (Union)", then files that were meant to be deleted get repopulated accidentally. No good.
If you brute-force "Two-Way Sync (Intersection)", then unique files on both sides that are meant to be there will get deleted accidentally. No good.

For example, Dino took a bunch of new photos in the Field that Gertrude obviously does not have yet in the Office. (These folks didn't think to use Dropbox, of course.) With your proposed Intersection, all of Dino's new photos would get blasted into oblivion upon sync. That's obviously not too useful, either.

Here's my analysis:
1. For the average, extremely quick sync, one-way sync will probably work fine for most people.
2. For two-way sync, you really have to propagate deletions to make it useful. And to usefully propagate deletions (and additions), you really have to know which files were there before, which files were added intentionally, and which files were deleted intentionally. You can't figure that out by just looking at the files. You have to know what happened to the files over time.

How else would you know:
- That a file was deleted from one side and should be deleted on the other side, vs. being reintroduced;
- That a file was added on one side and should be added on the other side, vs. being deleted.

When comparing files for overwrite, you can always look at the timestamp to figure out the newer file. But there is no timestamp on a deleted (aka nonexistent) file. Thus tracking state is really the only way to do useful 2-way sync.

[xyplorerköln]

I do realize that you're not going to reinvent a full sync program in XYplorer. But as long as we're talking 2-way sync, I'd like to add some thoughts from GoodSync to the mix.

Quoting from https://www.goodsync.com/manual.html
-----------------------------------------------------------------
Synchronization
Synchronization ensures that two folders remain in sync while user may change files on both sides. This is achieved by copying changes that have been made on the left side to the right side and changes that have been made on the right side are copied to the left side.

Why Synchronization Is Not Trivial
Synchronization task would appear to be trivial (compare file modification files and let the newer file win) if it was not for deletions that need to be propagated. If you delete file on one side, GoodSync will propagate this deletion to the other side of sync job by deleting this file on the other side. The sophistication of GoodSync algorithm allows us to propagate deletions correctly through multiple chained folders.

Another complication is that user may decide to replace newer version of the file with its older version. GoodSync can handle this situation correctly too, it will propagate the older file version, if it detects that user made such replacement.

Synchronization Algorithm
This is a general outline of the synchronization algorithm (the actual algorithm is rather complex, it contains several important inventions and trade secrets that cannot be disclosed here):

Compute difference between current file state and stored file state for the left folder,
Compute difference between current file state and stored file state for the right folder,
Propagate changes that appeared on the left to the right side,
Propagate changes that appeared on the right to the left.
If change happened to same file on both sides, declare a Conflict.

GoodSync will correctly record file changes in states of all folders in which the changes occurred.

Sync State action
If file is the same on the left and right side of the sync but the stored state of the file is not the same then GoodSync would recommend a special Sync State action for such files.

The results of the Analyze step will inform you that Sync State will be done, if this is necessary.
You cannot change direction of Sync State action, as it is not directional action.
The following are the potential causes of the CopyState situation described above:
You synchronized two folders using programs other than GoodSync.
You copied files manually.
You did Delete State in GoodSync.

[Sebbel]

Thank you so much that you're working on this.
I only need One-Way Sync (with delete in target-panel).
But Two-Way Sync would be great too.
I'm excited.

[admin]

@xyplorerköln: That would be overkill for XYplorer. So, if your conception of 2-way sync is consensus, that would leave us with just 1-way sync. Advantage: So easy to code that it would be ready to ship on Monday.

[Filehero]

Let's evolve it - as always. Don will invent some completely new aspect along the weekend anyway.

[admin]

Does it make any sense to offer an option whether to delete the items to recycler or permanently?

[highend]

Recycle bin should be the default but an option for permanent deletion would be fine

[Filehero]

Does it make any sense to offer an option whether to delete the items to recycler or permanently?

Yes, options ar always good (be quick or be dead - at least 2)

I do imagine that there could be sensitive sync jobs where one rather prefers to trigger the digital death explicitly.

Cheers,
Filehero

[admin]

Here is my current interface design. This dialog will popup when you click on Sync Folders..., you can configure a bit, and when you OK it the process starts as configured.
I want it much simpler than Custom Copy on the surface. All properties of Custom Copy exist (it's the same machine), but most of the options are hard-coded internally (e.g. you will get Safe Overwrite, and a summary report). You will get full access to all options via scripting.

As you see I found it a cool idea to make the copying and the deleting optional (checkable). Gives you some interesting options. Below you see the factory defaults checked/chosen:

Code: Select all

Design "One-Way Sync"

Sync Folders
============

Source: [path]
Target: [path]

[x]  Copy items from source to target
     
     On name collisions:
     o Overwrite if newer
       Overwrite if different size or date
       Overwrite
       Skip
     

[x]  Delete items in target that have no matches in source
     
     o Delete to Recycle Bin
       Delete permanently

What do you think?

[Filehero]

What do you think?

It's fine for me.

Will there be a kind of (optional) report at file item level for a given sync run?

[admin]

Yes. Optional for scripting sync, hard-coded for GUI sync.

[Filehero]

Hopefully some of those other "sync lovers" will jump in here ...

[admin]

Hmm... quiet scene. I'll wait till tomorrow then the window closes...

[Fernando1958]

what about keeping the timestamp of last sync?
if file was deleted on the left and timestamp of the file on the right is older than LAST SYNC, the file should be deleted

also, it crossed my mind, if the file on the left and on the right have a timestamp after the LAST SYNC, a warning should be issue.

[admin]

what about keeping the timestamp of last sync?
if file was deleted on the left and timestamp of the file on the right is older than LAST SYNC, the file should be deleted

Wouldn't that mean that a file that was modified after last sync would survive this sync but not the next sync?