Synchronize 2 folders

Features wanted...
admin
Site Admin
Posts: 60357
Joined: 22 May 2004 16:48
Location: Win8.1 @100%, Win10 @100%
Contact:

Re: Synchronize 2 folders

Post by admin »

That would make one-way sync simpler, fine, but it would the relation between one-way sync and two-way sync more complex and confusing, since one-way sync overwrites unconditionally but two-way sync does not. Should they really both be called "sync"?

highend
Posts: 13274
Joined: 06 Feb 2011 00:33

Re: Synchronize 2 folders

Post by highend »

When you make a one-way sync you want the destination to be exactly the same as the source afterwards - or not?

This is still a sync (and not a backup) because backups don't delete files / folders
One of my scripts helped you out? Please donate via Paypal

admin
Site Admin
Posts: 60357
Joined: 22 May 2004 16:48
Location: Win8.1 @100%, Win10 @100%
Contact:

Re: Synchronize 2 folders

Post by admin »

Yes, makes sense.

So, sync means: Make source and target identical. There are options about how this identity is achieved.
1) one-way sync: target is made to look like source
2) two-way sync: source and target are sort of merged; in case of collisions there are rules that control which overwrites what.

highend
Posts: 13274
Joined: 06 Feb 2011 00:33

Re: Synchronize 2 folders

Post by highend »

Yep, good description. The debatable thing is: How many options
are necessary (for two-way syncs) to fit everybody's needs...
One of my scripts helped you out? Please donate via Paypal

xyplorerköln
Posts: 177
Joined: 01 Jan 2016 18:59

Re: Synchronize 2 folders

Post by xyplorerköln »

Hi there - I'd like to jump in on this a bit.

A quick note: I think the program GoodSync is probably the most comprehensive and well-built program to use as reference. (There are other, more- or less-complicated solutions, but this one seems the most directly relatable to the XY paradigm.) So you might review its version 10 just for gaining the vast thinking put into it, as it can handle just about every conceivable sync scenario and is quite elegant.

I do use tools like GoodSync. But the reason we want/need sync tools in XYplorer is to save time and simplify things. If we've already got employees bought into using XY, it's tedious to have them jump in and out of multiple programs to create settings and relationships with the same folders/files over and over again - when XY could be a one-stop-shop.

So here's what I would absolutely need in the sync solution:

1. The one-way sync would simply make (ABC > CD) into (ABC > ABC) as you say. If you could offer some GoodSync-like options (perhaps in a special settings pane) that would be a bonus, but for common use, it could just default to essentially mirroring.

2. The two-way sync would need to, at a minimum: (a) copy new files both left and right; (b) delete files on one side that were deleted on the other side; (c) overwrite older files with newer files. As with 1 above, additional settings and parameters would be great as options. Users would have the option to propagate deletions (and perhaps back up those deleted files somewhere for safety - but that's a nice-to-have). Importantly, XYplorer would need to keep track of file states, which GoodSync does through flat files, in order to facilitate deletions. The first time a sync between locations runs, it's a full sync: just copying new files and overwriting old files. Once XYplorer is aware of state, then subsequent syncs can delete files removed on one side.

Why is keeping track of state necessary? Imagine a situation where several syncs have been run between A and B. On the B side, a user manually deletes some files one day. If sync occurs without knowledge of state, then those files will simply be copied from A to B again. But if XYplorer keeps track of state, then it knows that B had those files but they were deleted; thus it knows to delete them from A.

Also:
- Pane-to-pane ad-hoc sync would be awesome. Literally just open two locations on left and right and sync them, without setting anything up.
- It would be cool to be able to set up stored Sync Jobs, in a manner similar to Catalog; basically keep a list of location-to-location syncs that can be fired off without opening panes.

Filehero
Posts: 2644
Joined: 27 Feb 2012 18:50
Location: Windows 10 Pro x64

Re: Synchronize 2 folders

Post by Filehero »

highend wrote:The debatable thing is: How many options
are necessary (for two-way syncs) to fit everybody's needs...
All robocopy has. :biggrin:


We already have
- mirror (one-way sync)
- union (two-way sync)

These are are my regular use cases (one-way sync -> backups, two-way-syncs -> keeping folders in sync across machines). And I think it's save to say these are the canonical sync operations.

For more options it boils down which parts of the Set theory you can/want additionally cover w/o confusing the average XYplorer user too much.

Only sometimes I have the scenario
- intersection (two-way sync)


The core (canonical above) sync operations should be exposed to the GUI, additional ones (if offered) could be by scripting only.


FH

admin
Site Admin
Posts: 60357
Joined: 22 May 2004 16:48
Location: Win8.1 @100%, Win10 @100%
Contact:

Re: Synchronize 2 folders

Post by admin »

OK, thanks all for the input! I'll enter a meditation phase now (combined with some carnival for enhanced enlightment), and then come up with a kick-ass solution.

Filehero
Posts: 2644
Joined: 27 Feb 2012 18:50
Location: Windows 10 Pro x64

Re: Synchronize 2 folders

Post by Filehero »

admin wrote: (combined with some carnival for enhanced enlightment),.
translation: 3 days completely off. :mrgreen:

Jerry
Posts: 805
Joined: 05 May 2010 15:48
Location: The UnUnited States of America

Re: Synchronize 2 folders

Post by Jerry »

I have long used AJCSync for my synchronization needs, for entire drives. So FWIW (no doubt more than FWIW), below are screen shots of the configuration panels.

Standard Logic (Left to Right)
AJCSync-standardLogic.png
AJCSync-standardLogic.png (62.05 KiB) Viewed 3421 times
Standard Logic (Both Ways)
AJCSync-standardLogic (BothWays).png
AJCSync-standardLogic (BothWays).png (69.52 KiB) Viewed 3418 times
Manual Logic
AJCSync-manualLogic.png
AJCSync-manualLogic.png (59.26 KiB) Viewed 3421 times
File Patterns
AJCSync-FilePatterns.png
AJCSync-FilePatterns.png (14.62 KiB) Viewed 3421 times
Transforms
AJCSync-Transforms.png
AJCSync-Transforms.png (53.14 KiB) Viewed 3421 times
Date Ranges
AJCSync-DateRanges.png
AJCSync-DateRanges.png (47.88 KiB) Viewed 3421 times
Time Stamps
AJCSync-Timestamps.png
AJCSync-Timestamps.png (50.71 KiB) Viewed 3421 times
Running on Windows 10 Pro 64-bit quad-core ASUS G752-VY notebook with 64 GB RAM, over 26 external USB3 drives attached via multiple powered hubs with letters and mount points, totaling 120+ TB.

Sebbel
Posts: 12
Joined: 29 Jul 2014 14:53
Location: Germany

Re: Synchronize 2 folders

Post by Sebbel »

Bump :cry:

admin
Site Admin
Posts: 60357
Joined: 22 May 2004 16:48
Location: Win8.1 @100%, Win10 @100%
Contact:

Re: Synchronize 2 folders

Post by admin »

Yes, I know I'm behind schedule... it will come...

admin
Site Admin
Posts: 60357
Joined: 22 May 2004 16:48
Location: Win8.1 @100%, Win10 @100%
Contact:

Re: Synchronize 2 folders

Post by admin »

Jerry wrote:I have long used AJCSync for my synchronization needs, for entire drives. So FWIW (no doubt more than FWIW), below are screen shots of the configuration panels.
Thanks Jerry (and sorry for the late reply). This will help me in designing it (although I will make it less complex).

admin
Site Admin
Posts: 60357
Joined: 22 May 2004 16:48
Location: Win8.1 @100%, Win10 @100%
Contact:

Re: Synchronize 2 folders

Post by admin »

OK, I'm coding it now. As I already said above I want to make it simple. There can be some options, and more options can be added later, but for starters I need a default setup for each sync method (there are two).

Basically, sync means: Make source and target identical. There are two basic methods to achieve identity; and options how to handle name-collisions.

1) One-Way Sync (Mirror): Target is made to look like source; one-way overwriting; unique items in the target are deleted.

Code: Select all

Before: Source=ABC, Target=CD
After:  Source=ABC, Target=ABC
Overwriting on name collision: 
a) Source always overwrites target. [Default]
b) If source is newer than target.
2) Two-Way Sync (Union): Both folders are merged; two-way overwriting; unique items are copied to the other side; nothing is deleted.

Code: Select all

Before: Folder1=ABC,  Folder2=CD
After:  Folder1=ABCD, Folder2=ABCD
Overwriting on name collision: The newer overwrites the older.
There will be GUI and scripting access to sync. GUI sync will make use of dual panes (active pane = source/folder1, inactive pane = target/folder2).

xyplorerköln
Posts: 177
Joined: 01 Jan 2016 18:59

Re: Synchronize 2 folders

Post by xyplorerköln »

Really excited you're now on this.

I'd like to point out, though, that the 2-way sync you describe (without deletions) isn't particularly useful in my view.

Why is that?

In a common use scenario, you have two folders that each represent ongoing work on the same content - including cleaning up those folders. For whatever real-world business reason (let's say you're dealing with a less tech-savvy operation and their ad hoc solution is to merge back and forth between a local and a remote location), the only option you have is to synchronize work.

So let's say there's a bunch of photos in a folder for a small design company. Dino has a local fileserver drive folder "Office" and then there's a portable "Field" USB drive folder that is used to access the photos when working in the field. While on business travel, Dino goes through the Field drive and deletes a bunch of old files he doesn't want anymore. When he gets back to the office, he wants to sync his work with the Office folder.

In the meantime, Gertrude has been making changes to the Office folder while Dino was out. She's added some new photos and deleted some photos.

If Dino were using AJCSync, GoodSync, 2BrightSparks SyncBackPro, or whatever other sync utility, he could achieve what he wants with almost no fuss.

But he's working in XYplorer, enjoying the fact that this sync feature is now built-in. So he chooses to 2-way-sync... and in minutes, all the work that Dino and Gertrude did pruning and cleaning the folder(s) is eradicated as XYplorerSync populates all the deleted photos right back into the folders.

Gertrude comes running around the corner screaming. Her world is now that much darker, thanks to XYplorer.

--------------------

In all seriousness, this is why the ability to choose to propagate deletions is crucial to 2-way sync. Without the <optional> possibility of propagating deletions, 2-way sync isn't very useful in the real world.

You might say, "I'm not reinventing the wheel here. If you want detailed features, go use those other tools." But then, that's a disruption to workflow and having to go manually apply locations in another tool, set up a sync job, then return to XYplorer, etc. If sync can happen within the XYplorer context, much time is saved and workflow is streamlined.

admin
Site Admin
Posts: 60357
Joined: 22 May 2004 16:48
Location: Win8.1 @100%, Win10 @100%
Contact:

Re: Synchronize 2 folders

Post by admin »

I see, so that would be this:

3) Two-Way Sync (Intersection): Both folders are intersected; two-way overwriting; unique items are deleted on each side.

Code: Select all

Before: Folder1=ABC, Folder2=CD
After:  Folder1=C,   Folder2=C
Overwriting on name collision: The newer overwrites the older.
OK?

PS: Looks like running One-Way Sync (Mirror) two times, once in each direction.

Post Reply