Synchronize 2 folders
-
- Site Admin
- Posts: 60357
- Joined: 22 May 2004 16:48
- Location: Win8.1 @100%, Win10 @100%
- Contact:
Re: Synchronize 2 folders
That would make one-way sync simpler, fine, but it would the relation between one-way sync and two-way sync more complex and confusing, since one-way sync overwrites unconditionally but two-way sync does not. Should they really both be called "sync"?
FAQ | XY News RSS | XY Twitter
Re: Synchronize 2 folders
When you make a one-way sync you want the destination to be exactly the same as the source afterwards - or not?
This is still a sync (and not a backup) because backups don't delete files / folders
This is still a sync (and not a backup) because backups don't delete files / folders
One of my scripts helped you out? Please donate via Paypal
-
- Site Admin
- Posts: 60357
- Joined: 22 May 2004 16:48
- Location: Win8.1 @100%, Win10 @100%
- Contact:
Re: Synchronize 2 folders
Yes, makes sense.
So, sync means: Make source and target identical. There are options about how this identity is achieved.
1) one-way sync: target is made to look like source
2) two-way sync: source and target are sort of merged; in case of collisions there are rules that control which overwrites what.
So, sync means: Make source and target identical. There are options about how this identity is achieved.
1) one-way sync: target is made to look like source
2) two-way sync: source and target are sort of merged; in case of collisions there are rules that control which overwrites what.
FAQ | XY News RSS | XY Twitter
Re: Synchronize 2 folders
Yep, good description. The debatable thing is: How many options
are necessary (for two-way syncs) to fit everybody's needs...
are necessary (for two-way syncs) to fit everybody's needs...
One of my scripts helped you out? Please donate via Paypal
-
- Posts: 177
- Joined: 01 Jan 2016 18:59
Re: Synchronize 2 folders
Hi there - I'd like to jump in on this a bit.
A quick note: I think the program GoodSync is probably the most comprehensive and well-built program to use as reference. (There are other, more- or less-complicated solutions, but this one seems the most directly relatable to the XY paradigm.) So you might review its version 10 just for gaining the vast thinking put into it, as it can handle just about every conceivable sync scenario and is quite elegant.
I do use tools like GoodSync. But the reason we want/need sync tools in XYplorer is to save time and simplify things. If we've already got employees bought into using XY, it's tedious to have them jump in and out of multiple programs to create settings and relationships with the same folders/files over and over again - when XY could be a one-stop-shop.
So here's what I would absolutely need in the sync solution:
1. The one-way sync would simply make (ABC > CD) into (ABC > ABC) as you say. If you could offer some GoodSync-like options (perhaps in a special settings pane) that would be a bonus, but for common use, it could just default to essentially mirroring.
2. The two-way sync would need to, at a minimum: (a) copy new files both left and right; (b) delete files on one side that were deleted on the other side; (c) overwrite older files with newer files. As with 1 above, additional settings and parameters would be great as options. Users would have the option to propagate deletions (and perhaps back up those deleted files somewhere for safety - but that's a nice-to-have). Importantly, XYplorer would need to keep track of file states, which GoodSync does through flat files, in order to facilitate deletions. The first time a sync between locations runs, it's a full sync: just copying new files and overwriting old files. Once XYplorer is aware of state, then subsequent syncs can delete files removed on one side.
Why is keeping track of state necessary? Imagine a situation where several syncs have been run between A and B. On the B side, a user manually deletes some files one day. If sync occurs without knowledge of state, then those files will simply be copied from A to B again. But if XYplorer keeps track of state, then it knows that B had those files but they were deleted; thus it knows to delete them from A.
Also:
- Pane-to-pane ad-hoc sync would be awesome. Literally just open two locations on left and right and sync them, without setting anything up.
- It would be cool to be able to set up stored Sync Jobs, in a manner similar to Catalog; basically keep a list of location-to-location syncs that can be fired off without opening panes.
A quick note: I think the program GoodSync is probably the most comprehensive and well-built program to use as reference. (There are other, more- or less-complicated solutions, but this one seems the most directly relatable to the XY paradigm.) So you might review its version 10 just for gaining the vast thinking put into it, as it can handle just about every conceivable sync scenario and is quite elegant.
I do use tools like GoodSync. But the reason we want/need sync tools in XYplorer is to save time and simplify things. If we've already got employees bought into using XY, it's tedious to have them jump in and out of multiple programs to create settings and relationships with the same folders/files over and over again - when XY could be a one-stop-shop.
So here's what I would absolutely need in the sync solution:
1. The one-way sync would simply make (ABC > CD) into (ABC > ABC) as you say. If you could offer some GoodSync-like options (perhaps in a special settings pane) that would be a bonus, but for common use, it could just default to essentially mirroring.
2. The two-way sync would need to, at a minimum: (a) copy new files both left and right; (b) delete files on one side that were deleted on the other side; (c) overwrite older files with newer files. As with 1 above, additional settings and parameters would be great as options. Users would have the option to propagate deletions (and perhaps back up those deleted files somewhere for safety - but that's a nice-to-have). Importantly, XYplorer would need to keep track of file states, which GoodSync does through flat files, in order to facilitate deletions. The first time a sync between locations runs, it's a full sync: just copying new files and overwriting old files. Once XYplorer is aware of state, then subsequent syncs can delete files removed on one side.
Why is keeping track of state necessary? Imagine a situation where several syncs have been run between A and B. On the B side, a user manually deletes some files one day. If sync occurs without knowledge of state, then those files will simply be copied from A to B again. But if XYplorer keeps track of state, then it knows that B had those files but they were deleted; thus it knows to delete them from A.
Also:
- Pane-to-pane ad-hoc sync would be awesome. Literally just open two locations on left and right and sync them, without setting anything up.
- It would be cool to be able to set up stored Sync Jobs, in a manner similar to Catalog; basically keep a list of location-to-location syncs that can be fired off without opening panes.
Re: Synchronize 2 folders
All robocopy has.highend wrote:The debatable thing is: How many options
are necessary (for two-way syncs) to fit everybody's needs...
We already have
- mirror (one-way sync)
- union (two-way sync)
These are are my regular use cases (one-way sync -> backups, two-way-syncs -> keeping folders in sync across machines). And I think it's save to say these are the canonical sync operations.
For more options it boils down which parts of the Set theory you can/want additionally cover w/o confusing the average XYplorer user too much.
Only sometimes I have the scenario
- intersection (two-way sync)
The core (canonical above) sync operations should be exposed to the GUI, additional ones (if offered) could be by scripting only.
FH
-
- Site Admin
- Posts: 60357
- Joined: 22 May 2004 16:48
- Location: Win8.1 @100%, Win10 @100%
- Contact:
Re: Synchronize 2 folders
OK, thanks all for the input! I'll enter a meditation phase now (combined with some carnival for enhanced enlightment), and then come up with a kick-ass solution.
FAQ | XY News RSS | XY Twitter
Re: Synchronize 2 folders
translation: 3 days completely off.admin wrote: (combined with some carnival for enhanced enlightment),.
Re: Synchronize 2 folders
I have long used AJCSync for my synchronization needs, for entire drives. So FWIW (no doubt more than FWIW), below are screen shots of the configuration panels.
Standard Logic (Left to Right)
Standard Logic (Both Ways)
Manual Logic
File Patterns
Transforms
Date Ranges
Time Stamps
Standard Logic (Left to Right)
Standard Logic (Both Ways)
Manual Logic
File Patterns
Transforms
Date Ranges
Time Stamps
Running on Windows 10 Pro 64-bit quad-core ASUS G752-VY notebook with 64 GB RAM, over 26 external USB3 drives attached via multiple powered hubs with letters and mount points, totaling 120+ TB.
-
- Site Admin
- Posts: 60357
- Joined: 22 May 2004 16:48
- Location: Win8.1 @100%, Win10 @100%
- Contact:
Re: Synchronize 2 folders
Yes, I know I'm behind schedule... it will come...
FAQ | XY News RSS | XY Twitter
-
- Site Admin
- Posts: 60357
- Joined: 22 May 2004 16:48
- Location: Win8.1 @100%, Win10 @100%
- Contact:
Re: Synchronize 2 folders
Thanks Jerry (and sorry for the late reply). This will help me in designing it (although I will make it less complex).Jerry wrote:I have long used AJCSync for my synchronization needs, for entire drives. So FWIW (no doubt more than FWIW), below are screen shots of the configuration panels.
FAQ | XY News RSS | XY Twitter
-
- Site Admin
- Posts: 60357
- Joined: 22 May 2004 16:48
- Location: Win8.1 @100%, Win10 @100%
- Contact:
Re: Synchronize 2 folders
OK, I'm coding it now. As I already said above I want to make it simple. There can be some options, and more options can be added later, but for starters I need a default setup for each sync method (there are two).
Basically, sync means: Make source and target identical. There are two basic methods to achieve identity; and options how to handle name-collisions.
1) One-Way Sync (Mirror): Target is made to look like source; one-way overwriting; unique items in the target are deleted.
2) Two-Way Sync (Union): Both folders are merged; two-way overwriting; unique items are copied to the other side; nothing is deleted.
There will be GUI and scripting access to sync. GUI sync will make use of dual panes (active pane = source/folder1, inactive pane = target/folder2).
Basically, sync means: Make source and target identical. There are two basic methods to achieve identity; and options how to handle name-collisions.
1) One-Way Sync (Mirror): Target is made to look like source; one-way overwriting; unique items in the target are deleted.
Code: Select all
Before: Source=ABC, Target=CD
After: Source=ABC, Target=ABC
Overwriting on name collision:
a) Source always overwrites target. [Default]
b) If source is newer than target.
Code: Select all
Before: Folder1=ABC, Folder2=CD
After: Folder1=ABCD, Folder2=ABCD
Overwriting on name collision: The newer overwrites the older.
FAQ | XY News RSS | XY Twitter
-
- Posts: 177
- Joined: 01 Jan 2016 18:59
Re: Synchronize 2 folders
Really excited you're now on this.
I'd like to point out, though, that the 2-way sync you describe (without deletions) isn't particularly useful in my view.
Why is that?
In a common use scenario, you have two folders that each represent ongoing work on the same content - including cleaning up those folders. For whatever real-world business reason (let's say you're dealing with a less tech-savvy operation and their ad hoc solution is to merge back and forth between a local and a remote location), the only option you have is to synchronize work.
So let's say there's a bunch of photos in a folder for a small design company. Dino has a local fileserver drive folder "Office" and then there's a portable "Field" USB drive folder that is used to access the photos when working in the field. While on business travel, Dino goes through the Field drive and deletes a bunch of old files he doesn't want anymore. When he gets back to the office, he wants to sync his work with the Office folder.
In the meantime, Gertrude has been making changes to the Office folder while Dino was out. She's added some new photos and deleted some photos.
If Dino were using AJCSync, GoodSync, 2BrightSparks SyncBackPro, or whatever other sync utility, he could achieve what he wants with almost no fuss.
But he's working in XYplorer, enjoying the fact that this sync feature is now built-in. So he chooses to 2-way-sync... and in minutes, all the work that Dino and Gertrude did pruning and cleaning the folder(s) is eradicated as XYplorerSync populates all the deleted photos right back into the folders.
Gertrude comes running around the corner screaming. Her world is now that much darker, thanks to XYplorer.
--------------------
In all seriousness, this is why the ability to choose to propagate deletions is crucial to 2-way sync. Without the <optional> possibility of propagating deletions, 2-way sync isn't very useful in the real world.
You might say, "I'm not reinventing the wheel here. If you want detailed features, go use those other tools." But then, that's a disruption to workflow and having to go manually apply locations in another tool, set up a sync job, then return to XYplorer, etc. If sync can happen within the XYplorer context, much time is saved and workflow is streamlined.
I'd like to point out, though, that the 2-way sync you describe (without deletions) isn't particularly useful in my view.
Why is that?
In a common use scenario, you have two folders that each represent ongoing work on the same content - including cleaning up those folders. For whatever real-world business reason (let's say you're dealing with a less tech-savvy operation and their ad hoc solution is to merge back and forth between a local and a remote location), the only option you have is to synchronize work.
So let's say there's a bunch of photos in a folder for a small design company. Dino has a local fileserver drive folder "Office" and then there's a portable "Field" USB drive folder that is used to access the photos when working in the field. While on business travel, Dino goes through the Field drive and deletes a bunch of old files he doesn't want anymore. When he gets back to the office, he wants to sync his work with the Office folder.
In the meantime, Gertrude has been making changes to the Office folder while Dino was out. She's added some new photos and deleted some photos.
If Dino were using AJCSync, GoodSync, 2BrightSparks SyncBackPro, or whatever other sync utility, he could achieve what he wants with almost no fuss.
But he's working in XYplorer, enjoying the fact that this sync feature is now built-in. So he chooses to 2-way-sync... and in minutes, all the work that Dino and Gertrude did pruning and cleaning the folder(s) is eradicated as XYplorerSync populates all the deleted photos right back into the folders.
Gertrude comes running around the corner screaming. Her world is now that much darker, thanks to XYplorer.
--------------------
In all seriousness, this is why the ability to choose to propagate deletions is crucial to 2-way sync. Without the <optional> possibility of propagating deletions, 2-way sync isn't very useful in the real world.
You might say, "I'm not reinventing the wheel here. If you want detailed features, go use those other tools." But then, that's a disruption to workflow and having to go manually apply locations in another tool, set up a sync job, then return to XYplorer, etc. If sync can happen within the XYplorer context, much time is saved and workflow is streamlined.
-
- Site Admin
- Posts: 60357
- Joined: 22 May 2004 16:48
- Location: Win8.1 @100%, Win10 @100%
- Contact:
Re: Synchronize 2 folders
I see, so that would be this:
3) Two-Way Sync (Intersection): Both folders are intersected; two-way overwriting; unique items are deleted on each side.
OK?
PS: Looks like running One-Way Sync (Mirror) two times, once in each direction.
3) Two-Way Sync (Intersection): Both folders are intersected; two-way overwriting; unique items are deleted on each side.
Code: Select all
Before: Folder1=ABC, Folder2=CD
After: Folder1=C, Folder2=C
Overwriting on name collision: The newer overwrites the older.
PS: Looks like running One-Way Sync (Mirror) two times, once in each direction.
FAQ | XY News RSS | XY Twitter