smart copy verification

Please check the FAQ (https://www.xyplorer.com/faq.php) before posting a question...
Post Reply
admin
Site Admin
Posts: 67270
Joined: 22 May 2004 16:48
Location: Win8.1, Win10, Win11, all @100%
Contact:

smart copy verification

Post by admin »

What would you think of an optional "smart" verification which only checks first 5% and the least 5% of the transferred data? Would be 10 times faster than the current full verification. And be 10 times nicer to your hard disk.

serendipity
Posts: 3360
Joined: 07 May 2007 18:14
Location: NJ/NY

Re: smart copy verification

Post by serendipity »

admin wrote:What would you think of an optional "smart" verification which only checks first 5% and the least 5% of the transferred data? Would be 10 times faster than the current full verification. And be 10 times nicer to your hard disk.
Maybe randomly check 5% from anywhere? or a tweak to increase 5% to more.

grindax

Re: smart copy verification

Post by grindax »

.
Last edited by grindax on 22 Jan 2016 16:28, edited 1 time in total.

admin
Site Admin
Posts: 67270
Joined: 22 May 2004 16:48
Location: Win8.1, Win10, Win11, all @100%
Contact:

Re: smart copy verification

Post by admin »

grindax wrote:For backups and copy operations, I think the smartest verification would be to checksum the files rather than do the current byte-by-byte simultaneous comparison.

Calculate the checksum of the source file while it's being read in the first place and then, after copying is complete, read the destination file in one go (on it's own, without any unnecessary seeking), and calculate it's checksum for comparison.
Holy crap, you are right! I had totally forgotten that I had discovered the fast crypt API recently. I short test showed that they are much faster than my own method. Cool! :D

LittleBiG
Posts: 1848
Joined: 08 Apr 2011 12:57
Location: Win10x64

Re: smart copy verification

Post by LittleBiG »

Just for the record: all serious duplicate finder software have a byte-to-byte comparison as well, because they said that is the most accurate method for checking. However, checksum generating is really faster.

eil
Posts: 1923
Joined: 13 Jan 2011 19:44

Re: smart copy verification

Post by eil »

idea with 5% is good, but checksum is real faster and easier way of solution :)
byte to byte is not needed here, IMO.
Win 7 SP1 x64 100% 1366x768|1900x1080

nas8e9
Posts: 2232
Joined: 21 Jun 2008 14:50

Re: smart copy verification

Post by nas8e9 »

LittleBiG wrote:Just for the record: all serious duplicate finder software have a byte-to-byte comparison as well, because they said that is the most accurate method for checking. However, checksum generating is really faster.
Accuracy/reliability depends on two things, I guess:
- choice of hash algorithm: as far as I know, crc and md5 aren't considered as bullet-proof as the later sha-versions;
- making sure the destination is read from disk rather than the cache (reading the *source* from cache shouldn't matter and could be faster).

If both conditions are met, I'd say hashing should work here.

Post Reply