Page 1 of 1
smart copy verification
Posted: 01 Jun 2011 21:14
by admin
What would you think of an optional "smart" verification which only checks first 5% and the least 5% of the transferred data? Would be 10 times faster than the current full verification. And be 10 times nicer to your hard disk.
Re: smart copy verification
Posted: 01 Jun 2011 21:24
by serendipity
admin wrote:What would you think of an optional "smart" verification which only checks first 5% and the least 5% of the transferred data? Would be 10 times faster than the current full verification. And be 10 times nicer to your hard disk.
Maybe randomly check 5% from anywhere? or a tweak to increase 5% to more.
Re: smart copy verification
Posted: 01 Jun 2011 21:24
by grindax
.
Re: smart copy verification
Posted: 01 Jun 2011 21:32
by admin
grindax wrote:For backups and copy operations, I think the smartest verification would be to checksum the files rather than do the current byte-by-byte simultaneous comparison.
Calculate the checksum of the source file while it's being read in the first place and then, after copying is complete, read the destination file in one go (on it's own, without any unnecessary seeking), and calculate it's checksum for comparison.
Holy crap, you are right! I had totally forgotten that I had discovered the fast crypt API recently. I short test showed that they are much faster than my own method. Cool!

Re: smart copy verification
Posted: 01 Jun 2011 23:18
by LittleBiG
Just for the record: all serious duplicate finder software have a byte-to-byte comparison as well, because they said that is the most accurate method for checking. However, checksum generating is really faster.
Re: smart copy verification
Posted: 02 Jun 2011 01:00
by eil
idea with 5% is good, but checksum is real faster and easier way of solution
byte to byte is not needed here, IMO.
Re: smart copy verification
Posted: 02 Jun 2011 11:48
by nas8e9
LittleBiG wrote:Just for the record: all serious duplicate finder software have a byte-to-byte comparison as well, because they said that is the most accurate method for checking. However, checksum generating is really faster.
Accuracy/reliability depends on two things, I guess:
- choice of hash algorithm: as far as I know, crc and md5 aren't considered as bullet-proof as the later sha-versions;
- making sure the destination is read from disk rather than the cache (reading the *source* from cache shouldn't matter and could be faster).
If both conditions are met, I'd say hashing should work here.