Page 1 of 1
Calculate MD5 checksum to file (like TotalCMD)
Posted: 18 Mar 2017 19:37
by Litoy
I was looking to replacement for TotalCMD's "Calc checksum" in XYplorer but didn't found it.
So I looked up to scripts and realized that I can do an analog via XYplorer scripting language.
I created a script that will calc MD5 checksum for all selected files / folders and save all info in .md5 file (just like TotalCMD do).
I want to share it with you because it may be useful. I appreciate any feedback on it.
Also I'm planning to create the script that will parse this .md5 file and check that all checksums are correct.
Code: Select all
"Calculate MD5 for selected files and folders"
$filelist = getinfo('SelectedItemsPathNames', '|');
$sel_num = getinfo("CountSelected");
if ($sel_num > 0) {
// getting current folder by first file/folder selected
$curdir = gettoken($filelist, 1, "|");
$curdir = gpc($curdir);
if ($sel_num == 1) {
$checksum_file_path = $curdir . "\" . gpc(gettoken($filelist, 1, "|"), "base") . ".md5";
end $checksum_file_path == gettoken($filelist, 1, "|");
}else {
$checksum_file_path = $curdir . "\" . gpc($curdir, "file") . ".md5";
}
status "Getting file list...", , "progress";
$filelist = unwrap_folder_list($filelist);
$file_count = count_items($filelist);
// looping for generated file list
$crc_content = "";
$count = 1;
while(1)
{
$file = gettoken($filelist, $count, "|");
if("$file" == ""){
break;
}
status "Calculating MD5 checksum (" . $count . " of " . $file_count . "): " . gpc($file, "file"), , "progress";
$crc = hash("md5", $file, 3);
$crc_content = $crc_content . $crc . " *" . trim_left($file, strlen($curdir) + 1).chr(13).chr(10);
$count++;
}
writefile($checksum_file_path, $crc_content);
status "Saved MD5 checksum to " . gpc($checksum_file_path, "file");
}
function unwrap_folder_list($file_list) {
//returns file list with only files, recursive for folders in the input
$files_in_curdir = "";
$rez = "";
foreach($token, $file_list, "|") {
if (exists($token) == 2) {
// check for folders
if(strlen($rez) > 0) {
$rez = $rez . "|";
}
$rez = $rez . folderreport("files", "r", gettoken($token, 1, "|"), "r", , "|");
}elseif (exists($token) == 1) {
// check for files
if(strlen($files_in_curdir) > 0) {
$files_in_curdir = $files_in_curdir . "|";
}
$files_in_curdir = $files_in_curdir . $token;
}
}
if((strlen($rez) > 0) and (strlen($files_in_curdir) > 0)) {
$rez = $rez . "|";
}
$rez = $rez . $files_in_curdir;
return $rez;
}
function count_items($list, $sep = "|") {
// counts items in list (f.e. file list)
$count = 0;
foreach($token, $list, $sep) {
$count++;
}
return $count;
}
function trim_left($str, $cnt) {
// trims string by $cnt symbols from the left
return gettoken($str, $cnt + 1, "", , 2);
}
Re: Calculate MD5 checksum to file (like TotalCMD)
Posted: 18 Mar 2017 20:19
by bdeshi
Welcome to the club, Litoy!
Now, I had some thoughts on your script. (haven't actually looked at or executed your code very very attentively though
)
0. Try to follow a consistent coding style.
1. Use
get() instead of
getinfo(). There's no difference, but getinfo is deprecated.
2.
count_items(){...}: You can replace this whole function with a call to
gettoken():
3.
trim_left(): Any reason for [ab]using gettoken()? XY has a
substr() function builtin you know.
4.
chr(13) . chr(10) 
Just use
<crlf>. It's a builtin constant in XY that stands for Windows newline aka carriage return+linebreak aka \r\n.
HTH.
Re: Calculate MD5 checksum to file (like TotalCMD)
Posted: 18 Mar 2017 20:49
by Litoy
SammaySarkar, big thanks for the quick answer!
Please check the second version, I've updated it with your comments:
Code: Select all
"Calculate MD5 for selected files and folders"
$file_list = get('SelectedItemsPathNames', '|');
$sel_num = get("CountSelected");
if ($sel_num > 0) {
// getting current folder by first file/folder selected
$cur_dir = gettoken($file_list, 1, "|");
$cur_dir = gpc($cur_dir);
if ($sel_num == 1) {
$checksum_file_path = $cur_dir . "\" . gpc(gettoken($file_list, 1, "|"), "base") . ".md5";
end $checksum_file_path == gettoken($file_list, 1, "|");
}else {
$checksum_file_path = $cur_dir . "\" . gpc($cur_dir, "file") . ".md5";
}
status "Getting file list...", , "progress";
$file_list = unwrap_folder_list($file_list);
$file_count = gettoken($file_list, 'count', "|");
// looping for generated file list
$crc_content = "";
$count = 1;
foreach($file, $file_list, "|") {
status "Calculating MD5 checksum (" . $count . " of " . $file_count . "): " . gpc($file, "file"), , "progress";
$crc = hash("md5", $file, 3);
$crc_content = $crc_content . $crc . " *" . substr($file, strlen($cur_dir) + 1) . <crlf>;
$count++;
}
writefile($checksum_file_path, $crc_content);
status "Saved MD5 checksum to " . gpc($checksum_file_path, "file");
}
function unwrap_folder_list($file_list) {
//returns file list with only files, recursive for folders in the input
$files_in_curdir = "";
$rez = "";
foreach($token, $file_list, "|") {
if (exists($token) == 2) {
// check for folders
if(strlen($rez) > 0) {
$rez = $rez . "|";
}
$rez = $rez . folderreport("files", "r", gettoken($token, 1, "|"), "r", , "|");
}elseif (exists($token) == 1) {
// check for files
if(strlen($files_in_curdir) > 0) {
$files_in_curdir = $files_in_curdir . "|";
}
$files_in_curdir = $files_in_curdir . $token;
}
}
if((strlen($rez) > 0) and (strlen($files_in_curdir) > 0)) {
$rez = $rez . "|";
}
$rez = $rez . $files_in_curdir;
return $rez;
}
Re: Calculate MD5 checksum to file (like TotalCMD)
Posted: 31 Mar 2017 00:02
by Litoy
I've finished the script to check MD5 similar to TotalCMD one:
Code: Select all
"Check MD5 for files listed in .md5 file"
$file_list = get("SelectedItemsPathNames", "|");
$sel_num = get("CountSelected");
end $sel_num != 1;
//
$file = gettoken($file_list, 1, "|");
$cur_dir = gpc($file);
$rez_text = gpc($file, "file") . ":" . <crlf>;
$file = readfile($file);
//
if (strpos($file, <crlf>) > 0) {
// Support for linux-like line endings
$line_sep = <crlf>;
} else {
$line_sep = chr(10);
}
$fails = "";
$not_founds = "";
$ok_count = 0;
$fail_count = 0;
$not_found_count = 0;
//
foreach($line, $file, $line_sep) {
if ((strlen($line) > 0) and substr($line, 0, 1) != ";" and substr($line, 0, 1) != "#") {
if (strlen(regexmatches($line, "^[0-9a-f]{32}\s[*\s]")) > 0) {
if (strlen(trim(substr($line, 32, 2))) == 0) {
// Some MD5 generators create <md5><two_spaces><rel_path> lines
// instead of <md5><space><asterisk><rel_path> lines
$line = substr($line, 0, 33) . "*" . substr($line, 34);
}
}
$rel_path = gettoken($line, 2, "*");
status "Checking MD5, File: " . $rel_path, , "progress";
$file_path = $cur_dir . "\" . $rel_path;
$saved_md5 = recase(trim(gettoken($line, 1, "*")), "lower");
// Checking if file exists
if (exists($file_path) == 0) {
$rez_text = $rez_text . "FAIL: Can't find file """ . $rel_path . """" . <crlf>;
$not_found_count++;
if (strlen($not_founds) == 0) {
$not_founds = $not_founds . <crlf> . <crlf> . "Not Found Errors:";
}
$not_founds = $not_founds . <crlf> . " File " . $not_found_count . ": " . $rel_path;
continue;
}
// Calculating and verifying MD5
$crc = hash("md5", $file_path, 3);
if ($saved_md5 == $crc) {
$rez_text = $rez_text . " OK: ";
$ok_count++;
} else {
$rez_text = $rez_text . "FAIL: ";
$fail_count++;
if (strlen($fails) == 0) {
$fails = $fails . <crlf> . <crlf> . "MD5 Errors:";
}
$fails = $fails . <crlf> . " File " . $fail_count . ": " . $rel_path . <crlf> . " Saved MD5:" . <crlf> . " " . $saved_md5 . <crlf> . " Calculated MD5:" . <crlf> . " " . $crc;
}
$rez_text = $rez_text . $rel_path . <crlf>;
}
}
// Showing result
$total_fails = $fail_count + $not_found_count;
if ($total_fails > 0) {
status "Checked MD5, found " . $total_fails . " errors", , "alert";
} else {
status "MD5 verified, all is OK", , "ready";
}
$rez_text = $rez_text . <crlf> . "Total error count: ". $total_fails;
$rez_text = $rez_text . <crlf> . "OK: " . $ok_count . ", not found: " . $not_found_count . ", CRC errors: " . $fail_count;
text $rez_text . $fails . $not_founds, 550, 600, "MD5 Check: " . $total_fails . " errors";
I added it to Portable OpenWith Menu like this:
Preview of info that was generated by this script:
Re: Calculate MD5 checksum to file (like TotalCMD)
Posted: 31 Mar 2017 00:13
by highend
Looks rather complicated (the script)...
Are you sure this can't be simplified a bit?
Which tools are used to generate these .md5 files?
Re: Calculate MD5 checksum to file (like TotalCMD)
Posted: 31 Mar 2017 00:21
by Litoy
highend, I have checked with MD5 files generated by my script, TotalCMD, UltraISO and Android CWM recovery.
The main visual complexity of the script is just output formatting.
Re: Calculate MD5 checksum to file (like TotalCMD)
Posted: 31 Mar 2017 01:26
by highend
Ok
Personally, I'd find this a bit more readable
Code: Select all
$md5s = readfile(<curitem>);
end (gpc(<curitem>, "ext") != "md5" || !$md5s);
// Convert unix line breaks into windows ones
if !(regexmatches($md5s, "\r\n")) { $md5s = regexreplace($md5s, "\n", <crlf>); }
$sep = (regexmatches(gettoken($md5s, 1, <crlf>), " ")) ? " " : "*";
$not_found = ""; $result = ""; $fails = "";
$ok_cnt = 0; $fail_cnt = 0; $not_found_cnt = 0;
foreach($line, $md5s, <crlf>, "e") {
$md5 = recase(trim(gettoken($line, 1, $sep)), "lower");
$relPath = gettoken($line, 2, $sep);
$path = "<curpath>\" . $relPath;
status "Checking MD5, File: " . $relPath, , "progress";
if (exists($path) == 0) {
$result = $result . "FAIL: Can't find file '" . $relPath . "'<crlf>";
$not_found_cnt++;
if !($not_found) { $not_found = "<crlf 2>Not Found Errors:"; }
$not_found = $not_found . "<crlf> File $not_found_cnt: $relPath";
continue;
}
// Calculate & verify MD5
$crc = hash("md5", $path, 3);
if ($crc == $md5) {
$result = $result . " OK: ";
$ok_cnt++;
} else {
$result = $result . "FAIL: ";
$fail_cnt++;
if !($fails) { $fails = $fails . "<crlf 2>MD5 Errors:"; }
$failedLine = <<<>>>
<crlf> File $fail_cnt: $relPath
Saved MD5:
$md5
Calculated MD5:
$crc
>>>;
$fails = $fails . $failedLine;
}
$result = $result . "$relPath<crlf>";
}
// Showing results
$total_fails = $fail_cnt + $not_found_cnt;
if ($total_fails) { status "Checked MD5, found $total_fails errors", , "alert"; }
else { status "MD5 verified, all is OK"; }
$result = $result . "<crlf>Total error count: $total_fails";
$result = $result . "<crlf>OK: $ok_cnt, not found: $not_found_cnt, CRC errors: $fail_cnt";
text $result . $fails . $not_found, 550, 600, "MD5 Check: $total_fails errors";
Re: Calculate MD5 checksum to file (like TotalCMD)
Posted: 31 Mar 2017 23:13
by Litoy
Wow, didn't know about these possibilities to format strings.
Your implementation is great, but there is two missing points:
- Some tools generate comments in .md5 file starting with "#" or ";" symbols; the comments are placed above file list
- Using separator as double-space will not work if file name will contain double-space
Re: Calculate MD5 checksum to file (like TotalCMD)
Posted: 31 Mar 2017 23:51
by highend
Some tools generate comments in .md5 file starting with "#" or ";" symbols; the comments are placed above file list
Are these comments always single or can they span multiple lines?
Using separator as double-space will not work if file name will contain double-space
That requires only a minor fix. I'll add it once I know how the comments stuff look like
Re: Calculate MD5 checksum to file (like TotalCMD)
Posted: 31 Mar 2017 23:59
by Litoy
highend wrote:Are these comments always single or can they span multiple lines?
From the files that I've seen, it is always single line comment, like these:
Re: Calculate MD5 checksum to file (like TotalCMD)
Posted: 01 Apr 2017 00:12
by highend
Ok, here is the modified version.
Changes:
- Better detection of 2 spaces / * separator (now done on $md5s instead of first line (because of possible comments))
- Relative paths with multiple spaces correctly parsed
- Lines that begin with ; or # are treated as comments and are removed
Code: Select all
$md5s = readfile(<curitem>);
end (gpc(<curitem>, "ext") != "md5" || !$md5s);
// Convert unix line breaks into windows ones
if !(regexmatches($md5s, "\r\n")) { $md5s = regexreplace($md5s, "\n", <crlf>); }
$md5s = regexreplace($md5s, "^(;|#).*\r?\n"); // Remove all comment lines
$sep = (regexmatches($md5s, "[a-z0-9]{32} ")) ? " " : "*";
$not_found = ""; $result = ""; $fails = "";
$ok_cnt = 0; $fail_cnt = 0; $not_found_cnt = 0;
foreach($line, $md5s, <crlf>, "e") {
$md5 = recase(trim(gettoken($line, 1, $sep)), "lower");
$relPath = gettoken($line, 2, $sep, , 2);
$path = "<curpath>\" . $relPath;
status "Checking MD5, File: " . $relPath, , "progress";
if (exists($path) == 0) {
$result = $result . "FAIL: Can't find file '" . $relPath . "'<crlf>";
$not_found_cnt++;
if !($not_found) { $not_found = "<crlf 2>Not Found Errors:"; }
$not_found = $not_found . "<crlf> File $not_found_cnt: $relPath";
continue;
}
// Calculate & verify MD5
$crc = hash("md5", $path, 3);
if ($crc == $md5) {
$result = $result . " OK: ";
$ok_cnt++;
} else {
$result = $result . "FAIL: ";
$fail_cnt++;
if !($fails) { $fails = $fails . "<crlf 2>MD5 Errors:"; }
$failedLine = <<<>>>
<crlf> File $fail_cnt: $relPath
Saved MD5:
$md5
Calculated MD5:
$crc
>>>;
$fails = $fails . $failedLine;
}
$result = $result . "$relPath<crlf>";
}
// Showing results
$total_fails = $fail_cnt + $not_found_cnt;
if ($total_fails) { status "Checked MD5, found $total_fails errors", , "alert"; }
else { status "MD5 verified, all is OK"; }
$result = $result . "<crlf>Total error count: $total_fails";
$result = $result . "<crlf>OK: $ok_cnt, not found: $not_found_cnt, CRC errors: $fail_cnt";
text $result . $fails . $not_found, 550, 600, "MD5 Check: $total_fails errors";
