Calculate MD5 checksum to file (like TotalCMD)

Discuss and share scripts and script files...
Post Reply
Litoy
Posts: 6
Joined: 18 Mar 2017 19:25

Calculate MD5 checksum to file (like TotalCMD)

Post by Litoy »

I was looking to replacement for TotalCMD's "Calc checksum" in XYplorer but didn't found it.
So I looked up to scripts and realized that I can do an analog via XYplorer scripting language.

I created a script that will calc MD5 checksum for all selected files / folders and save all info in .md5 file (just like TotalCMD do).
I want to share it with you because it may be useful. I appreciate any feedback on it.
Also I'm planning to create the script that will parse this .md5 file and check that all checksums are correct.

Code: Select all

"Calculate MD5 for selected files and folders"
  $filelist = getinfo('SelectedItemsPathNames', '|');
  $sel_num = getinfo("CountSelected");
  
  if ($sel_num > 0) {
    // getting current folder by first file/folder selected
    $curdir = gettoken($filelist, 1, "|");
    $curdir = gpc($curdir);
    
    if ($sel_num == 1) {
      $checksum_file_path = $curdir . "\" . gpc(gettoken($filelist, 1, "|"), "base") . ".md5";
      end $checksum_file_path == gettoken($filelist, 1, "|");
    }else {
      $checksum_file_path = $curdir . "\" . gpc($curdir, "file") . ".md5";
    }
    status "Getting file list...", , "progress";
    $filelist = unwrap_folder_list($filelist);
    $file_count = count_items($filelist);
    // looping for generated file list
    $crc_content = "";
    $count = 1;
    while(1)
    {
      $file = gettoken($filelist, $count, "|");
      if("$file" == ""){
        break;
      }
      status "Calculating MD5 checksum (" . $count . " of " . $file_count . "): " . gpc($file, "file"), , "progress";
      $crc = hash("md5", $file, 3);
      $crc_content = $crc_content . $crc . " *" . trim_left($file, strlen($curdir) + 1).chr(13).chr(10);
      $count++;
    }
    writefile($checksum_file_path, $crc_content);
    status "Saved MD5 checksum to " . gpc($checksum_file_path, "file");
  }
  
  
  function unwrap_folder_list($file_list) {
    //returns file list with only files, recursive for folders in the input
    $files_in_curdir = "";
    $rez = "";
    foreach($token, $file_list, "|") {
      if (exists($token) == 2) {
        // check for folders
        if(strlen($rez) > 0) {
          $rez = $rez . "|";
        }
        $rez = $rez . folderreport("files", "r", gettoken($token, 1, "|"), "r", , "|");
      }elseif (exists($token) == 1) {
        // check for files
        if(strlen($files_in_curdir) > 0) {
          $files_in_curdir = $files_in_curdir . "|";
        }
        $files_in_curdir = $files_in_curdir . $token;
      }
    }
    if((strlen($rez) > 0) and (strlen($files_in_curdir) > 0)) {
      $rez = $rez . "|";
    }
    $rez = $rez . $files_in_curdir;
    return $rez;
  }
  
  
  function count_items($list, $sep = "|") {
    // counts items in list (f.e. file list)
    $count = 0;
    foreach($token, $list, $sep) {
      $count++;
    }
    return $count;
  }
  
  
  function trim_left($str, $cnt) {
    // trims string by $cnt symbols from the left
    return gettoken($str, $cnt + 1, "", , 2);
  }

bdeshi
Posts: 4249
Joined: 12 Mar 2014 17:27
Location: Asteroid B-612 / Dhaka
Contact:

Re: Calculate MD5 checksum to file (like TotalCMD)

Post by bdeshi »

Welcome to the club, Litoy! :cup:


Now, I had some thoughts on your script. (haven't actually looked at or executed your code very very attentively though :whistle: )

0. Try to follow a consistent coding style.

1. Use get() instead of getinfo(). There's no difference, but getinfo is deprecated.

2. count_items(){...}: You can replace this whole function with a call to gettoken():

Code: Select all

gettoken($list, 'count', $sep);
3. trim_left(): Any reason for [ab]using gettoken()? XY has a substr() function builtin you know.

Code: Select all

return substr($str, $cnt);
4. chr(13) . chr(10) :eh: Just use <crlf>. It's a builtin constant in XY that stands for Windows newline aka carriage return+linebreak aka \r\n.

HTH.
Icon Names | Onyx | Undocumented Commands | xypcre
[ this user is asleep ]

Litoy
Posts: 6
Joined: 18 Mar 2017 19:25

Re: Calculate MD5 checksum to file (like TotalCMD)

Post by Litoy »

SammaySarkar, big thanks for the quick answer!

Please check the second version, I've updated it with your comments:

Code: Select all

"Calculate MD5 for selected files and folders"
  $file_list = get('SelectedItemsPathNames', '|');
  $sel_num = get("CountSelected");
  
  if ($sel_num > 0) {
    // getting current folder by first file/folder selected
    $cur_dir = gettoken($file_list, 1, "|");
    $cur_dir = gpc($cur_dir);
    
    if ($sel_num == 1) {
      $checksum_file_path = $cur_dir . "\" . gpc(gettoken($file_list, 1, "|"), "base") . ".md5";
      end $checksum_file_path == gettoken($file_list, 1, "|");
    }else {
      $checksum_file_path = $cur_dir . "\" . gpc($cur_dir, "file") . ".md5";
    }
    status "Getting file list...", , "progress";
    $file_list = unwrap_folder_list($file_list);
    $file_count = gettoken($file_list, 'count', "|");
    // looping for generated file list
    $crc_content = "";
    $count = 1;
    foreach($file, $file_list, "|") {
      status "Calculating MD5 checksum (" . $count . " of " . $file_count . "): " . gpc($file, "file"), , "progress";
      $crc = hash("md5", $file, 3);
      $crc_content = $crc_content . $crc . " *" . substr($file, strlen($cur_dir) + 1) . <crlf>;
      $count++;
    }
    writefile($checksum_file_path, $crc_content);
    status "Saved MD5 checksum to " . gpc($checksum_file_path, "file");
  }
  
  
  function unwrap_folder_list($file_list) {
    //returns file list with only files, recursive for folders in the input
    $files_in_curdir = "";
    $rez = "";
    foreach($token, $file_list, "|") {
      if (exists($token) == 2) {
        // check for folders
        if(strlen($rez) > 0) {
          $rez = $rez . "|";
        }
        $rez = $rez . folderreport("files", "r", gettoken($token, 1, "|"), "r", , "|");
      }elseif (exists($token) == 1) {
        // check for files
        if(strlen($files_in_curdir) > 0) {
          $files_in_curdir = $files_in_curdir . "|";
        }
        $files_in_curdir = $files_in_curdir . $token;
      }
    }
    if((strlen($rez) > 0) and (strlen($files_in_curdir) > 0)) {
      $rez = $rez . "|";
    }
    $rez = $rez . $files_in_curdir;
    return $rez;
  }

Litoy
Posts: 6
Joined: 18 Mar 2017 19:25

Re: Calculate MD5 checksum to file (like TotalCMD)

Post by Litoy »

I've finished the script to check MD5 similar to TotalCMD one:

Code: Select all

"Check MD5 for files listed in .md5 file"
 $file_list = get("SelectedItemsPathNames", "|");
 $sel_num = get("CountSelected");
 end $sel_num != 1;
 //
 $file = gettoken($file_list, 1, "|");
 $cur_dir = gpc($file);
 $rez_text = gpc($file, "file") . ":" . <crlf>;
 $file = readfile($file);
 //
 if (strpos($file, <crlf>) > 0) {
  // Support for linux-like line endings
  $line_sep = <crlf>;
 } else {
  $line_sep = chr(10);
 }
 $fails = "";
 $not_founds = "";
 $ok_count = 0;
 $fail_count = 0;
 $not_found_count = 0;
 //
 foreach($line, $file, $line_sep) {
  if ((strlen($line) > 0) and substr($line, 0, 1) != ";" and substr($line, 0, 1) != "#") {
    if (strlen(regexmatches($line, "^[0-9a-f]{32}\s[*\s]")) > 0) {
     if (strlen(trim(substr($line, 32, 2))) == 0) {
      // Some MD5 generators create <md5><two_spaces><rel_path> lines
      //  instead of <md5><space><asterisk><rel_path> lines
      $line = substr($line, 0, 33) . "*" .  substr($line, 34);
     }
    }
   $rel_path = gettoken($line, 2, "*");
   status "Checking MD5, File: " . $rel_path, , "progress";
   $file_path = $cur_dir . "\" . $rel_path;
   $saved_md5 = recase(trim(gettoken($line, 1, "*")), "lower");
   // Checking if file exists
   if (exists($file_path) == 0) {
    $rez_text = $rez_text . "FAIL: Can't find file """ . $rel_path . """" . <crlf>;
    $not_found_count++;
    if (strlen($not_founds) == 0) {
     $not_founds = $not_founds . <crlf> . <crlf> . "Not Found Errors:";
    }
    $not_founds = $not_founds . <crlf> . " File " . $not_found_count . ": " . $rel_path;
    continue;
   }
   // Calculating and verifying MD5
   $crc = hash("md5", $file_path, 3);
   if ($saved_md5 == $crc) {
    $rez_text = $rez_text . "  OK: ";
    $ok_count++;
   } else {
    $rez_text = $rez_text . "FAIL: ";
    $fail_count++;
    if (strlen($fails) == 0) {
     $fails = $fails . <crlf> . <crlf> . "MD5 Errors:";
    }
    $fails = $fails . <crlf> . " File " . $fail_count . ": " . $rel_path . <crlf> . "  Saved MD5:" . <crlf> . "   " . $saved_md5 . <crlf> . "  Calculated MD5:" . <crlf> . "   " . $crc;
   }
   $rez_text = $rez_text . $rel_path . <crlf>;
  }
 }
 // Showing result
 $total_fails = $fail_count + $not_found_count;
 if ($total_fails > 0) {
  status "Checked MD5, found " . $total_fails . " errors", , "alert";
 } else {
  status "MD5 verified, all is OK", , "ready";
 }
 $rez_text = $rez_text . <crlf> . "Total error count: ". $total_fails;
 $rez_text = $rez_text . <crlf> . "OK: " . $ok_count . ", not found: " . $not_found_count . ", CRC errors: " . $fail_count;
 text $rez_text . $fails . $not_founds, 550, 600, "MD5 Check: " . $total_fails . " errors";
I added it to Portable OpenWith Menu like this:

Code: Select all

md5>:: load "Check_MD5";
Preview of info that was generated by this script:
Image Image

highend
Posts: 13274
Joined: 06 Feb 2011 00:33

Re: Calculate MD5 checksum to file (like TotalCMD)

Post by highend »

Looks rather complicated (the script)...
Are you sure this can't be simplified a bit?

Which tools are used to generate these .md5 files?
One of my scripts helped you out? Please donate via Paypal

Litoy
Posts: 6
Joined: 18 Mar 2017 19:25

Re: Calculate MD5 checksum to file (like TotalCMD)

Post by Litoy »

highend, I have checked with MD5 files generated by my script, TotalCMD, UltraISO and Android CWM recovery.
The main visual complexity of the script is just output formatting.

highend
Posts: 13274
Joined: 06 Feb 2011 00:33

Re: Calculate MD5 checksum to file (like TotalCMD)

Post by highend »

Ok :)

Personally, I'd find this a bit more readable :biggrin:

Code: Select all

    $md5s = readfile(<curitem>);
    end (gpc(<curitem>, "ext") != "md5" || !$md5s);
    // Convert unix line breaks into windows ones
    if !(regexmatches($md5s, "\r\n")) { $md5s = regexreplace($md5s, "\n", <crlf>); }
    $sep = (regexmatches(gettoken($md5s, 1, <crlf>), "  ")) ? "  " : "*";

    $not_found = ""; $result = ""; $fails = "";
    $ok_cnt = 0; $fail_cnt = 0; $not_found_cnt = 0;

    foreach($line, $md5s, <crlf>, "e") {
        $md5 = recase(trim(gettoken($line, 1, $sep)), "lower");
        $relPath = gettoken($line, 2, $sep);
        $path = "<curpath>\" . $relPath;
        status "Checking MD5, File: " . $relPath, , "progress";
        if (exists($path) == 0) {
            $result = $result . "FAIL: Can't find file '" . $relPath . "'<crlf>";
            $not_found_cnt++;
            if !($not_found) { $not_found = "<crlf 2>Not Found Errors:"; }
            $not_found = $not_found . "<crlf> File $not_found_cnt: $relPath";
            continue;
        }
        // Calculate & verify MD5
        $crc = hash("md5", $path, 3);
        if ($crc == $md5) {
            $result = $result . "  OK: ";
            $ok_cnt++;
        } else {
            $result = $result . "FAIL: ";
            $fail_cnt++;
            if !($fails) { $fails = $fails . "<crlf 2>MD5 Errors:"; }

            $failedLine = <<<>>>
<crlf> File $fail_cnt: $relPath
  Saved MD5:
   $md5
  Calculated MD5:
   $crc
>>>;
            $fails = $fails . $failedLine;
        }
        $result = $result . "$relPath<crlf>";
    }
    // Showing results
    $total_fails = $fail_cnt + $not_found_cnt;
    if ($total_fails) { status "Checked MD5, found $total_fails errors", , "alert"; }
    else { status "MD5 verified, all is OK"; }
    $result = $result . "<crlf>Total error count: $total_fails";
    $result = $result . "<crlf>OK: $ok_cnt, not found: $not_found_cnt, CRC errors: $fail_cnt";
    text $result . $fails . $not_found, 550, 600, "MD5 Check: $total_fails errors";

One of my scripts helped you out? Please donate via Paypal

Litoy
Posts: 6
Joined: 18 Mar 2017 19:25

Re: Calculate MD5 checksum to file (like TotalCMD)

Post by Litoy »

Wow, didn't know about these possibilities to format strings.

Your implementation is great, but there is two missing points:
  1. Some tools generate comments in .md5 file starting with "#" or ";" symbols; the comments are placed above file list
  2. Using separator as double-space will not work if file name will contain double-space

highend
Posts: 13274
Joined: 06 Feb 2011 00:33

Re: Calculate MD5 checksum to file (like TotalCMD)

Post by highend »

Some tools generate comments in .md5 file starting with "#" or ";" symbols; the comments are placed above file list
Are these comments always single or can they span multiple lines?
Using separator as double-space will not work if file name will contain double-space
That requires only a minor fix. I'll add it once I know how the comments stuff look like :)
One of my scripts helped you out? Please donate via Paypal

Litoy
Posts: 6
Joined: 18 Mar 2017 19:25

Re: Calculate MD5 checksum to file (like TotalCMD)

Post by Litoy »

highend wrote:Are these comments always single or can they span multiple lines?
From the files that I've seen, it is always single line comment, like these:
Image

highend
Posts: 13274
Joined: 06 Feb 2011 00:33

Re: Calculate MD5 checksum to file (like TotalCMD)

Post by highend »

Ok, here is the modified version.

Changes:
- Better detection of 2 spaces / * separator (now done on $md5s instead of first line (because of possible comments))
- Relative paths with multiple spaces correctly parsed
- Lines that begin with ; or # are treated as comments and are removed

Code: Select all

    $md5s = readfile(<curitem>);
    end (gpc(<curitem>, "ext") != "md5" || !$md5s);
    // Convert unix line breaks into windows ones
    if !(regexmatches($md5s, "\r\n")) { $md5s = regexreplace($md5s, "\n", <crlf>); }
    $md5s = regexreplace($md5s, "^(;|#).*\r?\n"); // Remove all comment lines
    $sep = (regexmatches($md5s, "[a-z0-9]{32}  ")) ? "  " : "*";

    $not_found = ""; $result = ""; $fails = "";
    $ok_cnt = 0; $fail_cnt = 0; $not_found_cnt = 0;

    foreach($line, $md5s, <crlf>, "e") {
        $md5 = recase(trim(gettoken($line, 1, $sep)), "lower");
        $relPath = gettoken($line, 2, $sep, , 2);
        $path = "<curpath>\" . $relPath;
        status "Checking MD5, File: " . $relPath, , "progress";
        if (exists($path) == 0) {
            $result = $result . "FAIL: Can't find file '" . $relPath . "'<crlf>";
            $not_found_cnt++;
            if !($not_found) { $not_found = "<crlf 2>Not Found Errors:"; }
            $not_found = $not_found . "<crlf> File $not_found_cnt: $relPath";
            continue;
        }
        // Calculate & verify MD5
        $crc = hash("md5", $path, 3);
        if ($crc == $md5) {
            $result = $result . "  OK: ";
            $ok_cnt++;
        } else {
            $result = $result . "FAIL: ";
            $fail_cnt++;
            if !($fails) { $fails = $fails . "<crlf 2>MD5 Errors:"; }

            $failedLine = <<<>>>
<crlf> File $fail_cnt: $relPath
  Saved MD5:
   $md5
  Calculated MD5:
   $crc
>>>;
            $fails = $fails . $failedLine;
        }
        $result = $result . "$relPath<crlf>";
    }
    // Showing results
    $total_fails = $fail_cnt + $not_found_cnt;
    if ($total_fails) { status "Checked MD5, found $total_fails errors", , "alert"; }
    else { status "MD5 verified, all is OK"; }
    $result = $result . "<crlf>Total error count: $total_fails";
    $result = $result . "<crlf>OK: $ok_cnt, not found: $not_found_cnt, CRC errors: $fail_cnt";
    text $result . $fails . $not_found, 550, 600, "MD5 Check: $total_fails errors";

One of my scripts helped you out? Please donate via Paypal

Post Reply