Script: Thumbnail Maintenance

[klownboy]

Hi,
I looked at the size of my XY "thumbnails" folder the other day and couldn't believe how large it had grown. On the laptop it was 1.4 gig and on the desktop it was over a 1.5 gig...way too much picture taking on trips I suppose.

Don had mentioned earlier in this thread that he may institute some thumbnail cleanup tool when he gets some time. In the meantime more as a learning tool for me, I'm attempting to come up a with a script that would look a the XYthumbs.txt file located in XY's thumbnails folder and allow the user to look at the entries and then select those thumbnail files (based on the locations and thumbnail sizes) which he wishes to delete. Note: It would be nice to grab the "date" from the actual files and put that in the inputselect list along side of the associated folder location and thumbnail size, but maybe later.

Code: Select all

  $selected = "";
  $thumbstxt = readfile("<xythumbs>\XYthumbs.txt");
  $sel_hash = inputselect("Select Thumbnails files to delete<crlf>based on thumbnail size and folder location.", $thumbstxt, "<crlf>", 2, , 700, 600, "Thumbnail File Selection");
  foreach($hash, $sel_hash, <crlf>)
   {
    $hash = gettoken($hash, -1, "|");      //extract the hash only portion of line
    $selected = $selected . $hash . "<crlf>";   //for testing only
    delete 0, 1, "<xythumbs>\". $hash .".*"; //using "1" to confirm deletion for testing only, but will change to "0" to avoid all the prompts when multiples selected
   }
    //text $selected;    //for testing only
   $newthumbs = formatlist($thumbstxt, "f", "<crlf>"), !"$sel_hash"); //trying to filter out the lines for the files which were deleted. 
   text $newthumbs;
   //writefile("<xythumbs>\XYthumbs.txt", $newthumbs);

It actually is working (well most of it ), but after deleting the appropriate hash.dat2 and hash.dbit files, I'd also like to remove the associated lines ($sel_hash) from XYthumbs.txt file and then write the new file, but I'm having problem coming up with the filter using "formatlist" (or any other method for that matter). I'm not sure if I need a foreach to filter out each removed line? Could someone point me in the right direct to do this? Thanks. I've temporarily put in a confirmation in the deletion process which I'd eliminate when the script is completed. For testing just hit "no" so you're not deleting the files until the script is done.
Thanks,
Ken

The latest version 1.20 is available here viewtopic.php?p=204361#p204361

[Marco]

I'd say

Code: Select all

$thumbs = regexreplace($thumbs, ".*".$hash.".*\r\n");

put this at the end of the foreach loop, and removing the creation of $newthumbs. Basically, for each selected hash you delete the corresponding files and then you remove the related line from the XYthumb.txt file.

[klownboy]

Hi Marco,
Thanks, that sounds like a smart idea to rid the line out of the XYthumbs.txt file as part of each foreach loop. I haven't been able to get it to work though (i.e., the lines were still present). Did you mean for me to have "$thumbstxt", which is the original XYthumbs file, in lieu of "$thumbs" as the "string" in your regexreplace line? I stepped through the script using both, and either way, it doesn't appear that the regexreplace is working properly. Could I use "$sel_hash", which is the actual full line selected, in lieu of "$hash" in each of the loops to make the regexreplace simpler? I've attached the stepping box for the regexreplace step.

thumbs_cap.JPG (46.32 KiB) Viewed 16137 times

Thanks,
Ken
Edit: Update - It's not deleting the last hash line only when the last selected line is the physical last line of the XYthumbs.txt file. I tried it again with "3" hash lines selected in the middle of the thumbs file, and all "3" were deleted. When I selected 3 hash lines, with the last one being at the very physical end, only 2 hash lines were deleted, not the last. Could it be because there is no return at the end of the last line of XYthumbs.txt?

[Marco]

Be careful, you typed the regex wrong!

[klownboy]

Thanks Marco,
Sorry Marco, that capture I posted was the wrong one. Unfortunately, that capture was from a stepping run-through of the script with a modified regexreplace after I had used yours a number of times. I had initially cut and pasted your regexreplace directly from the post. Here is a capture with your regexreplace. I initially didn't think it was working at all because I didn't notice that it deleted one line, but it did not delete the last line of the selected hashes. I tried it again with many lines and the outcome was always; all lines but the last one were deleted. So I'm not sure if that's a regexreplace issue or a looping "foreach" issue.

thumbs_cap2.JPG (51.52 KiB) Viewed 16110 times

Thanks again,
Ken

Edit update: I selected 3 lines in the middle of the XYthumbs file and all 3 lines were deleted. When I selected 3 hash lines with the last one being at the physical end of the XYthumbs.txt file, only 2 hash lines are deleted and not the last one at the physical end of the file. Could it possibly be because the XYthumbs.txt file has no physical hard return after the last hash line entry?

[Marco]

Yes, that's exactly the reason! You could use then

Code: Select all

$thumbstxt = regexreplace($thumbstxt, ".*".$hash.".*$\r?\n?");

does it work?

[klownboy]

Hey Marco,
Thanks, that last tweak to the regex did the trick. I ran the script through quite a few runs with stepping on, selecting hash lines in the middle of the file and in other runs, at the end, and it seems to be working fine. I used the "r" parameter with the writefile command so that I didn't over write the original XYthumbs.txt file until I more thoroughly test the script. Since I did this, I copied the newly generated file back as XYThumb.txt just to ensure it worked on the new XYThumbs.txt files generated by the script.

Sorry for reporting back the initial regex wasn't working. As it turned out, it was, just not with the last hash line in the file. When I was testing I was continually checking off (in the inputselect command) the last hash line along with some others so when the script completed, I looked at he last line and it was still there. So, I incorrectly assumed it wasn't working at all. I have to spend some time and try to learn regex at least a little.
Thanks again for your help,
Ken

[klownboy]

Hi again,
Sorry for asking questions again, but once again I'm struggling with this. I'm attempting to improve the script discussed above (see below) to provide more information for the user to know which thumbs folders they should delete. The script works fine in it's original state with Marco's help, but I'd like the user to see the "date" of the thumbs folder along with the "thumbs folder", "thumbs size" and "hash" value in the "inputselect" list box.

I've had success in breaking out all the required information, but the information is in two variable lists. In both list I have the common "hash" however. One list has many lines with this information:

Code: Select all

G:\Agent downloads\|240x180|b94829719a137547c237a3e1e3f5ea2a  -  thumbs folder|thumbs size|hash

and the other list has many lines in this format:

Code: Select all

18-06-2013|b94829719a137547c237a3e1e3f5ea2a  -  date|hash

I'd like to have a variable in a resultant list that is the following in each line: "

Code: Select all

thumbs folder|thumbs size|date|hash

I extracted the "date" and "hash" from one list, and extracted individual hashes, thumbs folders, and thumbs sizes from the other list and then tried nested foreach groups to try and meld the list to obtain the desired result with no success. Is there some way to compare the "hash" values in each variable listing and when each hash values are the same (i.e., for each case that you have identical hashes from both lists), format a new line that would be the thumbs folder|thumbs size|date|hash for each of the matching results? One reason the below nested foreach's do not work is because the hashes are not necessarily in the same order in the two list and in the XYthumb.txt generated lists ($thumbstxt)", the first two lines are "XYplorer Thumbs Index v1" and "2013-08-19 17:45:39" though with another method this probably wouldn't matter. So in essence, this means looking for the common value ($hash) from two different variable lists and when the information lines in the 2 list have the common $hash, putting together new lines in a new variable with "thumbs folder|thumbs size|date|hash" in each line if that makes sense.

Code: Select all

  $selected = "";
  $thumbstxt = readfile("<xythumbs>\XYthumbs.txt");    //read the XYthumbs file and assign variable   text $thumbstxt;

  $thumbslist = formatlist(folderreport("files:{Modified dd-mm-yyyy}|{Name}", "r", "<xythumbs>", , , "<crlf>"), "f", "<crlf>", "*.dat2"); //obtain date and hash file 
  
    $thumbsnew = "";
     foreach($file, $thumbslist, "<crlf>")
      {
      $filedate = gettoken($file, 1, "|");   //obtain date
      $filebase = gettoken(getpathcomponent($file, "base"), 2, "|");  //obtain hash file name without extension
     
        foreach($fileline, $thumbstxt, "<crlf>") {
          $filehash = gettoken($fileline, -1, "|");     //hash value from XYthumbs.txt
          $thumbfolder = gettoken($fileline, -3, "|");  //thumbs folder from XYthumbs.txt
          $thumbsize = gettoken($fileline, -2, "|");    //thumbs size from XYthumbs.txt
          if ($filebase == $filehash);                  //didn't think this would work and it doesn't
  
         $thumbsnew = $thumbsnew . "|" . $thumbfolder . "|" . $thumbsize . "|" . $filedate . "|" . $filehash . "<crlf>";
         }
      }

  $sel_hash = inputselect("Select Thumbnails files to delete<crlf>based on thumbnail size and folder location.", $thumbsnew, "<crlf>", 2, , 700, 600, "Thumbnail File Selection");
    foreach($hash, $sel_hash, <crlf>)
    {
     $hash = gettoken($hash, -1, "|");     //pull-out the hash only portion of line
     $selected = $selected . $hash . "<crlf>";  //for testing onlly
     delete 0, 0, "<xythumbs>\". $hash .".*";  //using "1" to confirm deletion for testing only, but will change to "0" to avoid prompt
     $thumbsnew = regexreplace($thumbsnew, ".*".$hash.".*$\r?\n?");
    }
   writefile("<xythumbs>\XYthumbs.txt", $thumbsnew,);

Thanks again,
Ken

[klownboy]

Wow, it must have been late when I wrote the last post or I was tired. I certainly over analyzed the problem. All I have to do is get the "date" using property command in a foreach loop for each file (which in this case happens to be the hash contained in each line of the XYTumbs.txt with a dat2 extension). The date can then be added to the thumbs folder and thumbs sizes information for the inputselect.

[klownboy]

Based on the discussions here Cache all thumbnails for current tab and sub folders at once and since it's been quite awhile since I revisited my earlier script which was originally in Tips and tricks...here. I figured it was about time to expand on the script to not only delete old, non-existent or caches in thumbnail sizes no longer desired, but also include the capability to regenerate selected thumbnail folder caches.

The thumb caches do build up in size substantially over time simply by deleting or moving images in and out of the folder, the thumbs remain in the cache. You can obviously refresh thumbs manually but this take some of the effort out of the chore by letting you select multiple folders to refresh their thumbnail caches. When multiple folder caches are selected, it will move from one folder to the next rebuilding the caches. Please note for the script to execute as desired, it will need the Configuration | Thumbnails setting for "Create all at once" to be set/checked. The script will do that if you don't already. I never had it checked in the past, but it's actually a good option to have selected since you don't have to page or scroll down to get the all the thumbs refreshed and it will not affect the subsequent viewing of thumbs. If you select many folders you may have to go get a coffee, but in the end if you've had these caches for some time you will see decrease in overall size. I know I did, and it was quite substantial.

thumbnail_maintenance1.png (6.45 KiB) Viewed 15871 times

thumbnail_maintenance2.png (38.35 KiB) Viewed 15871 times

If those "dots' (HEX 007) which separate the cache attributes do not display, edit the script to change them to whatever you want, maybe a " | " or " + ".

Please let me know if you have any issues. Thanks
Ken

ThumbnailMaintenance.zip

(5.52 KiB) Downloaded 531 times

Edit: I tweaked the script slightly to remove the deletion confirmation. It was "0" and I had changed it to "1" during testing and forgot to change it back.

[SkyFrontier]

Now finally I have a reason + method to cache thumbs on disk. And deal with consequences.
Thanks much, Ken!
Will report back as soon as I... phew, better not even mention it.

[bdeshi]

Another repetitive task killed! thanks!
===
a couple itches:

":cut"
Hide confirmation for each delete commands? Gets annoying with 10s of folder thumb file-pairs.

":del"
the delete filter should be just "XYthumbs.txt|*.dbits,|*.dat2".
People might have other files there, especially disorganized coders
(I was looking into the file formats and had some notes there... thankfully they were uninportant)

[Stef123]

Many thanks from my side, too.
Like Sammay said, it were nice to shortcircuit the repeated confirmations by a Confirm All or whatever.

Another small issue I noticed: If you happen to run the main script on an empty cache, it throws an error - telling me it cannot access the folder I have currently active in XY.

Will do some thorough testing over the weekend - unfortunately I am offline til Tuesday - cannnot report back before then.

EDIT: It were awesome to get the current tab + sub paths preselected when running "Refresh Thumb..."
Or any other way to generate/ add/ refresh a whole branch without having to patiently sit and watch XY til it finishes, then climb in and out of every sufolder to repeat the supervision.

[klownboy]

Hide confirmation for each delete commands? Gets annoying with 10s of folder thumb file-pairs.

Thanks Sammay & Stef123, totally forgot to change the deletion confirmation back to "0". It was "0" originally as you can tell from the comment line beside it. I had changed it to "1" for testing and forgot to change it back. I can see why that would be annoying.

Also made the "safety" change you mentioned Sammay when deleting the entire cache. I didn't think anyone would be crazy enough to try that option. ...kidding of course.

I updated the my post with the Thumbnail Maintenance script to correct those issues. Sorry, I should have had a revision on the original and uploaded a newer one instead of replacing the zip...next update for sure.

It were awesome to get the current tab + sub paths preselected when running "Refresh Thumb..." Or any other way to generate/ add/ refresh a whole branch without having to patiently sit and watch XY til it finishes, then climb in and out of every sufolder to repeat the supervision.

Honestly, I really didn't want to get into a tab related thumbnail cleanup. I guess personally, I think it should be handled an a folder-to-folder basis because that's the way XY handles it - totally unrelated to tabs. Once you've done some major cleanups and then refreshing, it becomes much easier to "maintain" the caches. Once the caches are cleaned and refreshed, they only become bloated/oversized after you do a bunch of file management deletion, moving, and yes, even renaming (the old thumbs are still there), etc. so subsequent runs require significantly less time. I think it's important though that the user takes a close look at the attributes for the caches to get familiar with them so they can say, "hey I don't look a those "pdf" files in large thumbnail mode, I should delete that one and keep the cache that has medium size thumbs. I'll take a closer look at your request though.

Also, I was originally going to add a menu option that would refresh thumbnails for selected folders, that the user selects (and I suppose that could be "tab" related). This would be totally be totally unrelated to the "dat2" files listing in <XYthumbs>. I could prompt the user for folder selection(s). At least initially, it seemed like it was a bit unnecessary since when you create a folder of images and go into a thumbnail viewing mode, XY is going to create the thumbs anyway.
Thanks for the comments and testing,
Ken

[Stef123]

I was originally going to add a menu option that would refresh thumbnails for selected folders, that the user selects (and I suppose that could be "tab" related). This would be totally unrelated to the "dat2" files listing in <XYthumbs>. I could prompt the user for folder selection(s).

Actually, this is very close to what I erroneously refered to as "Tab"-caching. Provided your intended solution includes subfolders, that's the crucial point.

Might be better to describe what I need it for:
I run XY portable, cleaning up machines I am not familiar with. When I have questions to the people in charge, they have no time to wait for the sluggish thumb generation on the fly, while browsing the subfolders.

Sometimes we use their usual tools (Adobe Lightroom, Bridge etc), but too often they're afraid to blow up their databases with those tons of archive material. Mostly we fall back on XnView's Cache > Add folder.

So basically, what I am hoping for is a feature that scans tree branches by itself, without manual folder browsing. This is not related to maintenance and cache cleaning, but rather to data I won't use again. For these tasks I set up a seperate thumbs-directory that I delete afterwards.