Script: Thumbnail Maintenance

[klownboy]

ThumbnailMaintenance v1.13 update includes a new option called "Cleanup Residue [if caching was off] and Backup Folder" which will cleanup any stray thumbnail dat2 files which may be residing in the thumbnails folder. Typically this will only happen if the cache was turned off or you turn off caching in some instances and turn it back on in others. Even though the thumbnail "dbits" files are deleted when you have caching turned off, the possibility exist that some corresponding stray dat2 files may still exist.

This option will also prompt the user if they want to update the thumbnail "backup" folder. It's very possible that the XY thumbnail backup folder could be substantially larger that the actual thumbnail folder since when XY does a "Backup Application Data folder", it does not remove old thumbnail caches it simple copies the existing thumbnail files to the backup folder. Consequently, this backup folder could contain many obsolete thumbnail cache files which you no longer need. This option will give you a file count and folder size for the actual thumbnail folder and the backup thumbnail folder. If they differ in size, then the user can make an informed decision to update the backup or not. The current thumbnail backup will be wiped only of any dat2, dbits, and txt files and replaced with the current contents of the XY thumbnails folder size. A full XY backup is not performed only the thumbnails are updated.

ThumbnailMaintenance_v1.13.zip

(6.38 KiB) Downloaded 154 times

Also updated some of the messages. I did not include the icons in this update since they are available in the previous releases. Thanks,
Ken

Edit: I rearranged the confirmation dialog in the new option so if you were the very first one to download you may want to re-download.

[Stef123]

Thanks Ken,

first observations: Cleanup Residue leaves mixed impressions, sometimes it tells me there was nothing to clean, but when I check up on it, there are plenty of old dat2 files that don't belong there.

If a backup folder is not present, I get XY's error dialog telling me "Invalid parameter: F:\......\thumbnails

One machine triggers this error when I run Refresh (current & subs)

error.png (44.8 KiB) Viewed 2713 times

[klownboy]

Hi Stef, sorry for the issues but thanks for testing. I'll take a look at the issues you mentioned.

I'm not sure what would be happening on the first issue since all the script does is look at all the "dat2" files in the <xythumbs> and check to see if there's an associated "dbits" file. If there is not then the dat2 file is deleted. I'll check the logic and code. Question though: You said there are plenty of old dat2 files still there. What do you mean or how are you defining "old"? Is there no dbit with the same base name (hash number)? If it turns out there are dbits files as well then a cleanup using the normal means should be done first.

Concerning the backup comment. The backup folder is taken directly from the XYplorer.ini file. It's listed under "General" | "PathBackupSettings". Is there is no folder listed the variable will be nothing or "". I'm curious what it is in your case. Is it possible it's on a thumb drive that 's not on-line or is now a different drive letter? Currently, if the backup folder is not designated in the xy ini file, an error message will come up that says, "No backup folder is designated in XYplorer". I've tested this and it does work. All I can do additionally is also perform an "exists" test to ensure the folder exists. So if for some reason your backup designated backup folder is not available or online you'll get a message.

I'll look into the last issue concerning the error screen during a Refresh (current and subs).

Thanks,
Ken

[Stef123]

Thanks Ken,
you're right, backup folders are not available. I hook up my various backup drives only when I need them. Considering the enourmous size of my caches, I'd probably cancel out of backup cleaning anyway, and rather overwrite my backups AFTER using your script on the regular cache.

All I can do additionally is also perform an "exists" test to ensure the folder exists.

Good idea. Because XY will NOT backup the thumbs cache if it resides outside of the application folder. And because portable drive letters are bound to change. Not to mention users owning multiple drives, re-organizing paths, getting rid of older backups or selecting differents paths on XY's destination dialog.

You said there are plenty of old dat2 files still there. What do you mean or how are you defining "old"?

Old in the sense of left-overs when cache was OFF but completely empty, when running my tests. Later I look at the same folders with cache ON, run your script, let it build and refresh and cleanup (regular, by db). The "old" residue remains. So I run Cleanup Residue and the results vary, I get pretty much every conceivable combination of leftovers.

There are dbits with no matching dat2. And also the other way round. But mostly, they're matching pairs that I can't get rid of, other than deleting the entire cache. What bothers me are possible overlaps, identical name parts that may mess up my existing caches. I get eerie errors that didn't occur before with the old XY version.

error2.png (19.34 KiB) Viewed 2698 times

[klownboy]

Thanks Stef for the additional information. I've already added the check on whether the folder exists to take care of that one issue. On the new issue on your last post about recursion, do you have the ScriptRecursionWarningOff set to "1" in XYplorer.ini file? The "1" value turns the recursion checker off. It should be set to "1". The script is referring back to itself and running another menu option.

I experienced some issues today while troubleshooting your problems above. See http://www.xyplorer.com/xyfc/viewtopic.php?f=3&t=12688 So, possibly something odd is going on.

I originally thought from your previous posts and someone else's that the issue with having the cache off was only stray dat2 files that no longer had an associated dbits files but you're saying you experience both (i.e., stray dbits files as well). You say you also have matching pairs of dat2 & dbits files that are old. Don't these show up as pre-checked items if you run the very first menu item? As you kbnow, any database files that are referring to folders which no longer exist or have no files in the folder should appear pre-checked. Is that the case or are you talking about a entirely different scenario. Keep in mind the database refresh is solely working with the "existing" thumbnail files. The listing in all the DB options is obtained from the actual files in the folder <xythumbs>.

Don had said in another old thread and this may be critical to what you're experiencing http://www.xyplorer.com/xyfc/viewtopic.php?f=3&t=12688 stated that the "hash" base file name is based on md5 of path and size, like this: md5("E:\TestFiles\pics\*48x48") So you'd think that if a new build was done on the same folder in the same thumb size it would overwrite the previous hash file. Since you work with flash or external drives, could it be that the drive letter is different therefore the hash generated will be different for the same folder?

In any case, I can also do a reverse check to see if there are any stray "dbits" files that have no associated dat2 file along with the other way around. The matching pairs of thumbs files bothers me. I'm not sure what's going on there to know how to take care of it. As I said the first menu item should pre-check any database files that are referring to no longer existing folders or folders with no files in them so they could be easily deleted that way. If you run across this again could you try the first menu item to see if any show up pre-checked.

Thanks for your help,
Ken

[Stef123]

Been reading along in the other thread. Apologies for causing so much extra work, Ken, I don't mean to keep you busy with this. No worries if you decide to put it on hold until Don returns.

ScriptRecursionWarningOff is set to "0", not "1" as you suggested. The wording confuses me, if "1" means ON, does that mean the Warning is OFF but recursion may be ON nevertheless? Anyway, it's "0" and that might be the problem. I don't remember touching it, though. Maybe it got changed around by the XY update? Earlier XY versions never caused these error messages.

You say you also have matching pairs of dat2 & dbits files that are old. Don't these show up as pre-checked items if you run the very first menu item?

I am not sure. There were too many items to bother double-checking them, nor did it occur to me to run the first subscript up front. Just ran the Cleanup Residue, then looked in the folder and spotted "older" files (thanks to color coding). Some of those matching pairs had no size, 0 bytes.

Tried to reproduce some issues, but test situations are safer than real scenarios. Thinking about it - leftovers that are unwanted but otherwise "intact" would not show up as problematic, would they? XY should not have created them in the first place, but per se they are not corrupted and would not flare up on your cleanup radar?

Since you work with flash or external drives, could it be that the drive letter is different therefore the hash generated will be different for the same folder?

Cannot rule it out. Although drive letters are usually kept constant and fixed by the Sys Admin. Folder content changes a lot, of course, and that is XY's main shortcoming compared to other viewers, the failure to update and sync to the changes in a non-bloating, more efficient way. Your script compensates nicely for this.

Thanks
Stef

[bdeshi]

ScriptRecursionWarningOff is set to "0", not "1" as you suggested. The wording confuses me, if "1" means ON, does
that mean the Warning is OFF but recursion may be ON nevertheless?

That tweak only turns off the warning, and doesn't stop script recursion.

Anyway, it's "0" and that might be the
problem. I don't remember touching it, though. Maybe it got changed around by the XY update? Earlier XY versions never caused these error messages.

The SC settingp can change the setting permanently. Maybe you ran some script which had used that SC.

[Stef123]

Thanks, Sammay. If I read you correctly, I could have simply clicked "continue" instead of canceling out of the script? It takes a daring user to continue with a dialog that pops up like that
Proceeding after a message box is one thing, but with an open-hood scripting dialog that says "error", well, you gotta have some nerve to risk an endless loop with unsaved work under way in a multi-tasking environment.

The SC settingp can change the setting permanently. Maybe you ran some script which had used that SC.

Could well be. All I know is, I wouldn't fool with a setting like that.

[klownboy]

Hi, here's another update to the ThumbnailMaintenance script (version 1.14). I noticed that the new cleanup function of stray "dat2" files wasn't working as it should. So I wanted to post this correction and update. I had 2 variables reversed in one command line which causes it to hiccup. With this change the script also checks for stray "dbits" file along with the dat2 files in the <xythumbs> folder. So it should clean out strays of either type. Now if you have matched pairs of dat2 and dbits files that are old, the script does already check for folders which no longer exist or folders which no longer contain image files and will pre-check the entries in the database listing. Beyond that I'm not sure if there's anything else I can do with"old" matched pairs without knowing how these stray matched pairs of these files is occurring. It may only occur in situation where you've had the cache turned off and on.

Thanks Sammay for the clarification on recursion. Yes it is just the warning that is turned off. I turned the warning off over 2 years ago based on a recommendations in other scripts [yes, and I'm still here]. The script is simply turning back on itself which many scripts do so setting it to "1" shouldn't be a problem. Stef if you are reluctant to set it to "1" yes you should be able to simple click continue.

Stef I'm still looking at the other issues you've encountered. I haven't been able to reproduce (yet) so it make is harder to correct. In the meantime I wanted to get this update out.
Thanks,
Ken

ThumbnailMaintenance_v1.14.zip

(6.63 KiB) Downloaded 159 times

[Stef123]

Thank you Ken
hope I'll get around to testing it before the weekend, business has suddenly picked up over here.
I have no qualms about turning off the warning. My remark was not geared towards your script, just a general observation.

[PeterH]

Sorry: I didn't read all, but saw the problem with recursion.

So a question: shouldn't a script using recursion set Setting 'AllowRecursion', 1; to avoid problems?
(Set it *not* permanent!)

The current script "knows" that it's needed, and the user doesn't lose protection for other scripts.

[bdeshi]

(setting/settingp)
AllowRecursion: Show no warning on script recursions

See, it controls only the warning dialog. Recursion itself is active either way.

[klownboy]

Thanks PeterH, I didn't look at it that way since I've encountered so many scripts that need it set to "1" to avoid the message. I believe only one or two portions of the script need it anyway. Most of the normal use script options would not need it. So maybe on the next change I'll add the temp setting change.

And yes, thanks Sammay, I rechecked the "setting" in help, it is only the warning being suppressed.

[bdeshi]

think Peter's suggestion could be recommended if the recurse-warning dlg was more friendly.
it's now just too scary at first glance.

[Filehero]

Hi Ken,

I've just installed your script (actually I was about to add a clear thumbs cache routine to my own script toolbox when I remembered that Ken has already published an entire suite on that matter ).

First impression: impressive work!

Of course, I haven't tried any single function (you know, I'm rather using Lightroom as my main picture DB) but it's already in my toolbox and will stay there.

Any ideas why there are some icons "!"-marked?

TM_Exmark-icons.png (3.89 KiB) Viewed 2563 times

Cheers, FH