Script: Thumbnail Maintenance

[FKF5482]

Hello again everyone.
Im getting an error when im trying to build the thumbnail cache for certain folders when using
"Build Thumbnail Cache...current folder only - Size Data" or "Build Thumbnail Cache...current folder && subs - Size Data",
im currently using the script in foreground because i can't run it in backround for some reason.
I've tried disabling the FVS as well but it makes no difference.
The error is like this:

Invalid pattern string "Folder name 1 UnlikeI Folder name 2"

and this line of code is highlighted

if(($path UnLikeI $cur_folder) AND ($path UnLikeI $oldpath)) {

Thanks in advance

[kileytoo]

I am still using 1.18 because 1.19 does not work for me. In the final report, for every single folder I get the message saying that there are no images in that folder. I do get the exact same message with version 1.18, but that version works nonetheless and creates all the thumbnails anyway. Any thoughts?

[klownboy]

Thanks for the comments. Off the top of my head I'm not sure what's causing it. You must be using the menu item for "Build thumbnail cache for current folder & subs...with size data". For my info, do you have any issues with building thumbnails if you select the other menu item, "Build thumbnail current folder & subs" - no data? It's been a long time since I've looked at that script though I do still use it often especially recently transferring images to a new computer. I remember the size data options being a nightmare, but I'll take a look at the script for the last two posts.

[klownboy]

Hi kileytoo and FKF5482, I've been trying to duplicate your issues without success. I've set up test folders and sub folders - some full of images and some with none, copied and moved images around moved and ran the menu items providing size data. The data was correct and no errors were encountered.

Can you provide me with some some additional information on your image folders and image such the image types you are talking about in the folders, anything unusual about the folder structure or naming of image files or folders etc. I know FKF5482 was using the menu item that provides size data.

kileytoo, what menu item are you receiving that error message concerning no images in the folder. Is it a particular menu item that you get this message? What type of image(s) are in the folder that you get the message yet the script says there are none - JPG, PDF, GIF, PNG, BMP? Are there non-image files also in these folders along with images?

Thanks for any help. I'm glad to fix this but I have to see the issue.

[FKF5482]

Hello klownboy, my issue with the script was fixed after this xyplorer bug was fixed:

viewtopic.php?f=2&t=20364

So most likely it was the folder naming scheme that was giving me that error. So far its working fine, i will let you know if something changes.

Thanks for replying

[klownboy]

Hey FKF5482, thanks for letting me know.

[eil]

need a hint from anybody understanding the mechanic of this script. as i understand thumbs are saved as database file(single) and lots of thumbs themselves, so i'd wish to know "Refresh Existing Cache...current folder && subs" makes a search in that database file and edits it or makes so to say search-check over hard-drive itself?
just trying to understand why moving 7 files to sub folder and using this function took some ~30-60 seconds to relocate thumbs = what takes that much time?

[klownboy]

Hi eil, in the thumbnail database there will be 2 files for each and every folder with thumbnails. The file names (e.g.1eb5699c986face40d43588f4d9f7237.dbits and 1eb5699c986face40d43588f4d9f7237.dat2) is basically the hash of the folder and thumbnail size with an * between them (e.g., H:\Pictures\Ireland Trip 2011\2011-06-29\*240x240). One file is the database (dat2) and the other stores the actual thumbnails (dbits). So if you have different thumbnail sizes there would be another set of thumbnail files, dat2 and dbits for that folder in the other size thumb as well (e.g., H:\Pictures\Ireland Trip 2011\2011-06-29\*128x128). There would also be a another set of thumbnail files for sub folders.

So for that particular menu item, the script will perform a thumbnail refresh of the existing caches in the thumbnail database associated with current folder and all its sub folders. It's been awhile and I'll look further later, but unlike the other option below it, that menu item only deals with existing caches. It's not building a new database if you had new folders because it is using the "existing" database to refresh those folders. It is strictly looking at the folders extracted from the database files in XY's thumbnail database directory and [edit: also looking at existing thumbnail sizes in the databases. It compares both the folder in the database and the thumbnail size in the database against the current folder and subs.] It is going to take time depending on the the folder and subs size since it's using the same XY CID #501 to completely rebuild that cache. It's not just adding new thumbnails to the cache for files added to a folder.

[Edit:The confusing difference between the rebuild thumbnail for current folder and subs is that the rebuild allows you to select thumbnails sizes to rebuild and it will build/rebuild for every sub folder in those sizes you specify and it does this regardless of what is specified in the existing databases.]

The whole idea of the script was to clean up the database due to moved, deleted or renamed files etc. Unfortunately using CID #501 is the only way to do that properly. Sure you could create missing thumbnails, but that does not completely clean up and rebuild that folder's thumbnail cache files. If you only used CID #507 which creates missing thumbnails, you could still have old thumbnails remaining in the cache for images that were moved or deleted.

I'll look at it further since it's been awhile and let you know if it's different than what I stated.

Edit: I did quickly review the script and that is how it works. I made some clarifications.

By the way, I had noticed your wish a while back concerning "Move" thumbnails along with pictures here viewtopic.php?f=5&t=22212. That would be a great, but based on the way these thumbnail databases are currently set up I think it would be a substantial amount of effort for Don. Both the dat2 file and dbits files would have to be modified for both the existing moved from folder and the new moved to folder.

I'm getting some errors in the script that I did not get back a month or two ago and with syntax checking on or off (I've kept syntax checking on all the time for the last couple of years). So I'll be looking at it to see what happened in scripting to cause them and hopefully correct them.

[eil]

Hello klownboy. thank you for explanation and the script itself in first place(quite a massive of code!). the only thing i still didn't understand is if the script makes a whole "Thumbnails" folder search or directly does modifications to files corresponding to path of interest(and its subs if there are any)?

yeah, as i understood Don found my wish too bothersome(too bad i didn't get any answer at all), so i started to think of some solutions when again got in problem of massive amount of images moving operations and absence of will to "delete zombie thumbs > create large amount of new ones". that's where i found your script.)

as i understand, "Relocate" function of script kinda makes what i wished for: after images were moved to other place, "move" thumbnails accordingly. but for the moment the path stating mechanic is somewhat "long to do", plus always including subs adds "unneeded" work for script = say simply selecting line in script menu to "remember" current path as "source" and then going to other place, again selecting a line in menu to state new path as a "target", would be much faster than digging in menus of drives and deep folder trees.
that's just my thoughts aloud, maybe i'll try dig in help file to add smt like this just for my comfort.

as a creator i hope you can tell me if couple scenarios are possible to do at all via scripting:
1. images were already moved from one folder to other, thumbs used to be created for all, so i give script old-path and new-path, and script modifies existing thumbnails to point to new-path, without touching sub-folders to make process fast enough.[i'm simply not sure if your "Relocation" does exactly this, at least it definitely takes subs in consideration]
2. similar to above, but first i select some images or/and folders, "mark" them with script, move those selected in new place and state this place for script to "re-point" thumbnails.

basically 1 is about moving items exactly from one exact place to another exact one, and then using script to "update" database, plus no subs included for faster modification. while 2 is about stating to script beforehand which items(pictures, folders) are meant to be moved, again hoping for maybe faster "re-pointing thumbs" as it is strictly stated what moved where(though here subs are included).

[klownboy]

Hi eil,

Thanks for the comments.

That particular menu item is probably the most complicated of the bunch since it does have to look through the entire database in the thumbnails folder using multiple nested foreach loops and pick out those individual cache files that match up with the current folder or its subs and at the same time also look at existing thumbnail sizes for those folders. It's actually quite surprising how fast XY does that part though. It's the rebuilding of each of those folders where it found a match which still takes the bulk of the time. When I'm updating "existing" thumbs, I tend to use the very first menu item where you are directly looking at and using the cache files to update those folders in the existing thumb sizes.

The next menu item will simply rebuild everything in those folders. It does however, still look through the database, but in this case it only looking for matches on the folder/thumb sizes so the menu can reflect what's existing for thumbnail sizes.

I made the "relocate" menu item after I got a new computer a year ago. I had placed all my photos in the exact same drive/folder locations and copied the actual thumbnail cache files to the same XY sub folder location. Everything should have matched up with the old computer, but for some reason XY insisted on making new thumbnails. If I remember correctly I think in the end the issue was the time stamps of the files. I thought I maintained the same dates on both the images and the thumbnails but something was off which caused XY to rebuild all the caches. I think Don had mentioned that even though the cache files themselves are only based on folder-thumbnail size and whether it's zoom to fill or branch, XY also looks at the date stamp to determine whether to build a cache.

That menu item is simply using SC thumbscacherename. It's strictly to indicate that all images in folder A are in folder B. But the key is, it's therefore going to include "all" the images in that folder. It's going to blindly update all database cache files that have folder "C:\Pictures\ to "G:\Photos\" even the subs like "C:\Pictures\bla bla bla". I know that menu item is confusing as is. As soon as you visit the new "moved to" location XY is going to generate new thumbnails on it's own so that's why I had an input box for this case and more importantly since it's taking care of the destination folder subs as well. Don states in the help file for SC thumbscacherename that XY does automatically update the cache when whole folders are moved or renamed as long as the move is done in XYplorer. If it's individual files being moved though that's something else. The thumbnails for those files are still in the database just as they are when you delete and rename images. Using CID #501 is the only command that's going to rewrite the database completely to clean up those "not really here anymore thumbs". If the destination folder cache is clean with no stray unused thumbs when the images are moved in, then creating the new ones with CID #507 is probably sufficient. If it's not, then you really need to use CID #501 on that folder as well. Again simply going to the destination will generate those new thumbs but that database may still be filled with old thumbs from images moved or renamed or deleted.

I think I may have partially touched on your scenarios, with the above. The SC thumbscacherename used in the relocation menu item is strictly for whole folders moved to a new location. I don't think we could properly apply the command to moving some of the images within a folder to a new location. I'd have to run some tests to see if it worked for scenarios where all the images were moved to different folder. That command is only concerned with the images folders not individual thumbnails.

As far as scenario 2, you could certainly have a script gather all the selected images filenames, copy them to a new location, and use CID #501 to update the thumbnails in the new location. Unfortunately (and I brought this request up to Don a couple of years ago), rebuilding thumbnails [CID #501] only works on the "current" folder. So the script has to take you to that folder first using SC goto. I was hoping Don could modify or give us a new SC command that would allow updating a folder's thumbs by specifying a folder (i.e., update the folder remotely). It would make the thumbnail script much simpler.

[eil]

That particular menu item is probably the most complicated of the bunch since it does have to look through the entire database in the thumbnails folder using multiple nested foreach loops and pick out those individual cache files that match up with the current folder or its subs and at the same time also look at existing thumbnail sizes for those folders.

as i supposed it's quite complicated task. thank you for description of mechanic.

When I'm updating "existing" thumbs, I tend to use the very first menu item where you are directly looking at and using the cache files to update those folders in the existing thumb sizes.
The next menu item will simply rebuild everything in those folders. It does however, still look through the database, but in this case it only looking for matches on the folder/thumb sizes so the menu can reflect what's existing for thumbnail sizes.

sorry, can't understand the difference between "update" and "rebuild"..
update is about changing(updating) thumb quality/size(all pictures already had thumbs), and rebuild is about creating missing thumbs of same size as already were created for other images in this folder?
can any of these two be used for moving folder that has thumbs created, to get same(existing ones) on new location?

That menu item is simply using SC thumbscacherename. It's strictly to indicate that all images in folder A are in folder B. But the key is, it's therefore going to include "all" the images in that folder. It's going to blindly update all database cache files that have folder "C:\Pictures\ to "G:\Photos\" even the subs like "C:\Pictures\bla bla bla".

hm, so this can be used when moving folder all together, nothing confusing seems to be here. useful for sure. does it make whole database search or "precise" changes?

[klownboy]

I was unfortunately using the word "update" in very generic terms and not trying to distinguish between refreshing, rebuilding, or whatever. In this case (i.e., using the very first menu item which uses the cache database), the cache for the specified folder/thumbnail size is rebuilt using CID #501. The key difference to keep in mind on "existing" in the 2nd menu item is that it won't do anything if there is no "existing" cache (file).

The real difficulty in the script is: since CID #501 has to be accomplished in the "current" folder you therefore have to go to the folder, and as you know as soon as you go to a folder that has a set thumbnail view, it's going to start (re)building thumbs. So it has to determine if there is exisitng thumbnails in the database and if there's a set folder view etc. This information is used to figure out whether CID #501 has to be invoked or not. If it's a new folder with no established thumbs it may not be needed. It was difficult in trying to avoid XY generating thumbnails twice - once when you enter the folder and again when invoking CID #501.

can any of these two be used for moving folder that has thumbs created, to get same(existing ones) on new location?

Not really, but if you read the remarks for SC thumbscacherename in the scripting reference of the help file, Don states, "When a folder is renamed or moved *within* XYplorer the thumbnails cache is updated automatically". He's saying folder not files so even if you moved all the files from folder A to folder B, it's not going to update at least not until you invoke CID #501 or enter the new folder and XY automatically generates new thumbs.

hm, so this can be used when moving folder all together, nothing confusing seems to be here. useful for sure. does it make whole database search or "precise" changes?

Concerning the use of SC thumbscacherename, from what I gather from the help file and how I think it would have to work, XY would have to scan the entire thumbnails folder - all the caches - to look for the "old" folder to make the "new" folder replacement.

Today to test I did move a folder of images and immediately afterwards went into the thumbnails folder, looked a the highlighted new cache files, viewed the dat2 in the Info Panel - Raw View, and could see that the folder was changed in the dat2 file. I also looked at the saved folder views afterwards and noticed the folder view was also changed to reflect the new location.

[eil]

klownboy, thank you for patience and explanations, now everything seems to be clear.)

[sireddie]

Just came across this script which is very impressive!

The only problem is that it doesn't seem to work for network paths (well I can't seem to get it working anyways).
I have all of my photos stored on a server on the network which I'm accessing without mounting a drive, so the path is something like this: \\192.168.1.123\d$\Photos

Whenever I try to create thumbnails using the script it says:
Before: NA cache did not exist. After: NA no images in folder.

Creating thumbnails manually in xyplorer works fine.

Any ideas?

[klownboy]

Thanks for the comment. I'd be glad to take a look, but unfortunately I'm not set up on a network at the moment. I'll see what I can do.