Slow thumbnails

[kileytoo]

Please ignore my last three posts in this thread, they should have been in a different thread, my mistake!

I have a question about the fix discussed in this thread (do not recreate thumbs for folders renamed within XYplorer), not sure if you prefer I post it here or open a new thread?

[admin]

Here.

[kileytoo]

Well, I wanted to thank you for implementing the fix, and to note that the usability while thumbnails are being created has much improved, the scrollbar is responsive and I can select files, open the context menu, etc. When I press Esc to cancel, the next time thumbnail creation resume from the last one already created, which is great.

The only issue I noticed is that occasionally the thumbnails are still being recreated after renaming a folder, but it happened only a few times today and I was not able to replicate the issue, so maybe something else went wrong (I will keep an eye on it though).

[kileytoo]

As for the issue mentioned in this thread that thumbs are recreated when a removable drive is assigned a different drive letter, I am currently using the workaround of always forcing Windows to use the same letter for the same drive. It was easy to set up on my Windows 7 desktop with a utility called USBDLM, but for some reason this did not work as expected on my tablet running Windows 8.1, where I am relying on Windows remembering the last used letter. It works for now, but it's not as reliable as I would like it to be.

Since I am also using XYplorer as a portable installation on the removable drive where the image files are, with a local thumbnails cache for all the files on the drive, I was wondering if it would be possible to have a thumbnail cache with relative file paths, not including the drive letter, to be used for XYplorer portable installs?

[jupe]

On your tablet without using USBDLM you can setup through Windows to always use the same drive letter, here is a tutorial if you are interested

https://www.howtogeek.com/96298/assign- ... windows-7/

[kileytoo]

Thanks but unfortunately the letters you assign with disk management are not permanent, you need to be careful about which letters you choose, which ports you plug each drive in, and possibly other factors. I tried using registry files to restore the HKLM/System/MountedDevices key when the letters change but the problem is that this key is not well documented and I regularly found multiple entries for the same drives, which I can't explain, so it doesn't feel safe to play with it.

[klownboy]

I posted this in this thread because it's along the lines of what the OP was asking concerning thumbnail cache with relative paths.

I recently bought a laptop and I copied all the XYplorer files/folders to the same drive/folder location on the new laptop as my desktop, but my extensive photos folder and wallpaper folders were unfortunately on different drives on the new laptop. Consequently, all the thumbnail dat2 files pointed to the old image location from the desktop and not the image locations on the new laptop. It seemed crazy to build or remake all the thumbnail files considering the amount of photos I've taken. Obviously the thumbnails themselves were fine but useless with old locations in the dat2 files. So, I wrote a script that would allow the user to specify the old folder location and the new location and it searches through the entire database for the old location(s) including sub folders and replace them with the new folder locations(s). It determines the new hash and then writes the new dat2 thumbnail file. The script then renames the corresponding dbits file with the new hash and deletes the old dat2 file.

Don, this script works fine if I simply change drives and the new folder structure and filepath length is identical to the old folder structure and filepath length (e.g., images were previously in "g:\photos" now they're in "d:\photos"). If I change the folder names, it will fail because the new dat2 file generated is not quite the same as it would be if it was built normally with CID #501. I believe it's because of the hex data written after the alloted 276 bytes for a folder*thumbnail size and before the listing of the image folders. This would work across the board if I could change that as well when I rewrite the dat2 file. Do you have any idea what the hex data written in that area is or how I could duplicate it?

My plan was to eventually add this script as a new menu item on my Thumbnail Maintenance script at some point. I wasn't in a rush since I didn't think there would be a big need for it, but maybe it would be helpful to some in situations similar to mine. It needs some enhancements..it doesn't take into account zoom-to-fill thumbs yet or folders wth multiple thumb sizes.

Code: Select all

//"Search Thumbnail DB and Replace cache folder location|:find"

// *** Obtain listing of all dat2 files in the thumbnail database
	$dat2 = formatlist(folderreport("files", "r", "<xythumbs>", , , "<crlf>"), "f", "<crlf>", "*.dat2");

// *** determine the old or relocated path where images were kept since this could be a location no longer on the computer I couldn't use SC inputfolder
	$path2change = input("Enter the old or relocated folder path which you want replace.", "If for example, all your photo sub folders were previously under parent folder 'G:\Photos', like G:\Photos\Mom, but are now under 'D:\Photos', enter 'G:\Photos' below.", "E:\3\", "s", ,300,250);
	if(substr("$path2change", -1) != "\") {$path2change = "$path2change" . "\";}

// *** Obtain the new image folder location
	$newfolderpath = inputfolder("C:\", "Select the new image folder path location to be updated in the thumbnail database files.");
	if(substr("$newfolderpath", -1) != "\") {$newfolderpath = "$newfolderpath" . "\";}

// *** Safety belt to confirm folder relocation in the thumbnail database
   if (confirm(quote("OK") . "  to replace all instances of:<crlf 2>Old folder location:  $path2change      ...with...<crlf>New folder location:  $newfolderpath<crlf 2> in the thumbnail database files (i.e., you have relocated images from the old path location to the new path, or<crlf 2>" . quote("CANCEL") . " quit and/or select different folder(s).")) {continue;} else {end 1;}       //safety belt which you can comment out once you are more familiar with the script

// *** List all folders under the new folder location recursive
	$newfolders = $newfolderpath . <crlf> . (folderreport("dirs", "r", $newfolderpath, "r", , "<crlf>"));

// *** Obtain listing of corresponding sub folders under $path2change which match the new $path2change
	$oldfolders = replacelist($newfolders, $newfolderpath, $path2change, "<crlf>",0,0);

// *** Build side to side table of new versus old folders
		$totalfolders = gettoken($newfolders, "count", "<crlf>");
		$oldVSnewfolder = "";
		$i="1";
		foreach($eachOLDfolder, $oldfolders, "<crlf>", "e") {
				$eachNEWfolder = gettoken($newfolders, $i, "<crlf>");
				$oldVSnewfolder = $oldVSnewfolder . $eachOLDfolder . "|" . $eachNEWfolder . "<crlf>";
				$i++;
		}
// *** Makes the image folder replacement in all the applicable thumbnail dat2 files and determines the new file hash and then writes the new dat2 file with the new hash base file name
	foreach($dat2file, $dat2, "<crlf>", "e") {
		$dat2info = readfile("$dat2file", "b", 170, , 17);     //reads each thumbnail ".dat2" file for the folder and thumbnail size
//		$dat2infoFront = readfile("$dat2file", "b", 276, , 1); //not used currently
//		$dat2infoEnd = readfile("$dat2file", "b", , , 277);    //not used currently
		$thumbsfolder = gettoken($dat2info, 1, "*", "t");      //extract the folder
		$thumbssize = gettoken($dat2info, "2", "*", "t");//    //extract the thumbnail size
		foreach($eachMatchfolders, $oldVSnewfolder, "<crlf>", "e") {
			$eacholdfolder = gettoken($eachMatchfolders, "1", "|");
			if(substr("$eacholdfolder", -1) != "\") {$eacholdfolder = "$eacholdfolder" . "\";};
			if($thumbsfolder LikeI $eacholdfolder) {
				$eachnewfolder = gettoken($eachMatchfolders, "2", "|");
				if(substr("$eachnewfolder", -1) != "\") {$eachnewfolder = "$eachnewfolder" . "\";}
				$fileHash = getpathcomponent($dat2file, "base");      //gets the base file name (i.e., the "hash")
				$content = replace(readfile($dat2file, "b"), $eacholdfolder, $eachnewfolder, , 17, 1);
				$newHash = hash(MD5, $eachnewfolder . * . $thumbssize);
				$newdat2file = replace($dat2file, $fileHash, $newHash);
				writefile($newdat2file, $content,,"b");
// *** Renames the associated dbits file with the new hash of the dat2
				$dbitsfile = replace($dat2file, "dat2", "dbits");
				renameitem($newHash, $dbitsfile, 1);
// *** Delete the old dat2 file with the old folder location.  The dbits file was simply renamed
				delete 0, 0, $dat2file;
				break 1;
				}
		}
	}

[highend]

Judging from the hex view of it, it looks pretty straight forward

First 16 bytes are always identical. From 17th on, write the path + size, pad everything
behind with 0x20 (until 276th byte)?

Untested ofc, I don't use thumbnails normally...

[klownboy]

Hi highend, yes exactly, the image folder starts at byte '17'. I've been using that in my thumbnail maintenance sctipt for a long time. As you say the first 16 bytes are identical on all dat2 files. Don leaves the 276 bytes for the 'folder * thumbnail size' information to be written assuming that should be plenty of space for a very long folder path. The problem is there's data written after the 276 bytes and before the image listing. That data written seems to change when the folder changes (i.e., name, path lengths, etc.). So writing the new drive/folderpath in the allotted 17 to 276 space doesn't seem to solve the problem.

[highend]

I see

E.g. from R:\testfolder with 001.JPG
to T:\testfolder with the same 001.JPG

bytes 277-279 are changed in the .dat2 file...

Apart from that part (which Don needs to explain), the rest is rather simple:

Code: Select all

    $srcFile = "D:\Users\Highend\Downloads\test1.dat2";
    $dstFile = "D:\Users\Highend\Downloads\test2.dat2";
    $srcFileContent = readfile($srcFile, "r");
    $hexSrcFileContent = hexdump($srcFileContent, 1, "r");     // Has trailing space
    $headerSrcFile = gettoken($hexSrcFileContent, 16, , , 1);  // No trailing space
    $footerSrcFile = gettoken($hexSrcFileContent, 277, , , 2); // Has trailing space

    $newPath = "R:\a new folder where items are stored\*96x72";
    $hexNewPath = hexdump($newPath, , "r");                    // Has trailing space
    $cntHexNewPath = gettoken(trim($hexNewPath), "count");
    $toPad = 276 - $cntHexNewPath - gettoken($headerSrcFile, "count");
    $hexDstFileContent = $headerSrcFile . " " . trim($hexNewPath) . strrepeat(" 20", $toPad) . " " . $footerSrcFile;
    writefile($dstFile, hexdump($hexDstFileContent, 1, "ri"), , "r");

[admin]

You mean the 8 bytes 272-279. That's just a date (the exact date when this file was written as an 8-byte VB date value). It's currently not used, just written. So it does not matter. Or are there any problems?

[klownboy]

Thanks Don, good to know that it's the VB date value and that it doesn't affect use when the folder path is changed. Are you sure about the byte number range? I ask only because it looks like you allot 17-276 for the image folder path * thumbnail size. I checked quite a few dat2 files and the 17- 276 area is clear after the folder*thumbnail size. If that's the case, it would be 277-282 ? Just for my info is there a way to translate a typical date like 2018-01-23 to the VB date value in XY scripting? Goine here https://docs.microsoft.com/en-us/dotnet ... -data-type didn't help me much. Thanks again.

The script does appear to be working fine now for varying folder paths and path lengths now. I incorporated highend's hexdump handywork into the original script. So a case like, an original folder of E:\Photos\3\ to a new location of, F:\Wallpaper\3\ works fine. Usually someone would simply be going from D:\Wallpaper to F:\Wallpaper, but it's good to know it works for unusual situations also. Thank you highend and Don.

[admin]

Yes, whoops, you are right! When the first position is 0 (zero), then it's 276 thru 283:

Code: Select all

00000110: 20 20 20 20 D6 CF 74 84 8C 0E E5 40 00 00 00 00 ;     ÖÏt„Œ.å@....
                      ^=276                ^=283                        

[highend]

Just for my info is there a way to translate a typical date like 2018-01-23 to the VB date value in XY scripting?

[admin]

Tricky but possible. Do you want an 8 character string that corresponds to those 8 bytes?