Bug in XY thumbnail memory management

[klownboy]

Since the change in thumbnail memory management, the last thumbnail cache "dbits" file (not the "dat2" file) is staying in memory and XY is not allowing deletion (XY reflects that it is open even though we've left that folder and are no longer in a thumbnail view). Never did this before. One has to exit XY so the memory is released and restart to be able to do this manually. I initially came across this issue in using my ThumbnailMaintenance script. To reproduce all you have to do is generate some thumbs in a folder, go to the <xythumbs> folder and find the dat2 and dbits files generated, and then attempt to delete the "dbits". You can't. You can easily find the new cache files by filtering or highlighting the files created today.

Potentially related,but probably not ...probably is: When caching is turned off, "dbits" files remain in the thumbnails folder. It's easily seen if you turn caching off and at the same time, have the cache folder changed to another temp location. When you switch back to caching on using your normal cache location, and look at that temp folder, it will have "dbits" files.

I know Don isn't available until the end of the month, I'll bump later if needed. Thanks.

Edit (2): Also probably related, sequence of events...1) a second instance of XY is used to rebuild the thumbnail cache of an image folder, 2) exit 2nd instance of XY and then view the thumbnails in the first instance of XY, and half the thumbnails will look to be corrupted. In reality though, they are not corrupted they probably are in memory, but if you restart XY and look at the thumbs again they are all just fine.

[TheQwerty]

I think that's exactly what we were getting at in this thread: http://www.xyplorer.com/xyfc/viewtopic. ... 55#p112755

[klownboy]

Yes, in part, but there's actually 2 issues, one all the stray "dbits" leftover in the cache as discussed in that thread. That was the reason for my last change to my ThumbnailMaintenance script to clean stray dbits or dat2 files. The second issue is the fact that the last generated "dbits" file is staying in memory and therefore can't be deleted until XY is restarted. This was never the case.

[TheQwerty]

I believe that is precisely why there are stray dbits - they aren't getting deleted because they are still "in use" because XY isn't releasing the file handle.

If that is the case there is only a single issue.

[klownboy]

Hi TheQwerty, I was thinking that as well as far as being related (I hinted at it earlier in the thread but wasn't sure). The only part I'm questioning though is the fact that when I've disabled the cache and set a temp folder as the cache folder when I exit there are not just one but multiple entries (or dbits files). In normal use (cache ON), it's only the last dbits file used/accessed that remains left and cannot be deleted. With Cache OFF, evidently each access to an image folder or at least some of them is or may not be released. In any case as you said, I'm sure they're related problems or boil down to just one problem. A single issue makes it easier for Don to correct. It must have something to do with the thumbnail memory management change Don made a short while back or the changes relating to Cache off thumbnail cleanup. Thanks.

[m48tx]

I have the Linux emulator CygWin installed on my Win7 system which allows me access to most all Linux commands as well as Bash shell scripting. I wrote a shell script to bang thru the XYThumbs.txt file and remove obsolete folder entries and the associated dat2/dbits files (creating a new XYThumbs.txt file). I then bang thru the dat2/dbits files and delete any that are unreferenced in the new XYthumbs.txt file. For this to work properly, XYP must be closed when the script is run.

Your keen observations explain something I didn't pay much attention to since Don changed the thumbnail memory management. If I had been accessing thumbnails in XYP, there is almost always one unreferenced pair of dat2/dbits files that get deleted when I run the purge script.

[klownboy]

Hi m48tx, I'm sure Don will take care of this problem with the stray dbits files in the cache ON and cache OFF conditions. This never was the case until the recent changes.

It doesn't sound like you are, but be careful using the XYthumbs txt file solely to manage "cleanup" of the dat2 and dbits files. The XTthumbs.txt file is really nothing more than a log of the "changes" (thumbnail builds/rebuilds) done to the thumbnail cache files in XY's thumbnails folder. Make a copy of the XYthumbs.txt file, delete contents of the original and then refresh the thumbnails in a folder. Go back and look and you'll see only the entries only for the folder(s) you refreshed and everything still works fine for the other image folders (i.e., XY doesn't rebuild the txt file only log the changes). In other words, the text file can be totally hosed up and everything will still function properly (though if all is well it should be correct). I think taking the approach of performing maintenance and cleanup based on the actual dat2 files and associated dbits present in the thumbnails folder is probably a wiser approach and then rebuilding the txt file based on the new data. In my ThumbnailMainenance script http://www.xyplorer.com/xyfc/viewtopic.php?f=7&t=12257 I take that approach and also look at folders which no longer exist and folders which no longer contain image files and delete any associated cache files. Rebuilding thumbs or refreshing thumbs can be important is you like keeping the database slim and tidy especially since with image file moves, deletions, or renames, the thumbnails will remain in the dbits files and be still referenced in the dat2 file. So they can grow big time.
Thanks,
Ken

[m48tx]

Ken,

Thanks for your input. I use your script to build initial thumbnails recursively, but I use my script to maintain the database folder (I wrote this script before I stumbled across your script). I am not sure I am following what you are telling me. The logic of my script is:

Pass 1:
Copy XYThumbs.txt to a work folder (required because the bash script does not support unicode and I have to reformat the file to ascii text as well as translate \ path separators to /).
Pass 2:
For each line in the work XYThumbs.txt file:
- Check if the referenced folder exists.
- Check if the thumbnail size is correct (I only keep size #1 in database but sometimes accidentally create size #2 or #3).
- Check if referenced dat2 and dbits files both exist.
- If any condition is false delete the existing dat2/dbits files and report otherwise if all conditions are true write the line as is to a new work file.
When all lines have been read, overwrite the work XYThumbs.txt with the new work file.
Pass 3:
Produce a unique sorted list of the existing dat2/dbits files in the database folder without the extensions.
For each file name in the list:
- Check if the filename exists in the new work XYThumbs.txt file
- If the condition is false, delete the corresponding dat2/dbits files and report otherwise if the condition is true take no action.
Pass 4:
Convert new XYThumbs.txt file by translating / path separators to \ and ascii text to unicode. Rename database XYThumbs.txt file as XYThmbs.old and move the work XYThumbs.txt to the database folder.

I'll investigate anything in the error report that looks suspicious.

I tend to rename and move folders often. When I do the rename and move, I do it in thumbnail view so the new folder thumbnails will get built by XYP but obviously the old folder names still exist in the database so I end up with obsolete entries in the XYThumbs.txt file and the corresponding dat2/dbits files. So my question is, for what I am trying to accomplish, how is my logic flawed? I am not trying to build/rebuild thumbnails, just get rid of obsolete XYThumbs.txt entries and the corresponding dat2/dbits files and get rid of unreferenced dat2/dbits files.

This is in no way intended to minimize the goodness of the script you wrote! Like I said, I use your script for certain functions.

Cheers,
Gary

[klownboy]

Hi again m48tx, it looks like you have it well covered especially since you work from both directions, xythumb.txt file to database and via versa. Though I'm a little worried that may be relying on the validity of the XYthumbs txt a bit too much since you're using that as a basis. My approach, at least for those menu items that work off the database, is to use the cache files themselves as the basis for action. Using "readfile" in binary mode, each dat2 file is read internally for the information stored, hash (file name base), folder and thumbnail size , etc. The actual folders of images are checked for "existence" and if they still contain images. Surprisingly if you delete all the image files in the folder, the cache files still remain.

As I mentioned previously, XY will obviously generate a new thumb when you rename an image, but the old one will remain in the cache file. XY does not rebuild the database when you move and rename, etc. even when you select all and refresh all selected thumbnails in a folder. You have to specifically choose refresh thumbnails for it to rid the cache of old thumbs from moved or rename files. I don't do anything really with xytumbs.txt other than rewriting it when done. In a way, the xythumbs.txt file is meaningless and is nothing more than a log but hopefully it's accurate especially when maintained as you are. Gotta run...
Thanks,
Ken

[m48tx]

Ken,

Thanks for the additional information. Now my rhetorical questions are what does XYP use the XYThumbs.txt file for (I would guess to locate folders and corresponding database files) and how would it become corrupt? If the dat2 files are used to rebuild the XYThumbs.txt file then one would have to test if the folder referenced in the dat2 file actually exists. Ugh. This issue seems like a circular firing squad to me.

Anyway, I've gone on enough about this subject. Its up to Don and you to solve.

Cheers,
Gary

[klownboy]

Hey Gary,
Honestly, I'm not positive but I don't think XY uses the XYthumbs.txt file at all. Don's code writes the dat2 and dbits files with a hash base file name generated by...

md5 of path and size, like this: md5("E:\TestFiles\pics\*48x48")

as Don explained here http://www.xyplorer.com/xyfc/viewtopic. ... 590#p90590. Any rebuilds of an existing image folder in the same thumb doesn't get an update in the XYthumbs.txt file. It makes sense because no information held in the txt file is changed. I just tried it. Of course if you generated thumbs in a new image folder, or an existing folder which had thumbs but in a new size, than the txt file will change obviously. In any case I don't believe Don's code goes back to look at the data in the txt file at any time. Move it temporarily out of <xythumbs> and you'll see everything will still function properly. Don's code doesn't ever rebuild the txt file. It only updates it as new dat2/dbits file are created for folders in a particular thumb size (the above hash criteria). Now you're right it shouldn't get corrupted, but did you see this thread a few day ago http://www.xyplorer.com/xyfc/viewtopic.php?f=3&t=12688 That one was strange - 2 different folders that had the same hash in xythumbs.txt - I'm not sure how I made that happen, but those 2 lines were cut and pasted out of my xythumbs.txt file.
Ken

[m48tx]

Hey Ken,

OK. I'm a bit (or possibly a lot) dense. I got it now. XYP does not use the .txt file to located the database files - it simply hashes the appropriate information then looks directly for the dat2/dbits files. I did what you said - deleted the .txt file and it has no bearing on anything - XYP continues to find the correct thumbnail database files (but then places an entry in the .txt file). If XYP does not use the .txt file for anything, Don should eliminate it!

Thanks for hanging with me to the point I understand what is going on and what you have so patiently explained (over and over).

Cheers,
Gary

[klownboy]

Hi Gary,

You're quite welcome. I really enjoy XYplorer thumbnails; it's speed, MDBU, etc. As far XYthumbs.txt, yes, I suppose it's not really of much value, at least that I know of...nothing more than a log really. It's not really hurting or helping anything so it doesn't really matter if it stays or goes. I'm not sure why I bothered to rebuild it in the script. Back a couple of years ago I probably wasn't so sure about it so I figured I better maintain it or someone using the script may get nervous (including me).

Take care,
Ken

[m48tx]

Hi Ken,

I quite agree ... XYP thumbnails are awesome and along with the MDBU eliminate the need for an external image viewing program.

I did modify my purge script ignore the .txt file altogether and to instead look at each .dat2 file and delete it and its associated .dbits file if the referenced folder does not exist. Definitely more effective and efficient.

Again, thanks for helping sweep the cobwebs from my brain.

Kind regards,
Gary

[klownboy]

Small bump now that you're back. Definite memory bug in thumbnail management, please see the first post. Thanks and glad you're back, alive and still kicking ...!