Long Delay For Folders with many files

[mqsymth]

I have a folder that has 20 sub folders and each sub folder has 100-200 files in them. Whenever I exit xyplorer while it is pointing to one of the files in one of the sub folders and then shut down the computer, the next time I start the computer and run xyplorer, it take a very long time to come up and display the the directory and files.

In fact anytime I start xyplorer for the first time and then click on the directory xyplorer takes a very long time to display that directory.

This doesn't happen with directories with less files.

Is there any way to speed-up xyplorer when I first access this directory?

I usee Win XP Home.

[JustinF]

Do you have folder sizes turned on? (Tools | Configuration | General | Show folder sizes in file list) I know if I have that turned on and I view something like my Windows folder, it will take 10 seconds or so to go through and calculate the size of the subfolders. Hitting ESC will terminate the process and just list the folders with a size of 0...or you can turn that option off.

If that option is already off, then maybe someone else has some idea.

[mqsymth]

Do you have folder sizes turned on? (Tools | Configuration | General | Show folder sizes in file list) I know if I have that turned on and I view something like my Windows folder, it will take 10 seconds or so to go through and calculate the size of the subfolders. Hitting ESC will terminate the process and just list the folders with a size of 0...or you can turn that option off.

If that option is already off, then maybe someone else has some idea.

I checked and this option is off (check box empty). The slow load only happens on the first view of this folder. On any subsequent view of the folder it displays quickly. I guess the folders are cached somewhere and the cache is destroyed when the computer shuts down.

Win XPHome SP2

[JustinF]

Hmmmmm...well, I tried. When I click on my C:\Windows folder, which contains 1500+ folders and 12,000+ files, it displays instantly. There must be some other issue; the mighty Don will have to step here and fix you up.

It's not a mapped drive or network share, is it?

[mqsymth]

Hmmmmm...well, I tried. When I click on my C:\Windows folder, which contains 1500+ folders and 12,000+ files, it displays instantly. There must be some other issue; the mighty Don will have to step here and fix you up.

It's not a mapped drive or network share, is it?

No it is not a mapped or network shared drive.

My windows directory also displays fast.

All files in the slow loading directory are 1-2meg in size. Also many of the files are fragmented. When I did a defrag awhile ago the folder displayed faster but still it was slow when compared to the C:\Windows directory display speed.

Also all the files in the slow load directory are csv (comma delimited) files.

I'm using v5.00.0015. However, I've had his problem since v4.6

[admin]

I have a folder that has 20 sub folders and each sub folder has 100-200 files in them...

... and no folders, right?

The time-consuming process here is checking whether a folder has any subfolders. This has to be done to decide whether the branch is expandable and should get a little plus-symbol in the tree. The only way to do this is to check every item inside a folder until a folder is found (at which time the checking procedure can be left). Now in your special case there are no folders to be found (I assume), so about 3,000 files are accessed on the hard disk (at first access, as you observed) before your 20 subfolders are listed (as unexpandable). Any non-first accesses are much faster because the OS caches these informations.

I doubt that any other file manager can be faster here and determine the correct exandable state of each folder.

[jacky]

hmmm.. wild guess here, as I couldn't really test this since I already went to my folders with lotsa files & no folders, so it's already fast no matter what but the way you describe it looks like you're doing a FindNextFile loop; couldn't this be any faster if you're only looking for that flag:

Code: Select all

Private Const MAX_PATH = 260
Private Const SFGAO_HASSUBFOLDER = &H80000000
Private Const SHGFI_ATTRIBUTES = &H800

Private Type SHFILEINFO
    hIcon As Long
    iIcon As Long
    dwAttributes As Long
    szDisplayName As String * MAX_PATH
    szTypeName As String * 80
End Type

Private Declare Function SHGetFileInfo Lib "Shell32" Alias "SHGetFileInfoA" _
    (ByVal pszPath As Any, ByVal dwFileAttributes As Long, psfi As SHFILEINFO, _
    ByVal cbFileInfo As Long, ByVal uFlags As Long) As Long

Function FolderHasSubFolders(ByVal sPath As String) As Boolean
    Dim FInfo As SHFILEINFO
    ' retrieve the folder's attributes
    SHGetFileInfo sPath, 0, FInfo, Len(FInfo), SHGFI_ATTRIBUTES
    ' check che presence of the HASSUBFOLDER attribute
    FolderHasSubFolders = (FInfo.dwAttributes And SFGAO_HASSUBFOLDER)
End Function

Edit: hmmm... The SFGAO_HASSUBFOLDER attribute is only advisory and may be returned by Shell folder implementations even if they do not contain subfolders. Note, however, that the conversefailing to return SFGAO_HASSUBFOLDERdefinitively states that the folder objects do not have subfolders.
Returning SFGAO_HASSUBFOLDER is recommended whenever a significant amount of time is required to determine whether or not any subfolders exist. For example, the Shell always returns SFGAO_HASSUBFOLDER when a folder is located on a network drive.

Guess that's what Explorer uses, and why it sometimes shows the plus-symbol where it shouldn't....

[mqsymth]

I doubt that any other file manager can be faster here and determine the correct exandable state of each folder.

Yes you are correct, no other file manager, that I like, is faster. I use "I Like" because I have found one or two file managers that don't have the slow load problem but I didn't like the file manager as a whole.

It seems that winXP OS does the cacheing. When I started my computer and went to Windows Explorer and tried to display the slow directory, Explorer took 10sec to dispay the slow directory and it's subdirectories. I then closed Explorer and started xyplorer. When I displayed the slow directory in xyplorer it displayed instantly. So xyxplorer must have used the OS cache started by Explorer. The OS must automatically cache the windows directory at startup because the windows directory always loads fast.

It seems there can be a better solution to this slow load problem as suggested by one of your users in this forum.

[admin]

hmmm.. wild guess here, as I couldn't really test this...

I tested this and other ways years ago: it's not any faster. (I suppose it uses FindNextFile internally...)

[admin]

It seems that winXP OS does the cacheing. When I started my computer and went to Windows Explorer and tried to display the slow directory, Explorer took 10sec to dispay the slow directory and it's subdirectories. I then closed Explorer and started xyplorer. When I displayed the slow directory in xyplorer it displayed instantly. So xyxplorer must have used the OS cache started by Explorer. The OS must automatically cache the windows directory at startup because the windows directory always loads fast.

Yes.


ADDED:

It seems there can be a better solution to this slow load problem as suggested by one of your users in this forum.

No. Unfortunately not.

[Gandolf]

Donald, any chance of providing an option to disable the checking of folder for sub-folders? My images backups is now getting so large and, since almost all the folders contain only files, XY last took 154,202ms to start when closed on that folder!

There is one file manager I know of that enables the logging of sub-folders only, so displays the list of sub-folders instantly. Obviously, it cannot show if the sub-folders contain files or folders.

If the folder had not been checked for sub-folders would it be possible not to display the dotted line between the sub-folder and the vertical line from the parent folder? I don't know how visible this would be but I think it should show that the folder had not been checked for sub-folders. The box with the plus-symbol, or just the tree node, would then be displayed when the sub-folder was logged. I assume this would be quicker so it would be possible to navigate to the sub-folder and view it. I think any alternative method of showing that the sub-folder check had not been done would work just as well.

[admin]

There is one file manager I know of that enables the logging of sub-folders only, so displays the list of sub-folders instantly. Obviously, it cannot show if the sub-folders contain files or folders.

Sorry, I don't exactly understand what you mean by "logging of sub-folders only". You mean storing them in the INI between sessions?

I think any alternative method of showing that the sub-folder check had not been done would work just as well.

A box with a 1 pixel dot in the center is a way I have seen.

[Gandolf]

Sorry, I don't exactly understand what you mean by "logging of sub-folders only". You mean storing them in the INI between sessions?

No, I wasn't sure if I'd explained it very well.

When a folder is expanded, rather than checking the contents of each sub-folder of that folder for files / folders, simply list the first level of sub-folders, without checking their contents. Does that make it clearer? That should work if a large folder is expanded in XY; but now you make me wonder how it would work when XY is started for the first time, having been closed with that large folder displayed. That is the most annoying situation since I don't necessarily want to see that folder, but have to wait for it to load.

I'm not sure how saving the information in the ini file would help since it would not be correct if another application had updated the folder contents. How quick would a refresh be if some of the folders had been changed? I assume the refresh would still have to check all the sub-folders since it would have no way of knowing if the contents had changed.

A box with a 1 pixel dot in the center is a way I have seen.

Yes, that would work fine.

[admin]

You exactly described the problem. There is no good solution. I have thought about many times in the last years I can say. There are a number of half-work-arounds but ultimately I found nothing satisfying.

To check folder's expandable state (its having subfolders) on demand (putting an 1-dot-box in the meantime) would gain a lot of speed but lose a lot of good information. You can watch Explorer doing it on any network location: puts plus-boxes everywhere, and when you click a false one it just vanishes... hmm.

[mqsymth]

You can watch Explorer doing it on any network location: puts plus-boxes everywhere, and when you click a false one it just vanishes... hmm.

Ahh I was wondering why that happened..thanks.

I think Gandolf has a good idea. I too would like to have the option to have xyplorer load instantly and only display the 1st level directories. It is annoying to close xyplorer when it is displaying a directory with many sub directories with 1000's of files and then have to wait and wait and wait for xyplorer to cache all those directories again when the computer is rebooted or the cache is erased.

I know what's in my directories and sub-directories. I would like to have xyplorer display the first level directories instantly so I can got to the directory of interest at that time and then wait if I wanted to display the subdirectory with the many subdirectories and 1000's of files.