Improving XY load times?

Please check the FAQ (https://www.xyplorer.com/faq.php) before posting a question...
Filehero
Posts: 2644
Joined: 27 Feb 2012 18:50
Location: Windows 10 Pro x64

Improving XY load times?

Post by Filehero »

Hi Don,

Code: Select all

XY ver: 	XYplorer 18.70.0104
OS:     	Windows 10, 64-bit
Loaded: 	2018-02-24 21:42:13 in 1.698 ms

     0 ms:     153 ms - Start Load
   153 ms:      20 ms - Init Language
   173 ms:      46 ms - Load Language
   219 ms:       8 ms - Load Settings
   227 ms:     225 ms - Init Custom File Icons
   452 ms:      18 ms - Init Controls
   470 ms:      64 ms - Init Graphics
   534 ms:      19 ms - Init Tabs
   553 ms:      45 ms - Create Info Panel
   598 ms:      26 ms - Set Interface Font
   624 ms:      23 ms - Init Panes
   647 ms:       0 ms - Init Tree
   647 ms:       0 ms - Add Spec Folders
   647 ms:       0 ms - Add Links Folder
   647 ms:       0 ms - Add Desktop Folder
   647 ms:       0 ms - Add Documents Folder
   647 ms:       0 ms - Add Downloads Folder
   647 ms:       0 ms - Add User Folder
   647 ms:       1 ms - Drives To Array
   648 ms:      51 ms - Add Drives
   699 ms:       2 ms - Add Portable Devices
   701 ms:       0 ms - Init Start Path
   701 ms:      10 ms - Tab Icons
   711 ms:       6 ms - Load Catalog
   717 ms:       1 ms - Load Tags
   718 ms:     673 ms - Load Folder Size Cache
 1.391 ms:       0 ms - Load Maxi Tree
 1.391 ms:       0 ms - Calc Locked Maxi Tree
 1.391 ms:      28 ms - Go Start Path: E:\
 1.419 ms:     114 ms - Init Custom Toolbar Buttons
 1.533 ms:      12 ms - Init UDC and KS
 1.545 ms:      17 ms - Layout Window
 1.562 ms:      10 ms - Layout Catalog
 1.572 ms:      73 ms - Layout List
 1.645 ms:      27 ms - Show Window
 1.672 ms:       0 ms - Layout Catalog
 1.672 ms:       9 ms - Layout List
 1.681 ms:       0 ms - Load This Pane
 1.681 ms:      15 ms - Load Other Pane
 1.696 ms:       2 ms - Init Menus
 1.698 ms:            - Load Done

MaxiTree: Rows=15, Nodes=17, Expanded=1
Remember Tree State: Yes (1 stored paths)
Check Existence of Subfolders in Tree: Yes
Icon Overlays: Yes
Pane 1: Items=14/14, Pane 2: Items=0/0, Catalog: Items=11
Get Folder Sizes: Yes
1. Is it required to have the Folder Size Cache loaded before the GUI comes up? If I don't use the cache, the sizes get calculated upon the GUI is shown as well.

2. Why is the Folder Size Cache load still reported when List>Cache folder sizes is un-ticked?

JLoftus
Posts: 577
Joined: 22 Jan 2014 14:58

Re: Improving XY load times?

Post by JLoftus »

If you're not currently using Folder Size Caching, you can delete fsc.dat and it will not be read at startup (I agree, this probably could be ignored if you're not using it).

When I delete that file, the entry is now missing from LoadTimes, and it is not recreated if the setting is disabled.

admin
Site Admin
Posts: 60357
Joined: 22 May 2004 16:48
Location: Win8.1 @100%, Win10 @100%
Contact:

Re: Improving XY load times?

Post by admin »

Yep, I can optimize that, and will.

BTW, I'd never expected that this can take so long:
Filehero wrote:718 ms: 673 ms - Load Folder Size Cache
How big is your cache??? (How many lines?)

JLoftus
Posts: 577
Joined: 22 Jan 2014 14:58

Re: Improving XY load times?

Post by JLoftus »

@Don, as another point of reference, if I enable FSC and close XY in my FLAC folder, with >10,000 subfolders, it only addded 69ms. The file is 2.5MB with 12,420 lines. I recall the FSC had been adding a few hundred ms before I disabled it, so maybe that's something that loses some efficiency over time?

Bigger issue for me, is not load time, rather refresh lag. Even with auto-refresh disabled, there are times when the folder sizes appear to refresh anyway (based on some action I've done). In this FLAC folder, that is over 7TB worth of data, the refresh takes a really long time, so I've been leaving Always Show Folder Sizes and Cache Folder Sizes both disabled, only forcing it occasionally if I need it. Instead, I have been using CTRL+Select and the "Calculate Folder Sizes" button, which for just a couple folders works very quickly (near instantly).

Another thing, Load MiniTree is taking 307ms.. I have AutoOptimize MiniTree enabled, does the optimize code get called on shutdown?

One other thing I notice, see the "Exist Drive M:\" etc? That seems to be getting called twice? Even though it's single-digit MS, I'd still rather not have it called twice.


XY ver: XYplorer 18.70.0105
OS: Windows 10, 64-bit
Loaded: 2018-02-25 11:53:57 in 1,705 ms

0 ms: 235 ms - Start Load
235 ms: 20 ms - Init Language
255 ms: 40 ms - Load Language
295 ms: 7 ms - Load Settings
302 ms: 67 ms - Init Custom File Icons
369 ms: 30 ms - Init Controls
399 ms: 25 ms - Init Graphics
424 ms: 9 ms - Exist Drive M:\
433 ms: 8 ms - Exist Drive N:\
441 ms: 8 ms - Exist Drive O:\
449 ms: 115 ms - Exist Drive P:\
564 ms: 29 ms - Init Tabs
593 ms: 62 ms - Create Info Panel
655 ms: 60 ms - Set Interface Font
715 ms: 27 ms - Init Panes
742 ms: 0 ms - Init Tree
742 ms: 0 ms - Add Spec Folders
742 ms: 0 ms - Add Links Folder
742 ms: 0 ms - Add Desktop Folder
742 ms: 0 ms - Add Documents Folder
742 ms: 0 ms - Add Downloads Folder
742 ms: 0 ms - Add User Folder
742 ms: 1 ms - Drives To Array
743 ms: 8 ms - Exist Drive M:\
751 ms: 8 ms - Exist Drive N:\
759 ms: 9 ms - Exist Drive O:\
768 ms: 8 ms - Exist Drive P:\
776 ms: 101 ms - Add Drives
877 ms: 1 ms - Add Portable Devices
878 ms: 1 ms - Init Start Path
879 ms: 8 ms - Tab Icons
887 ms: 1 ms - Load Catalog
888 ms: 3 ms - Load Tags
891 ms: 69 ms - Load Folder Size Cache
960 ms: 307 ms - Load Mini Tree
1,267 ms: 10 ms - Load Mini Tree 2
1,277 ms: 37 ms - Init Custom Toolbar Buttons
1,314 ms: 5 ms - Init UDC and KS
1,319 ms: 10 ms - Layout Window
1,329 ms: 3 ms - Layout Catalog
1,332 ms: 17 ms - Layout List
1,349 ms: 44 ms - Show Window
1,393 ms: 0 ms - Layout Catalog
1,393 ms: 60 ms - Layout List
1,453 ms: 140 ms - Load This Pane
1,593 ms: 112 ms - Load Other Pane
1,705 ms: 0 ms - Init Menus
1,705 ms: - Load Done

MiniTree: Rows=303, Nodes=314, Expanded=6, Mapped Drives=1
Icon Overlays: Yes
Pane 1: Items=44/9875, Pane 2: Items=25/25, Catalog: Items=2
Get Folder Sizes: Yes

admin
Site Admin
Posts: 60357
Joined: 22 May 2004 16:48
Location: Win8.1 @100%, Win10 @100%
Contact:

Re: Improving XY load times?

Post by admin »

The bigger the cache file the longer it loads, and the more memory it consumes. That's physics. Can't fix that within this universe.


Even with auto-refresh disabled, there are times ... When?


Load MiniTree is taking 307ms.. I have AutoOptimize MiniTree enabled ... Loadtime has nothing to do with AutoOptimize


the "Exist Drive M:\" etc? That seems to be getting called twice? ... Don't worry about it, that depends on the configuration.

Filehero
Posts: 2644
Joined: 27 Feb 2012 18:50
Location: Windows 10 Pro x64

Re: Improving XY load times?

Post by Filehero »

admin wrote:How big is your cache??? (How many lines?)
71.210 lines, ~ 18,5 MB.

I've never looked into the file. I'd expected most of the entries not to be there, because I've excluded the drives, like

Code: Select all

CacheFolderSizesExclude=C:\|D:\|E:\|F:\
They might be orphans from the time prior to introduction of this tweak. I'll delete the file right now and look what is going to happen the next days/weeks.

admin
Site Admin
Posts: 60357
Joined: 22 May 2004 16:48
Location: Win8.1 @100%, Win10 @100%
Contact:

Re: Improving XY load times?

Post by admin »

71.210 lines means 71.210 cached folders. Cough... :whistle:

Filehero
Posts: 2644
Joined: 27 Feb 2012 18:50
Location: Windows 10 Pro x64

Re: Improving XY load times?

Post by Filehero »

admin wrote:71.210 lines means 71.210 cached folders. Cough... :whistle:
Strange, after deletion and restart the very same fcs.dat reappears within a second although the list just has refreshed one folder. Even more, it again has all those folders which I thought be excluded from caching. :?:

Edit: Even with the cache is off, this file comes back again upon every restart. Manually deleting the entries doesn't help either, after restart the entries are back - although a tiny bit reduced to 71.345 lines.
There is no backup in AutoBackup. Does this make any sense?


In detail, I have excluded every drive except the one holding my backups. Could it be there is something wrong with my exclude syntax?

JLoftus
Posts: 577
Joined: 22 Jan 2014 14:58

Re: Improving XY load times?

Post by JLoftus »

I have not used that tweak, when I delete fsc.dat I thought it was not returning, but I confirm it does return for me as well (but just contains the header line "XYplorer Folder Size Cache v1")

admin
Site Admin
Posts: 60357
Joined: 22 May 2004 16:48
Location: Win8.1 @100%, Win10 @100%
Contact:

Re: Improving XY load times?

Post by admin »

When XY is running the FSC data are in its memory and it will write them completely to file whenever the file is written.

Use a fresh instance of XY to delete it.

admin
Site Admin
Posts: 60357
Joined: 22 May 2004 16:48
Location: Win8.1 @100%, Win10 @100%
Contact:

Re: Improving XY load times?

Post by admin »

Filehero wrote:In detail, I have excluded every drive except the one holding my backups. Could it be there is something wrong with my exclude syntax?
Nothing wrong. But the exclusion will not remove old data. It will just not add new data that match the exclusion.

Filehero
Posts: 2644
Joined: 27 Feb 2012 18:50
Location: Windows 10 Pro x64

Re: Improving XY load times?

Post by Filehero »

Code: Select all

v18.70.0106 - 2018-02-26 11:16
    % Folder Size Caching: Optimized reading and writing of the Folder Size 
      Cache at startup and shutdown. Now it's only read at startup when caching 
      is enabled (Configuration | Tree and List | List | Cache folder sizes), 
      and only written at shutdown when it is dirty.
      This will notably speed up startup and shutdown when your Folder Size 
      Cache is large.
Thanks, Don. Even with my ridiculously large cache startup is lightning fast again. :D

I also could get rid of the huge cache. :biggrin:

However, the exclude tweak doesn't seem to have an effect. With a new fsc.dat and

Code: Select all

CacheFolderSizesExclude=C:\|D:\|E:\|F:\
focusing two list tabs with folder paths on C: and E: results in these paths getting listed in the cache file although the leading "~" is not shown in the list views. I guess that's why the cache size exploded.

Have I missed another setting?

admin
Site Admin
Posts: 60357
Joined: 22 May 2004 16:48
Location: Win8.1 @100%, Win10 @100%
Contact:

Re: Improving XY load times?

Post by admin »

Filehero wrote:... I guess that's why the cache size exploded.
You are absolutely right! Stupid bug. Fix comes.

Filehero
Posts: 2644
Joined: 27 Feb 2012 18:50
Location: Windows 10 Pro x64

Re: Improving XY load times?

Post by Filehero »

admin wrote:Fix comes.
Fix confirmed, thanks.

PixelDrop
Posts: 38
Joined: 23 Jan 2018 20:43

Re: Improving XY load times?

Post by PixelDrop »

Maybe there should be a way to set an upper limit to folder size caching.

I finally got tired of Xyplorer starting really slow and figured I'd hunt down the issue.
It turned out that I had a nearly 100MB fsc.dat with 495,425 lines.
Most of which where folders on past hard drivers from years past.

Post Reply