Millions of Files

[tiago]

So I'm dealing with tons of files on external drives. With the following limitations:

1. I cant do a single 'backup to':

Code: Select all

Error:  	7 (00000007)
Desc:   	Out of memory
Dll:    	0
Proc:   	KFW@180, cntItems=3772385, cntItemFolders=0

Source: 	XYCOPY
XY ver: 	2.10.0082
OS:     	
Date:   	20/06/2016 04:57:01

8 GB total RAM here. As of now I'm splitting my tasks on bunches of no more than 2m files, which is pretty boring.

2. Tree drawing of +4.000 folders leads XYplorer to... misbehave, flashing other program's windows, periodically putting them on foreground when more folders are being added to that number. Too exotic.

3. All of a sudden XY shuts down. No scripts running, no ongoing copy, this seems to be related to the app uptime: generically speaking, after a week of execution it goes down. No warnings, no nothing, it simply closes. My wish here would be it auto-saving current set of tabs at least so in this event it could reopen last working set just like other tabbed apps often do. I tried some script based solutions and found boring to periodically do a manual save as in the event of this random problem I often have useless sets of folders to resource to.

4. [minor] Progress bar completely filled at the start of processing millions of files, displaying numeric '100%'.

[admin]

Millions of files... okay, you were asking for it.

1) So you are copying 3,772,385 items. Interesting task. I make a note for 17.10.

2) Tree drawing of +4.000 folders: Could not reproduce any issues here with those numbers.

3) AFAIK XY has no memory issues. I know of users that are running it for many months non-stop. So I assume some external force is causing this, probably some shell extension.

4) Tried to reproduce this in the lab (by simulation, I don't have millions of files), but failed. Mystery.

[Enternal]

3) Well I do think XYplorer has some slight memory problems if you're in an thumbnails mode a lot. After a while, there's a noticeable slowdown of XYplorer. It's a quick fix for me though since I simply saved and restart and voiila! I'm back to business with a quick XYplorer.

And wowza! I deal with hundreds of thousands of files (last me I was messing with over 700,000+ files) bu never millions! That's awes-- uhh I meant crazy!

[admin]

3) Well I do think XYplorer has some slight memory problems if you're in an thumbnails mode a lot. After a while, there's a noticeable slowdown of XYplorer. It's a quick fix for me though since I simply saved and restart and voiila! I'm back to business with a quick XYplorer.

I'm in thumbnails mode a lot myself and don't note any problems. Where exactly do you note the slowdown?

[tiago]

Another one:
it's 35 minutes now I'm waiting for the report of +106k files to be drawn so I can make record of them.

[admin]

Which kind of report?

[tiago]

The one from the button 'reports' at the custom copy window.
-still waiting...

[Enternal]

3) Well I do think XYplorer has some slight memory problems if you're in an thumbnails mode a lot. After a while, there's a noticeable slowdown of XYplorer. It's a quick fix for me though since I simply saved and restart and voiila! I'm back to business with a quick XYplorer.

I'm in thumbnails mode a lot myself and don't note any problems. Where exactly do you note the slowdown?

I noticed it in selecting files and transverse directories. My case might be a bit different though mainly because I rarely restart my computer or any of the programs running on it. So it usually becomes a problem after several days rather than immediately.

[admin]

You notice that selecting files gets slower? This should not take any noticeable time at all.

[admin]

The one from the button 'reports' at the custom copy window.
-still waiting...

There are several different reports under this button. Which one did you choose?

[tiago]

That's the point: pressing the main button I got no options at all to choose from.

Follow up: I expected the whole day for a prompt. The mouse pointer displayed the 'processing' animation each time I went to check. Using 3rd party tools I made a .zip replicating the dir structure, unzipped it into another folder, had +65k errors on one case and winRAR, which gives ~3x the size of what a XYplorer report would be, successfully exported the error report to the clipboard which was saved into a ~15 MB text file.

[admin]

I don't know what went wrong, but the malfunction was elsewhere. There is hardly any code under the Reports button, and also showing huge reports should be smooth and fast.

So, let's wait for 17.10 where I will split jobs into bites of one million files each (or so).

[tiago]

thanks.