"Out of String Space" when generating large Reports

[gernot.q]

Hi,
I'm currently generating reports for some old drives with the report function in "Current Folder".

For smaller drives, this works fine.
For larger drives with a real lot of files (like > 1 million) I'm running into trouble and getting error messages (see below for an example)
This is regardless of the kind of report I use (Classic Directory Dump, Tree Structure, or one of the CSV Exports)
Reports go directly to file, not clipboard etc.
With quitting and restarting XYPlorer immediately before the report, things seem to work a little better.
But still, with the larger drives, reporting stops with an error at, say, 60-70% of the drive contents.

Is there a workaround for this?
I would like to avoid doing the report for each of the many subfolders of the root of the drive if there is another solution ;-)

Cheers,
GQ
-----------------------------------------------------------
Error: 14 (0x0000000E)
Desc: Out of string space
Dll: 0
Proc: ReportByType@250: 0

Source: XYplorer
XY ver: XYplorer 21.40.0100
OS: Windows 10 Professional, 64-bit, Release 2009, Build 19042.685
Locale: 1033 (en-US)
ANSI: 0, ACP: 1252 (ANSI - Latin I)
Font: Segoe UI 12,86, DBCS: No
DPI: 144 (150%), 3840x2160

Date: 2021-01-05 13:10:31
---------------------------------------------------------------

[highend]

Just out of curiosity, how large is the file that has been written when the error pops up?

[gernot.q]

Sadly, no part file is saved, so I can't really say.
Estimating from what I was able to generate on another drive, we are in the 300-400MB category here.
A really classic dir dump with dir /s > bigfile.txt generates 75 MB with about 1,4 million lines (in just a little over a minute ...)
I was trying to generate a CSV - just estimating that would be about 5 times as big.

[highend]

So it tries to accumulate all data in a string first before trying to write it to the output file.

Apart from doing the report for each folder in the root there is probably no workaround at the moment...

[gernot.q]

Yeah I can see how it might be unfortunate to assemble a string of about half a Gig size

So I'll stick to the "dir /s" method for just getting a file list and do the detailed reports only for the subdirs where I really need them.

Thanks for the info.

[highend]

You could also use a script that does the report per folder and after each folder appends the string to the output file.
No stress on string variable size (at least not if a single folder has >1 mio files...).

But the thing is:
01. It will be slower than XY doing it internally (script is interpreted, not compiled)
But you can show a progress report while doing this...

02. Formatting it in such a nice way like the internal report would take some time...

But at least it would be a viable alternative...

Bad thing: No time to help you on that today

[highend]

Doing it with scripting won't work. Calculating total folder sizes will slow it down too much...

Try the attachment.

Put it anywhere you like, open a command prompt and start it with the parameter
/path="<a full path>"

It will open a save file requester and asks for the file to save the results to.

Try it with a smaller folder first (e.g. with a few thousand files).

When it's done (it's about 2x-3x times faster than XY's report), look a the result file and tell me if it looks fine...

If you need a differently formatted date, you can use the second possible parameter /dateFormat="<definition>"
The internal default is: "%yyyy-%mm-%dd %hh:%ii:%ss"
so e.g. for german date format you could use
/dateFormat="%dd.%mm.%yyyy %hh:%ii:%ss"
%ii is minutes^^

If everything looks fine I'll make it a console app to provide a bit of processing output...

Oh, ofc memory isn't a problem here, each line is written to the file once it's created^^

[gernot.q]

Hey thanks, this generates a nicer output than my "dir /s" method.
Works fine on the disk in question
Thanks a lot for your help

In the meantime, I've been thinking that MD5 sums for establishing file identity would come in handy.
So I'll probably look into freeware MD5 tools next.

[highend]

md5... you know that this will probably take ages to compute for larger files?

I have a xxhsum (32 bit) implementation here that I could use...

A speed comparison:

Code: Select all

6716 folders, 44966 files, 18,5 GB

No hash:   5.8 secs
xxH32  :   17  secs
md5    :   92  secs

[gernot.q]

Thanks for the suggestion. However, I deem any 32 bit hash too collision-prone since we are talking about tens of millions of potentially different files here.
If my calculations are right, provided you have a little over 40 million files, you already got a purely statistical collision probability of about 1% for random input set (based on the assumption that hash results are of equal probability). I don't know about xxH, but from what I gather about hash functions, most of the real hash algorithms perform a lot worse collision-wise than my "perfectly random assumption" example does.

Also, I'd like to have md5, since it's rather widely used.
Sadly, my holidays are over now, so it might take me some time to find a decent solution (and run it on my disks with an expected duration of x days per disk )

[highend]

v0.1.0.1

Create folder report_x64_v0.1.0.1.zip

Console window, outputs the current folder it's working on
When it's finished it looks like this:

Code: Select all

Processing: "D:\Tools\YDL-UI\locales\en\"

FINISHED in: 5 second(s), 541 millisecond(s)

Press any key to continue...

Use /useHash=1

to let it output lines (for files) like this (it's an md5 hash...):

Code: Select all

@Homepage.url                                            66  2020-06-16 10:12:31 [626ca0a33ea2d8f8db49914b16da2490]

[highend]

Probably the final version...

v0.1.0.2

Create folder report_x64_v0.1.0.2.zip

Added a bit more progress output.

Shows xxx / xxxx number of the path that is currently processed
If /useHash=1 is used, additionally the currently processed file

[gernot.q]

Wow, nice, this might really save me some time

I'll test this once i'm home from work tonight...

[gernot.q]

As was foreseeable, the program took some hours to complete on my test case -- well, no problem, I wanted it that way
Works fine and saves me time so I donated via Paypal.

There's one thing left I could think of that would be nice: An option to generate a CSV output instead of the directory dump with lines like "name|size|date|hash".
This would mainly imply just leaving out the formatting and summary lines of the dir dump and separating the fields by "|" instead of a number of blanks.
That way everything could be easily imported in any database format in the future if necessary at a later time.
If this is doable without too much of an effort, it would be a real improvement

Otherwise, we can just leave it at that. I can always do the necessary transformations myself when needed by using REGEX-based replacement tools or something

[highend]

Thanks for the donation.

v0.1.0.3 - Supports the switch /useCSV=<x>
Default is 0: Standard report, no csv
1: Output it with "|" as the separator between name|size|date|md5 (md5 only if /useHash=1 is used)

Create folder report_x64_v0.1.0.3.zip