Working with large number of files

[aka_db]

Hi,

I am trying to run reports or scripts for large numbers of files in recursive directories (400,000 files), we are trying to sort files out and decide what to archive etc..

If I run a Metadata Report to .csv I get the attached error.

I was wondering if I ran a script to retrieve just the data that I want, could I write this data to a file on the fly to eliminate a build up of records in memory? If this is possible, what would the syntax be? I tried a few pipe type commands, but these didn't seem to work.

Thanks, db.

[aka_db]

Hi, an update...

After my original process crashed after some 40 hours of processing, I changed the parameters and excluded about 20,000 directories, just included the files and actually managed to get a result after about 4 hours.

I would still like to find a way to push the results on the fly to a file if possible, that way if it crashes I could just restart from where it left off.

Cheers, db.

[binocular222]

You can make a loop:
- From item 1 to 50,000: Write to file1.txt
- From item 50,001 to 100,000: Write to file2.txt
etc

Then consolidate these .txt file
Refer to help file > Advanced Topic > Scripting Command reference
writefile(filename, data, [on_exist], [mode])

[admin]

After my original process crashed after some 40 hours of processing...

GOSH!

Which button did you press that started this process? Or was it a script?