Page 1 of 1

Problem reading large files.

Posted: 25 Nov 2011 12:34
by tiago
I cannot transfer an amount of data (~500 MB) for XY to deal with using clipboard as source, setting it as a variable
I can't make readfile() to read a file containing such data.

What are the limits? Documentation doesn't mention any.
It should be limited by RAM available amount only, isn't it?

Re: Problem reading large files.

Posted: 29 Nov 2011 08:59
by admin
What happens when you try?

Re: Problem reading large files.

Posted: 29 Nov 2011 14:23
by tiago
Error: 7 (<several zeroes>7)
Desc: Out of Memory
Dll: 0
Proc: script_Process
But there is memory, in one machine I had 4 GB and got the same for ~500/~700 MB files, one per script execution.
The error pops after a minute or so of script execution at the readfile() execution.
Same goes with a v.9.90 version (current: 10.40.0101).

Re: Problem reading large files.

Posted: 29 Nov 2011 14:36
by admin
What you do with scripting is scary once again. :mrgreen:

An out-of-memory error is usually not something I can fix in my code. It's a limit that's bigger than me.

Nevertheless, theoretically ReadFile should be able to handle "4GB minus 1" bytes. Without seeing your code it's not possible to say what's going on.

Re: Problem reading large files.

Posted: 29 Nov 2011 16:19
by tiago
Not a big deal.
I thought the problem was related to variables not accepting such a large input.
This will do the same anyway and prove that the problem is limited to readfile():

Code: Select all

readfile(<curitem>); echo "ok";
Same goes if you happen to have a large .avi file as <curitem>. Save your precious time.
Yet to test at which (fixed?) size of file the error begins to happen.

Code to generate a dummy file:

Code: Select all

//default values will generate a 64 MB file
//watch out as it is memory hungry.

   $trgt = "25"; // '25' creates a 64 MB file...
   $data = "10";
   writefile("Dummy.txt", "$data", o); // ensure it's clean

   $ctrl = 1;
   WHILE ($ctrl <= $trgt) {
   status "Op $ctrl/$trgt done...";
   writefile("Dummy.txt", "$data", a);
   $data = readfile("Dummy.txt");
   $ctrl++;
                        }
   echo "task finished";