increase general performance on scripting please!

Features wanted...
PeterH
Posts: 2830
Joined: 21 Nov 2005 20:39
Location: DE W11Pro 24H2, 1920*1200*100% 3840*2160*150%

Re: increase general performance on scripting please!

Post by PeterH »

admin wrote:
PeterH wrote:
admin wrote:String concatenation: For each adding the whole string must be allocated and copied in memory. However, adding 5000 items at once can be done very fast because only one allocation and byte copying is necessary.
If it were just like you said I'd await that double number of items means double number of time, but not 4 times.
That is: about linear, not quadratic?
Quick google search brings this among thousands of others:
http://axcs.blogspot.com/2006/07/asp-ti ... tring.html
Silly me :oops:
Quite intensively I thought about using linear more time for concatenating the Nth value - and forgot the also linear growing number of N iterations *for* the Nth value - together quadratic, of course.

zer0
Posts: 2676
Joined: 19 Jan 2009 20:11

Re: increase general performance on scripting please!

Post by zer0 »

admin wrote:Of course not.
Hmm, so my problem is likely to get worse as time goes on as I only see the list of S&R commands increasing. Damn! :evil:
Reporting a bug? Have a wish? Got a question? Use search - View roadmap - FAQs: Forum + XY site
Windows 7/10
Always using the latest stable two-decimal build

tiago
Posts: 589
Joined: 14 Feb 2011 21:41

Re: increase general performance on scripting please!

Post by tiago »

admin wrote:
Jibz wrote:I tend to agree that a factor 4 in the loop logic is probably not worth the investment, unless it is something really easy to do. For munching text it would be far better to let the script call an external tool that specializes in that.
When I said "I could improve" that was past tense: I did it.

Yes, correct: Appending to ever-growing strings sucks. The natural solution for this is arrays. Some day I will add them.
I'm a week and a half away from running tests onto the controlled test environment. :cry:
In hopes dramatically means it. :mrgreen:
I just have to be thankful for any performance gains made in this area.
Strictly speaking (almost) all the text crunching I have in mind comes in the form of converting paths to<>from online/offline and/or redistribution of files onto storage disks. All about file management, IOW.
Power-hungry user!!!

tiago
Posts: 589
Joined: 14 Feb 2011 21:41

Re: increase general performance on scripting please!

Post by tiago »

:shock:
I managed to produce a similar input file on another (older!) machine and applied the same script with an upgraded version of XY.

On that "control task" I just killed the app after 60 hours of running as I did the same with EditPad in seconds as suggested. Then the ETA was 20 hours for an 11 MB file (based on previous experiences with 10 hours/5 MB files of similar content). No, that wasn't an infinite loop as I run several small tests to discard such a thing prior to running the real deal.

!!!same 11 MB was processed in exacts 10 seconds on new version!!!

I just hope the gain is not limited to foreach loops, despite the whole thread impression tells me so.

Anyway it's an humongous improvement for which I thank admin for!
Power-hungry user!!!

tiago
Posts: 589
Joined: 14 Feb 2011 21:41

Re: increase general performance on scripting please!

Post by tiago »

The problem: I have a script which monitors a folder and after 15000 files it must move the pack somewhere else. In the meantime more files are filling the folder. The only way I can currently do that is by using folderreport/formatlist/replace script commands taking lots of time each total files' verification cycle. Monitored folders will be filled after ~5-10min, and each script cycle is taking ~15-18 min to check the amount of files so the whole math won't zero. While getting total of files should take no time, you must think that the calc is: 15-18 min for each of the up to 15000 files for just ONE verification cycle. Unreasonable as it is.

See the timings:

Code: Select all

Count: 21.387. (human friendly number; SCD only and no other mixed extensions)

Start:    21/11/2011 16:21:49.
Proc 1 = '21/11/2011 16:21:51' (after folderreport, total)
Proc 2 = '21/11/2011 16:21:51' (after formatlist by EXT)
Proc 3 = '21/11/2011 16:42:03' (after replacing '|' by CRLF) - 21 minutes!

Finished: 21/11/2011 16:42:03. (after count of filtered list)

----------------------------------------------------------------------------

Count: 21.387. ($cntMX) Total: 105.916. ($cntMXproc1) (human friendly number; mixed extensions)

Start:    21/11/2011 17:08:46.                                $refdt
Proc 1 = '21/11/2011 17:08:56' (after folderreport, total)    $rfdtPROC1
Proc1A = '21/11/2011 17:08:57' (after count of total tokens)  $rfdtPROC1A
Proc 2 = '21/11/2011 17:08:58' (after formatlist by EXT)      $rfdtPROC2
Proc 3 = '21/11/2011 17:29:11' (after replacing '|' by CRLF)  $rfdtPROC3 - 21 minutes!

Finished: 21/11/2011 17:29:12. (after count of filtered list) $refdate
So here are some questions:

1. how to improve timing on acquiring recursive count of items?

2. is it possible to have direct calculation of time differences from date(now) and date(now2), in days, hours, mins and seconds? Currently I must use this script: http://www.xyplorer.com/xyfc/viewtopic.php?f=2&t=6226 , which is hard to manage on very large scripts.

3. is there a way to suspend script execution until all files are moved to another location? For now I have to cheat by using wait() but it's a cheap cheat as time will differ depending on the (unknown) amount of files involved.

4. is there a way for replace() getting faster?

the test script used for the timings:

Code: Select all

   $refdt = now("");
// step;
          $files = folderreport("files", "r", , "r", , "|");
   $rfdtPROC1 = now("");
   $cntMXproc1 = gettoken($files, count, "|");
   $rfdtPROC1A = now("");
          $files = formatlist($files, "f", "|", "*.scd"); // input extension here! ======================================
   $rfdtPROC2 = now("");
          $lst = replace($files, "|", <crlf>);
   $rfdtPROC3 = now("");
   $cntMX = gettoken($lst, count, "<crlf>");

   $refdate = now("");
   

   $msg = "Count: $cntMX. Total: $cntMXproc1.<crlf 2>Start:    $refdt.<crlf>Proc 1 = '$rfdtPROC1' (after folderreport, total)<crlf>Proc1A = '$rfdtPROC1A' (after count of total tokens)<crlf>Proc 2 = '$rfdtPROC2' (after formatlist by EXT)<crlf>Proc 3 = '$rfdtPROC3' (after replacing '|' by CRLF)<crlf 2>Finished: $refdate. (after count of filtered list)";
   writefile("X Timing Gettoken FolderReport COUNT.txt", "$msg<crlf>", a);
   text $msg;
Power-hungry user!!!

admin
Site Admin
Posts: 67076
Joined: 22 May 2004 16:48
Location: Win8.1, Win10, Win11, all @100%
Contact:

Re: increase general performance on scripting please!

Post by admin »

The secret of speed is good code. As far as I see I've done my part. Hint hint...

TheQwerty
Posts: 4373
Joined: 03 Aug 2007 22:30

Re: increase general performance on scripting please!

Post by TheQwerty »

How about when using flat view?

Code: Select all

goto('? /:flatfiles');
$count = get('CountItems');
EDIT: The thought being that all of your attempts are building strings, but if you only care about the number of files let XY's more optimized functions do the real work.

zer0
Posts: 2676
Joined: 19 Jan 2009 20:11

Re: increase general performance on scripting please!

Post by zer0 »

admin wrote:The secret of speed is good code. As far as I see I've done my part. Hint hint...
Good code won't always make things faster. It may help in certain situations, but it's no silver bullet. For example, I have a script that runs circa 200 rename commands, one per line, one after the other. In my opinion, it takes longer than it should -- several seconds -- but my expectations may be unrealistic because XYplorer may not be able to run them any faster than it already does. Is that the case and, if yes, where is the bottleneck?
Reporting a bug? Have a wish? Got a question? Use search - View roadmap - FAQs: Forum + XY site
Windows 7/10
Always using the latest stable two-decimal build

tiago
Posts: 589
Joined: 14 Feb 2011 21:41

Re: increase general performance on scripting please!

Post by tiago »

TheQwerty: loading files on list takes a lot of time if you consider that each verification cycle must run within milliseconds, the required time for another processed file to join the list thus increasing group's total count. Not to mention the source folders are scattered away.

zer0: totally agree. The first snippet shows it in details dealing with a total of 21.387 substitutions.

...

Posted script is not bad code, is just a sample on how inconvenient the late of replace() can be. I could simply equalize all separators and the timing magically gets faster.
Problem is that AFAICR delete(), copy(), copyas(), moveto(), copyto(), backupto() all require (hardcoded!) "|" (pipe as separator) which will require conversion to/from <crlf> as human friendly reports and logs must contain <crlf>ed lists.

Far away from ideal I currently must count operations which statistically gives me a number of files varying from ~8k and ~19k each 105000 verification cycles when I should instead distribute them the closest possible to the FAT limit of ~23k, equally, beautifully and organized, which is unfair.
Remote locations (ie files that are not visible in current list) should be granted a way to provide us feedback with their current total of items (folders, files, folders and files) recursively or not.
Measuring and its control is a key concept on computing... :roll:

Script'ed move operations should allow moving all listed files at once like when you just select a folder containing a huge amount of files and drag it somewhere else in no time inside the current partition of course.

Below script is aimed to help on moving a huge amount of files:

Code: Select all

   $refdt = now("");

          $files = folderreport("files", "r", , "r", , "<crlf>");
   $rfdtPROC1 = now("");
   $cntMXproc1 = gettoken($files, count, "<crlf>");
   $rfdtPROC1A = now("");
          $lst = formatlist($files, "f", "<crlf>", "*.scd");
   $rfdtPROC2 = now("");

   foreach ($tk, $lst, "<crlf>") { moveas "*.?", "D:\Tests\MOVED\", "$tk"; }
   $rfdtPROC3 = now("");
   $cntMX = gettoken($lst, count, "<crlf>");

   $refdate = now("");
   

   $msg = "Count: $cntMX. Total: $cntMXproc1.<crlf 2>Start:    $refdt.<crlf>Proc 1 = '$rfdtPROC1' (after folderreport, total)<crlf>Proc1A = '$rfdtPROC1A' (after count of total tokens)<crlf>Proc 2 = '$rfdtPROC2' (after formatlist by EXT)<crlf>Proc 3 = '$rfdtPROC3' (after MOVEAS)<crlf 2>Finished: $refdate. (after count of filtered list)";
   writefile("X Timing MoveAs.txt", "$msg<crlf>", a);
   text $msg;
Test results:

Code: Select all

Count: 21385. Total: 21386.

Start:    22/11/2011 12:04:24.
Proc 1 = '22/11/2011 12:04:26' (after folderreport, total)
Proc1A = '22/11/2011 12:04:26' (after count of total tokens)
Proc 2 = '22/11/2011 12:04:26' (after formatlist by EXT)
Proc 3 = '22/11/2011 12:19:35' (after MOVEAS) - 15 minutes!!!

Finished: 22/11/2011 12:19:35. (after count of filtered list)
Power-hungry user!!!

tiago
Posts: 589
Joined: 14 Feb 2011 21:41

Re: increase general performance on scripting please!

Post by tiago »

for instance my statistical way of getting files count just failed at its 3rd run (crashed on 4th) with a total of ~21k files (1st: ~14k, 2nd: ~19k, 3rd: ~17k) after almost 20 hours running and several others of tests until I got the number of 105.000 verification cycles, now lowered to 84.000.

:evil:
Power-hungry user!!!

tiago
Posts: 589
Joined: 14 Feb 2011 21:41

Re: increase general performance on scripting please!

Post by tiago »

Can't locate the exact post, perhaps this isn't the correct thread I should be looking at but anyway:
a task I had running for +4 days and had to kill because not seeing an end to it, I've completed in ~23 hours after rewriting a code so concatenation is made in a temporary text file instead of a string in memory. Thanks to the person who gave the idea!
Power-hungry user!!!

Post Reply