Scripting Memory-Leak / Speed Degrading

Please check the FAQ (https://www.xyplorer.com/faq.php) before posting a question...
fishgod
Posts: 199
Joined: 03 Feb 2008 00:40
Location: Sankt Augustin (near Bonn), Germany

Scripting Memory-Leak / Speed Degrading

Post by fishgod » 04 Sep 2019 10:35

I discovered when running a Script for a lot of items via foreach over all <selitems> the execution speed degrades per item.
First items are processed at speeds of 10 items per second and more, while later items take multiple seconds per item.
The script per item is always the same, as also is the work to be done (lots of regex/string-operations, function-calls, etc... - quite complex script).
The Memory-Footprint from XYplorer raises from about 80MB to over 200MB while processing 500 items, and stays at 200MB even after the script has finished. I thought at least after the script has finished, all the memory the scripting-engine has consumed will be released.

Happens to latest Beta 20.30.0015 as also to Last official release 20.30 (did not test other releases)
Operating System: Win7 x64 / almost allways newest XY-beta
totally XYscripting-addicted

highend
Posts: 8304
Joined: 06 Feb 2011 00:33

Re: Scripting Memory-Leak / Speed Degrading

Post by highend » 04 Sep 2019 11:51

Never seen this behavior, so without the script to test...
One of my scripts helped you out? Please donate via Paypal or highend (at) web (dot) de

admin
Site Admin
Posts: 47703
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Scripting Memory-Leak / Speed Degrading

Post by admin » 04 Sep 2019 11:56

Generally, measuring memory footprints under modern Windows is not trivial. There is a lot of smartness and betting going on. Even if an app releases the memory, Windows might still clinch to it.

But as highend said, we need to see the script.

fishgod
Posts: 199
Joined: 03 Feb 2008 00:40
Location: Sankt Augustin (near Bonn), Germany

Re: Scripting Memory-Leak / Speed Degrading

Post by fishgod » 04 Sep 2019 13:30

Thx for the quick responses, memory is not the core problem here, that was just a side-effect that I observed and didn't want to left unmentioned (memory-values come from windows task-manager fyi).
The script as such is quite lengthy, calling a lot of subs and user-functions. Let me see if I can nail it down before I post a bunch of code you need to analyse that might have nothing to do with the issue.
Operating System: Win7 x64 / almost allways newest XY-beta
totally XYscripting-addicted

PeterH
Posts: 2579
Joined: 21 Nov 2005 20:39
Location: Germany

Re: Scripting Memory-Leak / Speed Degrading

Post by PeterH » 04 Sep 2019 17:40

One thing we had a long time ago: if, for example in al loop, you concatenate and concatenate data to a variable, this variable will become fragmented, and access to it might get slower and slower.
Don't know if this can be (part of) your problem.
W7(x64) SP1 German
( +WXP SP3 )

fishgod
Posts: 199
Joined: 03 Feb 2008 00:40
Location: Sankt Augustin (near Bonn), Germany

Re: Scripting Memory-Leak / Speed Degrading

Post by fishgod » 11 Sep 2019 23:32

I did a lot of testing and observation, and I think I now know the exact source of the slowdown.

I run this on a folder with 66 Media (MKV-Video) Files: (the more, the slower it gets)
::load 'xy-slowdown'
xy-slowdown.xys
(866 Bytes) Downloaded 3 times

Code: Select all

function parseFolder($path) {
  $files = listfolder($path, "*.mkv");
  $log = "";
  foreach($file, $files) {
    $log .= callFunction('mediaInfo', $file)." ";
  }
  text $log;
}

function callFunction($func, $path) {
  global $ML_FUNC, $ML_PATH, $ML_RETURN;
  $ML_FUNC = $func;
  $ML_PATH = $path;
  load 'xy-slowdown2.xys', '_callFunction';
  return $ML_RETURN;
}


"Execute MediaInfo"
  parseFolder(<curpath>);
xy-slowdown2.xys
(2.4 KiB) Downloaded 3 times

Code: Select all

//why does this include has influence of the whole execution speed
//messured below
//it also doesn't make a difference what script is included here,
//it just needs to have a bunch of code, the more the slower
include_once 'MovieLibMisc.inc.xys';

"Call Function : _callFunction"
  global $ML_FUNC, $ML_PATH, $ML_RETURN;
  $ML_RETURN = eval($ML_FUNC.'($ML_PATH)');

function mediaInfo($file) {
  $MEDIAINFO = "C:\PG\MediaInfo-CLI\MediaInfo.exe";
  $f = gpc($file, 'base');
  $info = runret($MEDIAINFO. ' "'.$file.'"');
  
  //start time messure
  $time = now("msecs");
  $target = '$start';
  foreach($line, $info, "<crlf>") {
    if(substr(trim($line), 0, 5)=='Video') {
      $target = '$video';
    } elseif(substr(trim($line), 0, 5)=='Audio') {
      $target = '$audio';
    } elseif(substr(trim($line), 0, 4)=='Text') {
      $target = '$text';
    } elseif(substr(trim($line), 0, 4)=='Menu') {
      $target = '$menu';
    } else {
      *$target .= "<crlf>".$line;
    }
  }
  //stop time messure
  $time = now("msecs")-$time;
  status "$time  -  $f";
  wait;
  
  $output = '/*do extended parsing of $video, $audio, $text, $menu, but this does not matter*/';
  
  return $time;
}
Output of execution times (from the text command at the end of parseFolder-function):

Code: Select all

78 78 78 78 78 78 78 78 78 93 78 94 93 94 94 94 93 94 109 93 93 93 94 93 109 125 109 109 125 109 110 124 125 109 110 125 125 140 125 140 124 140 125 124 125 141 141 140 141 141 140 141 141 140 140 156 156 140 156 156 156 156 156 156 156 156 
Time has roughly doubled here, the effect is even worse for my real world script where a lot more code is involved).
The behaviour is reproducable every time.

So, now what causes the slowdown?
I think, by the repeated execution of load 'xy-slowdown2.xys', '_callFunction'; the MovieLibMisc.inc.xys-file is included over and over again, and somehow pollutes the xy-scripting engine, causing general slowdown of execution.

The effect does not happen if I do the following:
Add include_once 'xy-slowdown2.xys'; to the beginning of xy-slowdown.xys.
Replace $log .= callFunction('mediaInfo', $file)." "; with $log .= eval($func.'($path)')." ";.
In this example that would be an acceptable solution, but in the real world I use xy-slowdown.xys in a custom column, and want it keep short for performance reasons. callFunction is used conditionally and only if I need to call it, the other script is loaded. If I load it every time, it slows down browsing. If the performance-issue cannot be solved a conditional include would also be solution.

For completeness the MovieLibMisc.inc.xys file that is used with include_once (and also the supplement file included there) is attached to this post. It doesn't matter what exact script is included. I even tried to include a different ~80kb script, the effect is the same.
Attachments
functions.inc.xys
(7.97 KiB) Downloaded 3 times
movieLibMisc.inc.xys
(6.03 KiB) Downloaded 2 times
Operating System: Win7 x64 / almost allways newest XY-beta
totally XYscripting-addicted

fishgod
Posts: 199
Joined: 03 Feb 2008 00:40
Location: Sankt Augustin (near Bonn), Germany

Re: Scripting Memory-Leak / Speed Degrading

Post by fishgod » 12 Sep 2019 01:49

Finally found a usable workaround.
I didn't know (until now), that I can replace include/include_once with load for the case the file to be included contains only function-definitions to build a conditional include.

So this is my callFunction() now, that allows:
  • to call a function named by a variable
  • residing in a different script
  • that is loaded on demand
  • and loaded only once.

Code: Select all

function callFunction($func, $path) {
  global $ML_LIBSLOADED;
  if($ML_LIBSLOADED != 'true') {
    load 'movieLibCode.xys'; //functions are located in here
    $ML_LIBSLOADED = 'true';
  }
  return eval($func.'($path)');
}
That load gives me a warning, that there are no valid lines or visible lables, but that can easly be worked around by creating a visible label in the inluded file that does just nothing, like so:

Code: Select all

"Dummy"
  wait;
Operating System: Win7 x64 / almost allways newest XY-beta
totally XYscripting-addicted

admin
Site Admin
Posts: 47703
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Scripting Memory-Leak / Speed Degrading

Post by admin » 13 Sep 2019 09:11

Nobody has downloaded your samples yet. :) It just looks like 2 hours of complicated testing of something you say you've found the solution to yourself in the meantime. :)

I moved this topic into "Tips & Tricks, Questions & Answers". But there might be actually a wish in it: Would it be interesting to add a way to define some permanent global user functions? Like a user-made function library that's always available in memory and does not have to be explicitly included. It would make it harder to share scripts, but easier (and faster) to use some of them. OTOH it would slow down all scripts somewhat. So maybe it's not a good idea after all.

fishgod
Posts: 199
Joined: 03 Feb 2008 00:40
Location: Sankt Augustin (near Bonn), Germany

Re: Scripting Memory-Leak / Speed Degrading

Post by fishgod » 13 Sep 2019 22:29

The thing is, that I don't understand why the whole script execution-speed (simple string-manipulation as shown above) is affected by repeatedly loading a big script (or a script that includes other big scripts, even with include_once, as shown in above post)
I did another test and inserted the code from included files directly into the second script, that is loaded via load. The effect is exactly the same, so this has nothing to do with the include. It also doesn't matter what code this is or if it is called, the only relevant thing seem to be code-size (or maybe function count).

What I found is more like a workaround than a solution, by using the load only once with a global variable to guard.

Global functions sound like a nice idea. This would be especially usefull for scripted columns where loading a big script for every row may takes more time than the actual execution.
That brings up another thought on execution of scripted columns: It seems that the script is parsed and includes are done for every row. Why not load the script once and only execute it for every row. That would save a considerable amount of time. (by the way, thats why I come up with this conditional loading in the first place)

It would also be nice to have the include/include_once-statement evaluated at runtime, this way I wouldn't need the workaround with the load-statement to include my function-definitions.
Operating System: Win7 x64 / almost allways newest XY-beta
totally XYscripting-addicted

admin
Site Admin
Posts: 47703
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Scripting Memory-Leak / Speed Degrading

Post by admin » 14 Sep 2019 10:03

OK, I'll check that. It's indeed interesting...

admin
Site Admin
Posts: 47703
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Scripting Memory-Leak / Speed Degrading

Post by admin » 14 Sep 2019 12:31

Ok, I tried to make it run but it does not work without MediaInfo.exe. Can you provide a test script without such dependencies?

highend
Posts: 8304
Joined: 06 Feb 2011 00:33

Re: Scripting Memory-Leak / Speed Degrading

Post by highend » 14 Sep 2019 12:47

It's a free, portable cli tool, just get it from here:
https://mediaarea.net/download/binary/m ... ws_x64.zip
One of my scripts helped you out? Please donate via Paypal or highend (at) web (dot) de

admin
Site Admin
Posts: 47703
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Scripting Memory-Leak / Speed Degrading

Post by admin » 14 Sep 2019 13:07

Merci!

admin
Site Admin
Posts: 47703
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Scripting Memory-Leak / Speed Degrading

Post by admin » 14 Sep 2019 13:58

I can confirm the case:

Code: Select all

78 63 63 62 78 62 78 63 78 63 79 62 62 78 78 78 93 78 94 78 93 78 94 78 93 78 94 94 94 94 94 94 94 93 109 94 109 94 110 94 110 93 109 109 109 109 125 109 125 109 125 109 125 109 110 125 125 125 140 125 125 125 141 125 140 125 140 125 141 125 125 140 141 140 125 140 156 141 156 140 
Going on a hunt... :twisted:

admin
Site Admin
Posts: 47703
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Scripting Memory-Leak / Speed Degrading

Post by admin » 14 Sep 2019 17:12

Well, 3 hours later I can say this:

1) There is no memory leak whatsoever in XYplorer. All 100% tight.
2) Loading the file "xy-slowdown2.xys" again and again from disk does not cost any measurable time. I completely removed this internally and replaced it by memory-cached contents, and nothing was won by this.
3) I wonder if this might be related to MediaInfo.exe??? -> Next test: replace MediaInfo.exe with some other file... tomorrow...
4) Anyway and notwithstanding: Off to party! :beer:

Post Reply