Scripting Memory-Leak / Speed Degrading

Please check the FAQ (https://www.xyplorer.com/faq.php) before posting a question...
fishgod
Posts: 199
Joined: 03 Feb 2008 00:40
Location: Sankt Augustin (near Bonn), Germany

Re: Scripting Memory-Leak / Speed Degrading

Post by fishgod » 14 Sep 2019 17:24

I appreaciate the time you allready invested, but I don't think it has something to do with mediainfo.exe, because the workaround above does the same calls to mediainfo.exe without the speed-degradation.
As far as I can say from observation, the thing is the repeated use of load causes something.
The slowdown is not permanent, but only only happens while a script is running. When the script has finished and is started again, it starts fast and degrades again.
Operating System: Win7 x64 / almost allways newest XY-beta
totally XYscripting-addicted

admin
Site Admin
Posts: 48013
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Scripting Memory-Leak / Speed Degrading

Post by admin » 14 Sep 2019 17:34

Yes, it's thrilling. I also tried this:

Code: Select all

"Execute MediaInfo"
  parseFolder(<curpath>);
  parseFolder(<curpath>);
  parseFolder(<curpath>);
  parseFolder(<curpath>);
and it gets slower and slower....

admin
Site Admin
Posts: 48013
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Scripting Memory-Leak / Speed Degrading

Post by admin » 15 Sep 2019 11:43

Got it. The user functions were loaded and added to the functions pool again and again for each scanned file in the file list (= for each load 'xy-slowdown2.xys', '_callFunction';). Fixed by checking if the function has already been loaded before. Numbers look better now:

Code: Select all

31 31 31 31 31 32 32 31 31 31 31 31 31 32 32 31 31 31 31 31 31 32 32 31 31 31 31 31 31 32 32 31 31 31 31 31 31 32 32 31 31 31 31 31 31 32 32 31 31 31 31 31 31 32 32 31 31 31 31 31 31 32 32 31 31 31 31 31 31 32 32 31 31 31 31 31 31 32 32 31 
Good bug! Thanks for the good analysis and detailed description! :beer: :tup:

fishgod
Posts: 199
Joined: 03 Feb 2008 00:40
Location: Sankt Augustin (near Bonn), Germany

Re: Scripting Memory-Leak / Speed Degrading

Post by fishgod » 15 Sep 2019 14:21

Thank you for fixing it.
That raises a new question that come in my mind in the last days:
Does redefining a function with the same name is:
  • ignored
  • overwrites the existing one
  • raises a warning/error (obviously not till now, maybe this would be a good thing)
Of cause, that info should be added to the help-file for future reference.
Operating System: Win7 x64 / almost allways newest XY-beta
totally XYscripting-addicted

admin
Site Admin
Posts: 48013
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Scripting Memory-Leak / Speed Degrading

Post by admin » 15 Sep 2019 14:27

ignored.

It is assumed that a same-named function from a same-named resource is identical. IOW, it is assumed that the resource file is not changed during the script execution.

And this is also the old behavior. All those added functions were just ignored.

fishgod
Posts: 199
Joined: 03 Feb 2008 00:40
Location: Sankt Augustin (near Bonn), Germany

Re: Scripting Memory-Leak / Speed Degrading

Post by fishgod » 15 Sep 2019 20:51

admin wrote:
15 Sep 2019 14:27
It is assumed that a same-named function from a same-named resource is identical.
Very reasonably. Your formulation indicate, that this might be different for different files. In a rational design nobody would do this on purpose, but it might happen by accident when multiple scripts are combined.
Would it be reasonable to raise a warning, when a function redefinition from a location different that first definition comes along?
Operating System: Win7 x64 / almost allways newest XY-beta
totally XYscripting-addicted

admin
Site Admin
Posts: 48013
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Scripting Memory-Leak / Speed Degrading

Post by admin » 16 Sep 2019 10:38

fishgod wrote:
15 Sep 2019 20:51
Would it be reasonable to raise a warning, when a function redefinition from a location different that first definition comes along?
Yes, good idea. It's a crass design error by the scriptor that should not be silently handled in whatever way.

admin
Site Admin
Posts: 48013
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Scripting Memory-Leak / Speed Degrading

Post by admin » 16 Sep 2019 11:27

admin wrote:
16 Sep 2019 10:38
fishgod wrote:
15 Sep 2019 20:51
Would it be reasonable to raise a warning, when a function redefinition from a location different that first definition comes along?
Yes, good idea. It's a crass design error by the scriptor that should not be silently handled in whatever way.
I could do it (raise a warning and terminate the script) for same-named functions loaded (SC load) from different resources. But not when they are included from different resources. Included stuff is integrated in the resource which includes it, so in that case the duplicate function is just ignored without a warning.

fishgod
Posts: 199
Joined: 03 Feb 2008 00:40
Location: Sankt Augustin (near Bonn), Germany

Re: Scripting Memory-Leak / Speed Degrading

Post by fishgod » 16 Sep 2019 20:35

There is at least one issue with the new warning (refering to the discussion going on here: viewtopic.php?f=2&t=20928&p=172636)

Setup a custom column with a script like this:

Code: Select all

include_once 'lib.inc.xys';

return f1();
The lib.inc.xys-file:

Code: Select all

function f1() {
  return 'f1';
}
Run the following script-file:test.xys (e.g. by passing ::load 'test.xys' in the address-bar)
Suprisingly copying this code into Scripting > Run Script does not trigger the warning :veryconfused:

Code: Select all

include_once "lib.inc.xys";

"Test"
  f1();
  #1001; //refresh
The Column as such is working as expected, but when the list gets refreshed by the #1001; it looks like the column script is run in the context of test.xys-file, complaining about the allready defined function.
The Warning pops multiple times (the same amount as files are in the list):
xy-function-warning.png
xy-function-warning.png (3.94 KiB) Viewed 305 times
It would really help debugging when there is some kind of stack-trace/include-tree in the warning to locate the redefinition and source-definition. (took me some time to figure out that the warning was raised from a column-script)


There may be a really simple solution to decide when to raise the warning that covers all include/load scenarios:
Compare the function-definition and silently ignore later definitions if they are equal, as honoring them would make no difference either.
Operating System: Win7 x64 / almost allways newest XY-beta
totally XYscripting-addicted

admin
Site Admin
Posts: 48013
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Scripting Memory-Leak / Speed Degrading

Post by admin » 17 Sep 2019 13:38

Fixed in next beta.

Your ideas are good, but not that easy to do. Maybe later.

Filehero
Posts: 2299
Joined: 27 Feb 2012 18:50
Location: Windows 10 Pro x64

Re: Scripting Memory-Leak / Speed Degrading

Post by Filehero » 18 Sep 2019 20:16

I have isolated my issue which still occurs with 20.40.008.

Put this into <xyscripts>:
testLib.xys
(95 Bytes) Downloaded 7 times
Put the following code into <xyscripts> as whatever.xys and run it.

Code: Select all

include_once "testLib.xys";

// call lib function
  callTest();

// build in-the-fly script code
  $mscript = <<<SCRIPT
include_once "testLib.xys";
SCRIPT;

  // NOW THE ERROR GETS THROWN HERE - this logic has been working for years
  load($mscript, , "s");

Once the script code build is loaded the error is thrown just by including a function set included once once again. To me the new check is still colliding with include_once under at least one more condition.

I'm still confused since my issue here is that it breaks a lot of code which was running perfectly fine for years over here.
Last edited by Filehero on 18 Sep 2019 21:02, edited 1 time in total.

admin
Site Admin
Posts: 48013
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Scripting Memory-Leak / Speed Degrading

Post by admin » 18 Sep 2019 20:50

OK, I'll upload a fix.

Filehero
Posts: 2299
Joined: 27 Feb 2012 18:50
Location: Windows 10 Pro x64

Re: Scripting Memory-Leak / Speed Degrading

Post by Filehero » 18 Sep 2019 21:35

Hi Don,

first of all, thanks a lot! The issue with my example is fixed. :tup:

But there's always a "but". That's why I got a "bad feeling" yesterday once I have gone through this thread. And that's also why I had to open a bottle Doppio Passo Salento Primitivo right now. :mrgreen:

After I saw my example was working without any error I ran my menu code again - and :blackstorm:: my error is still there!

Hmmmm. Checking my "production" code, shifting code around. Bamm. :idea:

Indenting the include_once statement in the Heredoc script ist the one to blame. To make the test code better I've included the critical call statement.

Code: Select all

include_once "testLib.xys";

// call lib function
  callTest();

// build in-the-fly script code
  $mscript = <<<SCRIPT
  include_once "testLib.xys"; // <-- INDENTING THIS STATEMENT EVOKES ERROR AGAIN -->
  
  callTest(); // <-- THIS IS WHAT SHOULD WORK AFTER ALL -->
SCRIPT;

  // NOW THE ERROR GETS THROWN HERE - this logic has been working for years
  load($mscript, , "s");
Two "Viertel" later I'm happy for this issue because it made re-realize how much I like Doppio Passo Salento Primitivo. :tup: :tup: :beer: :whistle:

Thanks for your patience,
Filehero

admin
Site Admin
Posts: 48013
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Scripting Memory-Leak / Speed Degrading

Post by admin » 19 Sep 2019 12:02

OK, in case you still can read this: Should be fixed in next beta. Prost! :beer:

Filehero
Posts: 2299
Joined: 27 Feb 2012 18:50
Location: Windows 10 Pro x64

Re: Scripting Memory-Leak / Speed Degrading

Post by Filehero » 19 Sep 2019 18:48

Yes, still can read. :wink:

Thanks for polishing. :-)

This drives me nuts. With my regular scripts, the problem is still there.
With my example script the error again has been cured by your fix. However, the loaded script doesn't seem to call the referenced function. So there some more quirks to be uncovered. :blackstorm:

I'll continue my analysis tomorrow (hopefully).

Citing Arnie: I'll be back! :wink:

Post Reply