increase general performance on scripting please!

[tiago]

Dealing with several hundred thousands of documents is becoming more and more common as the nature of my work is changing as well as HDD capacities. Some tasks are better performed by specialized tools but the few available are not flexible enough and this is a good enough reason why I am more and more into XYplorer. But I see a tiny huge room for improvement in these areas:

a. general performance: XY does not uses full system's power, CPU busy light barely blinks thus pretty simple tasks like renaming several hundreds of files and folders take a considerable amount of time. Other example is right now I am scanning a text source of only 11 MB and this has already taken 5 hours of processing, by the regular measurements it will last another 15-16 hours until I got the respective writefile() output. That's the only task running on a dual core machine.

b. unresponsiveness while executing a script: as I work for people I have to continuously feed them with details on current operations, predictions and reports. Sometimes I can have a spare PC so I am sure no one will mess with it and I process tasks on/from there when possible. But most of the times I have to rely on someone else's machine and often s/he needs to bring a window to foreground, screwing XY window thus leaving me with no statusbar messages I script to pop info on current operation's flow, and XY goes for good as it only becomes responsible once the whole task is finished. Not good.

[nas8e9]

This is probably because XYplorer is single-threaded. This was a huge problem with long-running file operations on Vista and later, which was solved by background processing; doing the same for script execution sounds very difficult.

By way of a workaround, would starting a second instance of XYplorer with a script to be executed as the parameter, work for you? This way, the second instance of XYplorer runs as a separate process, leaving your main XYplorer window/process free for other tasks. It wouldn't increase the speed of execution, of course.

As for Task Manager showing low CPU utilisation, I'm guessing your CPU has more than one core. Both Task Manager as well as Process Explorer, show a consolidated CPU utilisation, which in the case of a dual-core CPU, means that a single-threaded process by definition will never be shown as using more than 50%. Four and six cores skew this even further.

[tiago]

This is probably because XYplorer is single-threaded. This was a huge problem with long-running file operations on Vista and later, which was solved by background processing; doing the same for script execution sounds very difficult.

I am afraid of that...

[nas8e9]

The only two workarounds seem to be:

1. Run scripts in a separate instance:

Code: Select all

XYplorer.exe /script=test

This scripting instance won't be usable or probably able to convey its scripting status while the script is running, but it will leave you with a usable XYplorer instance.

On reflection, it doesn't matter whether you use a separate instance or whether Don would farm out the scripting to a separate process: it still would only be single-threaded either way.

2. Specifically for the text-crunching, you mention that you've tried specialised tools but haven't found something flexible enough. I wonder whether EmEditor might be useful to you: it is a fully scriptable and multi-threaded text editor that generally performs well, especially with large files?

Like XYplorer, EmEditor can run a macro file on startup through a command line argument; any parameters for the macro can be furnished by XYplorer by having writefile create a temporary file. Finally, by running EmEditor synchronously from XYplorer (with the /wait=1-parameter) and by having the EmEditor macro close EmEditor on completion, EmEditor can be used as an intermediary step in a script.

[tiago]

any quick hint on how to emulate under EmEditor the gettoken on a foreach loop?

I need this

123 string3 abc
12 string2 abc
1 string1 abc

to look like

string3
string2
string1

but I'm yet to find a tool able to get everything between those double spaces, in that sample.

Running a parallel instance of XY is not a problem, the time-consuming is the real PITA here.

Totally agree on the single thread part: what left me bugged is that the CPU is peeking 99% all of the time even on a single core but the busy light barely blinks within a whole hour. I really can't say what's going on, but apps that turn on the red lights like a flying saucer do a better performance show for sure...

Still, I'd prefer XY doing it better than relying on external tools as it should be just a set-and-forget task that should be accomplished few minutes later, not whole hours or days in some cases.

[nas8e9]

any quick hint on how to emulate under EmEditor the gettoken on a foreach loop?

I need this

123 string3 abc
12 string2 abc
1 string1 abc

to look like

string3
string2
string1

but I'm yet to find a tool able to get everything between those double spaces, in that sample.

I'm not much of a scripter, but going by the EmEditor help file you could use the functionality of the Windows Scripting Host (which EmEditor uses as its automation engine), which includes RegEx support. You can also use EmEditor's native scripting commands for the tricky bit, the double spaces, which can easily be replaced as EmEditor supports find and replace, allowing you to substitute the double spaces for an easier to digest separator.

Should you need any RegEx help, Stefan is the go-to guy on this forum.

[admin]

Tip: The key to speed is clever coding. Few things on a modern computer are well-coded AND take hours to complete.

[Jibz]

I need this

123 string3 abc
12 string2 abc
1 string1 abc

to look like

string3
string2
string1

If you are processing text in any way, I would suggest using a tool that was designed to do that, for instance (assuming all lines will be of the format something, double space, something, double space, something):

Code: Select all

perl -pe "s/.*  (.*)  .*/$1/" <inputfile.txt >outputfile.txt

will handle megabytes in seconds.

I am sure it would be just as easy to write a sed one-liner for it, but I am used to turning to perl when I need to munch text .

[tiago]

Tip: The key to speed is clever coding. Few things on a modern computer are well-coded AND take hours to complete.

Exactly. :razz:
To be specific my code is a simple "read this, get and store all tokens with index 2, write output", beside the obvious timing code I include on all operations as almost everything I must do usually take hours to go and that's why I came up with the proposition.

In a closer look I have found what nas8e9 meant by scriptability under EmEditor. Looks promising.

But Perl may do what I need with the urgency I need (assuming Jibz code is the perfect, ready to go answer). I'll feedback as soon as I get something.

The OT still remains.

Thanks for the precious tips, guys.

[Marco]

Another powerful editor is EditPad Pro (http://editpadpro.com/download.html). Assuming you're working with Windows 1252 codepage, just run in sequence these two regex-based search-and-replace:

Code: Select all

^.+?\x20

to

Code: Select all

and then

Code: Select all

\x20.+?$

to

Code: Select all

Basically first strips out everything before the first space (space included), and then everything after the second space (space included), with space being 0x20 (I think that's the way of writing characters in hex, not sure 'tho). Tested and working.
You'll crunch megabytes in seconds!

[tiago]

Job done with Marco's help.
I hate the fact that I'll have to kill that XY instance and the time spent so far, but one day a man has to learn...
Thank you, Marco! I just had to use "^.+?\x20\x20" and "\x20\x20.+?$" instead of what you wrote, you'll understand why. A huge thank you!

The OT still remains! That was just another reason... (plus the power failures which obligated me to start all over again thus far)

[tiago]

And it took barely 2 minutes to unique (remove duplicates leaving uniques only) the same document as specified in this thread: http://www.xyplorer.com/xyfc/viewtopic.php?f=7&t=7085.

Does anybody knows any way to outsource such tasks from XYplorer to such apps via command line and sand back the output to XY do its part when applicable? Clipboard should do as I can script that XY should wait X seconds (oh, gosh...) then read clipboard, presumably written by such 3rd party tool.

[Marco]

Job done with Marco's help.
I hate the fact that I'll have to kill that XY instance and the time spent so far, but one day a man has to learn...
Thank you, Marco! I just had to use "^.+?\x20\x20" and "\x20\x20.+?$" instead of what you wrote, you'll understand why. A huge thank you!

You're welcome!

The OT still remains! That was just another reason... (plus the power failures which obligated me to start all over again thus far)

Well, all the optimizations done to web browser js engines come to my mind. If it's taking years and several teams of coders to push the performance bar higher and higher maybe it's not a simple task. So perhaps that's why a general purpose scripting engine managed by just one (great) developer isn't fast enough for your needs.
But I'd like to know your system specs and how exactly you're processing this 11 MB (pure?) text file. Sound strange to me such a simple task can take 20+ hours... (although I've never used XY scripting capabilities)

[Jibz]

I would still say it's worth having a look at perl (or one of the other similar languages) if you do a lot of text processing on large files. You can do a lot of things with text editors, regular expression search and replace, and macro languages, but at some point you will need to do something that is beyond their scope, or takes forever .

The reason you are not seing any CPU usage on renaming a bunch of files is that the operation is bound by the harddisk I/O and requires no CPU power at all. Basically, XY is waiting for the operating system, which is waiting on the harddisk to update the name.

The unresponsiveness, as others have pointed out, is due to XY being single-threaded. An unfortunate consequence of it being written in a language that is over a decade old .

Would be interesting to see the XY script you were using?

[tiago]

Code: Select all

//after saving as .txt from Opera web browser

   $refdt = now("hh:nn:ss");
 $marker = "http://www.some.com/sourcedoc.aspx?name=%26quot%3B%3B // useful for further search and replacements
 $cur = readfile("<curitem>");
  $nm = (<curbase>);
 $exp = "";
 foreach($tk, $cur, "<crlf>") {
 $sel = gettoken($tk, 2, "  ");
 IF ($sel == "") {   }
 ELSEIF ($sel != "") { $exp = "$exp" . "$marker" . "$sel" . "<crlf>"; }
                                  }

   $refdate = now("hh:nn:ss");
 $name=%26quot%3B%24nm%26quot%3B . " - use.txt";
 writefile ("$name", "$exp", o);

   writefile("Timing - Some DOT com.txt", "Finally done! Report started: $refdt. Task finished: $refdate.", r);
   echo "Finally done! Report started: $refdt. Finished: $refdate.";
   status "ok";

I won't touch those machines for the full specs as I will let them finish their respective tasks but as a minimum reference we're talking about AMD Sempron 2600+, 1,5 GB/ram and an Intel T4400, 3 GB/ram, among others. Yes, pure text (.txt).

Of course I will take some Perl lessons as I was impressed by the statement that a single line could do everything I need in an infinitesimal fraction of our precious times. Just have to find... the time. :razz:

Thanks for the clarifications, Jibz. Makes sense when it comes to renaming, but I'm still intrigued as the behavior is the same regarding .txt search and replacement. EditPad was that fast but no busy light indication either, though, despite my previous observations on other apps' processing.

Edit: this is a modified version of the script as the original doesn't include the search and replacement of "http://www.some.com/source..." for performance reasons.
It would be later done by VIM/EditPad.