XYplorer Beta Club

nas8e9 wrote:Apart from the complexity of driving XYplorer quickly(!) from an external process, that would be a huge amount of work. I'm not hopeful, especially because as I understand it, regexreplace seems to solve the performance problems with text crunching.

The question is if this issue couldn't appear in other situations where regexp will not save you?

For instance, what if a user wants to write a script that creates a list of all his image files with the EXIF date (the property command can extract the date).

How will he compile the list if both writing one line at a time and collection the lines in a string are as inefficient as indicated here?

tiago wrote:

nas8e9 wrote:Apart from the complexity of driving XYplorer quickly(!) from an external process, that would be a huge amount of work. I'm not hopeful, especially because as I understand it, regexreplace seems to solve the performance problems with text crunching.

Keep in mind that it's not about finding the right method, it's about pushing up the whole performance a step beyond as there are issues, some previously named, regexreplace won't solve. In my point of view this shouldn't be a matter of "hope", either.

My (irrelevant, I'm just a user) pessimism is informed by some knowledge about the developer's search for moving beyond VB 6, as well as the difficulties surrounding multi-threading and multi-process work. Furthermore, I've always understood the developer considers speed a feature; based on this, I don't think the current scripting is unoptimised.

I don't mean to wish away your difficulties with XYplorer's performance, but based on the above, I'd be surprised if there were quick and easy fixes. Much more involved ones, if any and as always, will be weighed by the developer against other priorities.

tiago wrote:I have being working with XYplorer for a couple of months now on more critical tasks. I did everything on my side (apart regex, I admit) to get better results so when I have the option to work on offline, unprotected machines I do so. It is from that environment I'm talking about, iow, there's not much room to improve on my side (apart regex when applicable).

Good only in the sense that this can be excluded.

tiago wrote:Why I avoid regex as much as I can? I have quite limited understanding on it, and as far as it goes there are really few regex-skilled users lurking around, able to help. So little that often I come across a Cisco/other great company job announcement posted on regex forums. To exemplify it better, I'm still waiting for an answer to my pretty simple regex question above...

RegEx's reputation resembles that of Swiss cheese, I believe. Having said that, many if not all programming languages do offer it, so there must be advantages.

Well, I could improve the foreach loop speed dramatically (factor 4). But remember that this is a file manager, not a text muncher.

I tend to agree that a factor 4 in the loop logic is probably not worth the investment, unless it is something really easy to do. For munching text it would be far better to let the script call an external tool that specializes in that.

My remaining worry is, like stated above; are there situations where a user is likely to build a large string/list by continuously appending data (or continuously appending to a file), where the way these operations are implemented will cause a performance problem?

As an example, consider this test script: Basically, all it does is add a line of text to the end of a variable $limit times (think logging something, or building a list of information about files, etc.).

On my system, limit 1000 takes 0.6 seconds, limit 5000 takes 24 seconds, limit 10000 takes 2 minutes, and by extension limit 50000 should take around 50 minutes and limit 100000 3 hours.

Maybe it is unreasonable to think that anybody will ever write a script that creates a "list" of 50000 elements, I don't know?

Jibz wrote:I tend to agree that a factor 4 in the loop logic is probably not worth the investment, unless it is something really easy to do. For munching text it would be far better to let the script call an external tool that specializes in that.

When I said "I could improve" that was past tense: I did it.

Yes, correct: Appending to ever-growing strings sucks. The natural solution for this is arrays. Some day I will add them.

Is the time dependent from the while, or from $exp getting longer and longer?

And, by the way, could the (time for) processing of
$exp = "$exp" . "$foo";
be different from
$exp = $exp . $foo;

At least this should be correct for the coding time

As I see it: a string "abc" is a constant, *allowing* for variable substitution.
$exp is a variable. So it might have less overhead.

You were faster than my previous post...

admin wrote:

Jibz wrote:I tend to agree that a factor 4 in the loop logic is probably not worth the investment, unless it is something really easy to do. For munching text it would be far better to let the script call an external tool that specializes in that.

When I said "I could improve" that was past tense: I did it.

Yes, correct: Appending to ever-growing strings sucks. The natural solution for this is arrays. Some day I will add them.

Nice to hear you made it faster. (Knowing you I had expected this )

I do understand that repetitive appending to a string isn't easy to handle. But I don't know how arrays help for this situation? (If, in the end, you need the concatenated string, you must do it somewhen...)
And I don't understand the big non-linear effect, i.e. 5000 concats in 24 sec, 10000 in 120, and so on.

For this it could be worth for the scripter to concat maybe 1000 parts in a sub-variable, then concatenating this once to the main-string, and after resetting the sub-var concatenating the next 1000 parts, and so on...
(So avoiding such a "super-fragmented" string.)

PeterH wrote:Is the time dependent from the while, or from $exp getting longer and longer?

And, by the way, could the (time for) processing of
$exp = "$exp" . "$foo";
be different from
$exp = $exp . $foo;

The latter is better syntax and could be little faster (hardly notable).

String concatenation: For each adding the whole string must be allocated and copied in memory. However, adding 5000 items at once can be done very fast because only one allocation and byte copying is necessary.

What about a general increase in performance where loops aren't involved? In a situation where I want a bunch of S&R commands to be executed as quick as possible, since they are based on having a file selected, shaving off a few seconds would be a giant leap.

zer0 wrote:What about a general increase in performance where loops aren't involved? In a situation where I want a bunch of S&R commands to be executed as quick as possible, since they are based on having a file selected, shaving off a few seconds would be a giant leap.

Oh, good idea! I will simply make everything faster.

tiago wrote:Some extra info on performance: 84 k files took 8 hours 13 min to be merely moved (same partition, shorter paths) and renamed (shorter names) using "moveas" (it is faster than my previous concept, I insist!). FolderMatch took under 15 minutes to content compare them.

1. I'm guessing that for moveas, I should read moveto?

2. Are these 84,000 individual operations or are wildcards involved, meaning fewer calls to moveto?

3. Do you have background processing enabled for this script?
By way of background from the help file:

In scripting background processing may lead to undesired results when the script flow overtakes a backgrounded file operation before it is completed. To avoid this you can temporarily disable background processing by inserting this line to the beginning of such a script:
setting "BackgroundFileOps", 0;
Background processing will be automatically re-enabled when the script is finished.

admin wrote:

zer0 wrote:What about a general increase in performance where loops aren't involved? In a situation where I want a bunch of S&R commands to be executed as quick as possible, since they are based on having a file selected, shaving off a few seconds would be a giant leap.

Oh, good idea! I will simply make everything faster.

Not everything, but is there a possibility of having some S&Rs run sequentially and others in parallel?

zer0 wrote:

admin wrote:

zer0 wrote:What about a general increase in performance where loops aren't involved? In a situation where I want a bunch of S&R commands to be executed as quick as possible, since they are based on having a file selected, shaving off a few seconds would be a giant leap.

Oh, good idea! I will simply make everything faster.

Not everything, but is there a possibility of having some S&Rs run sequentially and others in parallel?

Of course not.

admin wrote:String concatenation: For each adding the whole string must be allocated and copied in memory. However, adding 5000 items at once can be done very fast because only one allocation and byte copying is necessary.

If it were just like you said I'd await that double number of items means double number of time, but not 4 times.
That is: about linear, not quadratic?

PeterH wrote:

admin wrote:String concatenation: For each adding the whole string must be allocated and copied in memory. However, adding 5000 items at once can be done very fast because only one allocation and byte copying is necessary.

If it were just like you said I'd await that double number of items means double number of time, but not 4 times.
That is: about linear, not quadratic?

Quick google search brings this among thousands of others:
http://axcs.blogspot.com/2006/07/asp-ti ... tring.html