PNGout script with 4 CPU cores

[Sander Bouwhuis]

Hi all, I'm in need of some advise.

1.
I want to call an external application with parameters for all the selected items.
I came up with this, but it doesn't work:

Code: Select all

"PNG out"
 $selectedfiles = get("SelectedItemsNames",|);
 foreach($file,$selectedfiles,|)
 {
  run '"C:\Utilities\XnView\AddOn\PngOut.exe" /y ' . "$file"
 }

Any file with a space in it doesn't work.

So, I changed it to this, which does work:

Code: Select all

"PNG out"
 $selectedfiles = get("SelectedItemsNames",|);
 foreach($file,$selectedfiles,|)
 {
  run '"C:\Utilities\XnView\AddOn\PngOut.exe" /y "' . $file . '"'
 }

Is this really the way to quote file names?


2.
Because I have a CPU with 4 cores, I would like to launch 4 PNGout processes, and then wait until they are done and then continue with the next 4. Ofcourse, if there is a way to have 4 in flight that would be even better.


I would really appreciate some help.

[Stefan]

1.) yes, this is one way. An other way is to use quote($file);

2.) lookup the run() command in the help. There is an wait-parameter.
run command, , [wait=1 Only continue when the shelled process has finished]

[Sander Bouwhuis]

Ok, I changed my script to this:

Code: Select all

"PNG out"
 $selectedfiles = get("SelectedItemsNames",|);
 foreach($file,$selectedfiles,|)
 {
  run '"C:\Utilities\XnView\AddOn\PngOut.exe" /y ' . quote($file), ,1;
 }

This only does 1 process at a time, not 4. How can I make it do 4?

[Stefan]

This only does 1 process at a time, not 4. How can I make it do 4?

I don't know right now really.

Perhaps something like this dummy code... but have no time to think about right now, sorry.

Code: Select all

$COUNT=0;
$File1="leer";
$File2="leer";
$File3="leer";
$File4="leer";

foreach($FILE
{
	$COUNT++;
	if ($File1=="leer"){$File1 = $FILE;}
	if ($File2=="leer"){$File2 = $FILE;}
	if ($File3=="leer"){$File3 = $FILE;}
	if ($File4=="leer"){$File4 = $FILE;}

	IF ($COUNT==4){
			if ($File1!=""){ run xyz $File1,, ,0;}
			if ($File2!=""){ run xyz $File2,, ,0;}
			if ($File3!=""){ run xyz $File3,, ,0;}
			if ($File4!=""){ run xyz $File4,,1,0;}
         if ($FILE==""){break;}
			$COUNT=0;
			$File1="leer";
			$File2="leer";
			$File3="leer";
			$File4="leer";
			$wait=0;  while($wait<10000){ $wait++;}
	}

}

[Sander Bouwhuis]

This doesn't work because it waits for each individual run job, so uses 1 CPU core only.
Alternatively, if you remove the 'wait' parameter then there is no wait and it will do all the jobs at the same time.

I also had a similar idea where I did 3 jobs with the no-wait parameter, and the 4th job with the wait parameter. Then, I would hope that on average it would work out. Turns out, it doesn't at all when you have a directory with a couple of 1000 files you want to handle.

I even considered generating batch files, but couldn't get it to work.

[Stefan]

This doesn't work because it waits for each individual run job, so uses 1 CPU core only.

Ohh, of course

Alternatively, if you remove the 'wait' parameter then there is no wait and it will do all the jobs at the same time.

Isn't that what you wanted?
I think now your app have to be clever enough to use all available cores.

[Sander Bouwhuis]

Alternatively, if you remove the 'wait' parameter then there is no wait and it will do all the jobs at the same time.

Isn't that what you wanted?
I think now your app have to be clever enough to use all available cores.

No, I want it to do exactly 4 jobs at a time. If I have a directory with 1000 files and I run the script without the wait parameter, it will open 1000 cmd line dos boxes. Even though I have an Ivy Bridge with 16GB RAM, Windows will not be able to handle that and will lock up my computer.

[Stefan]

And what if only the $File4 have the wait parameter?
Then the script will start 4 instances and wait till the fourth is done. Then do the next four.


As addition you may want to add an pause after each fourth run

Code: Select all

  $wait=0;  while($wait<10000){ $wait++;}

- - -

it will open 1000 cmd line dos boxes.

You know the fourth run() parameter?

Code: Select all

run command, [directory], [wait=0], [show=1]
show
0: Hides the window.
1: [Default] Activates and displays the window.
2:
3:

[PeterH]

No, I want it to do exactly 4 jobs at a time.

The word exactly is the problem.

If you start 4 instances and wait for the 4th, it may be that this 4th is the fastest - then the next 4 would be run before (some of) 1-3 are ended.

OK: pausing before starting the next 4 tasks can reduce the probability of this. But the more you delay, the lesser the risk. But then more and more time is wasted on waiting...

So for a good way for the wanted action you would need the possibility for the administrating task to asynchronously wait for all 4 task-ends, to always restart a new task when 1 old terminates. I doubt this is possible with windows?
(This "always" was possible on mainframe, with so called ECBs = Event Control Blocks...)

(By the way: the XY-command "Wait 1000" waits for 1000ms without stressing the CPU.)

[Sander Bouwhuis]

Thanks for both your replies.

You know the fourth run() parameter?

You miss the point. It will still open a dos box. The only difference is that it will be minimized.
Just try and run a script or 'open' or 'open with' command in XYplorer on 1000 files in a directory
and you'll see that Windows will lock up.

If you start 4 instances and wait for the 4th, it may be that this 4th is the fastest - then the next 4 would be run before (some of) 1-3 are ended.

In fact, I have done just such a thing, but to be safer I do 3 jobs. So, 2 are asynchronous and 1 is synchronous.
Unfortunately, the tasks are ranging from 1 second up to about 45 seconds. So, there is still a risk of runaway task count, but for now this will have to do (unless someone knows how to fix this).

If I really can't get it to work at all, I think I'll just write a simple C/C++ program that does exactly what I want. It's not too difficult to make, but I hoped it could be done automatically. The advantage is that I could have a command-line parameter which states how many concurrent jobs you want so that it can be used for any amount of CPUs/cores.

[PeterH]

Back to your other idea: 4 .bat files running parallel. Each with a list of commands for 1/4 of all items to handle. Dependend from the distribution of long- and short-running conversions you would not know which would run how long- but at least the number of parallel processes would be limited...

What's you problem on doing this?

[Marco]

Back to your other idea: 4 .bat files running parallel. Each with a list of commands for 1/4 of all items to handle. Dependend from the distribution of long- and short-running conversions you would not know which would run how long- but at least the number of parallel processes would be limited...

What's you problem on doing this?

+1 for this suggestion
And remember to set specifically each batch file to use a different cpu core.

[PeterH]

And remember to set specifically each batch file to use a different cpu core.

Can you explain why?