File segregation script

[nerdweed]

I am having some files with the following naming convention

[basevalue]-[widthxheight]-[xxx].jpg

Here, the basevalue can have hyphens as well and xxx is a three digit suffix.

What I am trying to do is
1. Find out all images with different resolutions
2. Delete all but except one resolution - which would be decided from the available options.

What would be the quickest way through loop with this. Any pointers please

[highend]

And the output for 1. should look like?

E.g.:

Code: Select all

    $selectedFiles = get("SelectedItemsPathNames", "<crlf>");
    $resolutions = formatlist(replace(regexmatches($selectedFiles, "-\d+x\d+-"), "-"), "dents");

    // List files with different resolutions
    $output = "";
    foreach($resolution, $resolutions) {
        $matchingFiles = regexmatches($selectedFiles, "^.*?$resolution.*$", "<crlf>");
        $output = $output . $resolution . ":<crlf>" . $matchingFiles . "<crlf 2>";
        }
    }

    text $output;

    // Remove all files except the ones for the selected resolution
    $resolutionToKeep = inputselect("Choose resolution to keep", $resolutions, "|", 4);
    $filesToRemove = formatlist($selectedFiles, "f", "<crlf>", "!*$resolutionToKeep*");
    text $filesToRemove;

    delete 1, 1, replace($filesToRemove, "<crlf>", "|");

[nerdweed]

Thanks Highend. I have written this below script. Test run looked OK. But it is very time consuming.

Code: Select all

"BestRes"
	$items = ListPane(a, "*.jpg", 1, "|") ;
	$resolutions = RegExReplace ($items, ".*?-(\d+x\d+)-\d+.jpg", "$1|") ;
	$resolutions = FormatList($resolutions, sed, "|") ;
	$res = InputSelect("Choose preferred resolution", $resolutions, "|", 18) ;
	$input= "" ;
	$output = "" ;
		ForEach ($curRes, $res, "|") {
		$data = FormatList($items,f,"|","*-$curRes-*.jpg") ; 
		$data = RegExMatches ($data, ".*?-$curRes-\d+.jpg","|") ;
		if ($data) {
			$data = RegExReplace ($data, "(.*?)-$curRes-(\d+.jpg)", "$1-*-$2") ;
			$data = FormatList($data, sed, "|") ;
			ForEach ($img, $data, "|") {
				$candidate = FormatList($items, f, "|", "$img") ; //All similar files
				$items = FormatList($items,f, "|", "!$img") ; //Reduce processed and ommited files
				$input = $input . "|". FormatList($candidate, f, "|", "*-$curRes-*") ; 
				$candidate = FormatList($candidate, f, "|", "!*-$curRes-*") ; //Candidates for deletion
				$output = $output . "|" . $candidate ;
				$count = GetToken ($items, "Count", "|") ;
				status "$count files to be processed", , progress ;
				wait 1 ;
			}
		}
		status "$curRes completed", , progress ;
		wait 1000 ;
	}
	PaperFolder ("BestYouHave", "$input", "|" , n) ;
	PaperFolder ("Dupes", "$output", "|" , nl) ;

Can this be further optimised.

[highend]

What exactly (apart from the part that processes only selected files instead of all .jpg in the current folder (which is just one line)) is wrong with my script?

I need 2 regexmatches & formatlists and one loop.

Your's 1 regexmatches, 7 formatlists, 2 regexreplaces and two nested loops.

[nerdweed]

Sorry, nothing wrong with your script - . Just that the number of files to check is huge - approx 30000 - so I thought reducing it might help.

P.S. I didn't see your edited post. I will try that as well and it certainly looks much better. Thanks again

[highend]

This is a bit more tuned for speed. Should be able to process 30.000 files within a few seconds.

Tested with 30k files in a ram disk: 500ms (excluding time for text / inputselect commands).

The only important change: Got rid of replace() commands. They are incredible slow on "large"
content. regexreplace is a few hundred times faster...

It's working fine even with files > 100.000...

Code: Select all

    $selectedFiles = listpane(, "*.jpg", 1, "<crlf>");
    $resolutions = formatlist(regexreplace($selectedFiles, "^.*?-(\d+x\d+)-.*?(\r?\n|$)", "$1|"), "dents");

    // List files with different resolutions
    $output = "";
    foreach($resolution, $resolutions) {
        $matchingFiles = regexmatches($selectedFiles, "^.*?$resolution.*$", "<crlf>");
        $output = $output . $resolution . ":<crlf>" . $matchingFiles . "<crlf 2>";
        }
    }

    text $output;

    // Remove all files except the ones for the selected resolution
    $resolutionToKeep = inputselect("Choose resolution to keep", $resolutions, "|", 4);
    $filesToRemove = formatlist($selectedFiles, "f", "<crlf>", "!*$resolutionToKeep*");
    text $filesToRemove;

    delete 1, 1, replace($filesToRemove, "<crlf>", "|");

[nerdweed]

I guess I wasn't very clear with

2. Delete all but except one resolution - which would be decided from the available options.

I understand the script you provided - deletes all images except the ones with the selected resolution. That's not what I intend to do. I would have gone with a non-scripted approach here. Set a !VF and delete it

I will try to explain it in a bit more detail

There are around 9 image resolutions - but not each of them has all the 9 resolutions. If an image is present with Resolution 1 - 1920x1080, I delete the remaining resolutions. If an image isn't present in resolution 1, I check if it is present in Resolution 2 - 1920x1200 and so on. i.e.; atleast one resolution is available for all files but it need not necessarily be the same (and it won't be - considering how the resolutions kept on changin 640x480, 800x600,1024x768,1280x800,1376x768,1440x900,1600x900,1600x1200,1920x1080,1920x1200). I am trying to keep only one resolution.

That was one of the reasons, I had to go with two foreach loops.

However; the script I wrote crawls. I expect it would run for atleast 4 hours.

This is what I want to achieve.

image-1920x1080-001.jpg //Keep
image-1920x1080-002.jpg //Keep
image-1920x1080-004.jpg //Keep
image-1920x1200-001.jpg //Delete
image-1920x1200-003.jpg //Keep as similar image not present in 1920x1080
image-1920x1200-004.jpg //Delete
image-1600x900-001.jpg //Delete
image-1600x900-002.jpg //Delete
image-1600x900-005.jpg //Keep as similar image not present in 1920x1080 and 1920x1200
image-1440x900-001.jpg //Delete
image-1440x900-002.jpg //Delete
image-1440x900-006.jpg //Keep as similar image not present in better resolution
image-1024x768-001.jpg //Delete
image-1024x768-002.jpg //Delete
image-1024x768-007.jpg //Keep as similar image not present in better resolution
image-800x600-001.jpg //Delete
image-800x600-002.jpg //Delete
image-800x600-007.jpg //Delete

[highend]

If I execute your script and just use the example file names from your last post then I get this:

$input:

R:\1\image-1920x1080-001.jpg
R:\1\image-1920x1080-002.jpg
R:\1\image-1920x1080-004.jpg

$output (manually sorted for a better overview):

R:\1\image-1920x1200-001.jpg
R:\1\image-1920x1200-004.jpg
R:\1\image-1600x900-001.jpg
R:\1\image-1600x900-002.jpg
R:\1\image-1440x900-001.jpg
R:\1\image-1440x900-002.jpg
R:\1\image-1024x768-001.jpg
R:\1\image-1024x768-002.jpg
R:\1\image-800x600-001.jpg
R:\1\image-800x600-002.jpg

Which is wrong.
From your example, these additional files should have been in the $input variable as well:

image-1920x1200-003.jpg //Keep as similar image not present in 1920x1080
image-1600x900-005.jpg //Keep as similar image not present in 1920x1080 and 1920x1200
image-1440x900-006.jpg //Keep as similar image not present in better resolution
image-1024x768-007.jpg //Keep as similar image not present in better resolution

And for the $output variable,

image-800x600-007.jpg //Delete

is missing...

Wouldn't it be easier to leave the selection of the "preferred resolution" out and just keep all files that:
1. Exist only once so they should be kept anyway (regardless of resolution)
or
2. Are taken only from the highest resolution and if they are present in a lower resolution, ignore / delete them

At least this would preserve the best resolution for all pictures and it's rather painless to write (and fast as well)

E.g.:

Code: Select all

    $selectedFiles = listpane(, "*.jpg", 1, "<crlf>");

    // Add the file's resolution as a TEMPORARY trailing parameter to allow sorting from high to low and remove it afterwards!
    $sortedFiles = regexreplace(formatlist(regexreplace($selectedFiles, "^(.*?-)(\d+x\d+)-(\d+)(\.jpg$)", "$2|$1$2-$3$4"), "dentr", "<crlf>"), "^.*?\|");

    // Create a list for only the first match for each unique file id (3 digit before the extension)
    $uniqueIDs   = "|";
    $filesToKeep = "";
    foreach($file, $sortedFiles, "<crlf>") {
        $uniqueID = regexreplace($file, "^(.*?-\d+x\d+-)(\d{3})(.*?$)", "$2");
        if (strpos($uniqueIDs, "|$uniqueID|") != -1) { // A file with a unique ID was already processed, skip this one
            continue;
        }
        $filesToKeep = $filesToKeep . gettoken(regexmatches($sortedFiles, "^.*?$uniqueID\.jpg$", "<crlf>"), 1, "<crlf>") . "<crlf>";
        $uniqueIDs = $uniqueIDs . $uniqueID . "|"; // Add a process file to the list of unique IDs
    }
    text $filesToKeep;

[highend]

Here is a variant that will do exactly what you posted in your example.

It let's you choose only one resolution to keep (not multiple ones).
Although this would only need an additional (not nested!) loop that shuffles the matching files on top of the $sortedFiles list (while removing them from the rest of the list).

It uses a few regex- matches/replaces but these commands are blazing fast (in comparison to a loop).
I've used it for 30k files in a ram disk and it took about 16 seconds (i5-3570k @3,4GHz).

You only need to add the deletion part...

Code: Select all

    $selectedFiles = listpane(, "*.jpg", 1, "<crlf>");
    $resolutions = formatlist(regexreplace($selectedFiles, "^.*?-(\d+x\d+)-.*?(\r?\n|$)", "$1|"), "dents");
    $resolutionToKeep = inputselect("Choose resolution to keep", $resolutions, "|", 4);


    // Add the file's resolution as a TEMPORARY trailing parameter to allow sorting from high to low and remove it afterwards!
    $sortedFiles = regexreplace(formatlist(regexreplace($selectedFiles, "^(.*?-)(\d+x\d+)-(\d+)(\.jpg$)", "$2|$1$2-$3$4"), "dentr", "<crlf>"), "^.*?\|");

    // Get all files for the resolution that should be kept
    $filesOnTop = regexmatches($sortedFiles, "^.*?$resolutionToKeep.*?$", "<crlf>");

    // Remove the files ($filesOnTop)
    $sortedFiles = regexreplace($sortedFiles, "^.*?$resolutionToKeep.*?\r?\n");

    // Rebuild the list of sorted files
    $sortedFiles = $filesOnTop . "<crlf>" . $sortedFiles;

    // Create a list for only the first match for each unique file id (3 digit before the extension)
    $uniqueIDs   = "|";
    $filesToKeep = "";
    foreach($file, $sortedFiles, "<crlf>") {
        $uniqueID = regexreplace($file, "^(.*?-\d+x\d+-)(\d{3})(.*?$)", "$2");
        if (strpos($uniqueIDs, "|$uniqueID|") != -1) { // A file with a unique ID was already processed, skip this one
            continue;
        }
        $filesToKeep = $filesToKeep . gettoken(regexmatches($sortedFiles, "^.*?$uniqueID\.jpg$", "<crlf>"), 1, "<crlf>") . "<crlf>";
        $uniqueIDs = $uniqueIDs . $uniqueID . "|"; // Add a process file to the list of unique IDs
    }
    text $filesToKeep;

[nerdweed]

First, thanks

Second, not sure. Why my script didn't return correct values for you. It gave me the correct results.

I had made some modifications to the script last night and that may have caused this. (updated the earlier post with the sluggish script)

Both the scripts provided by you work well and second works really well. However; the basenames can also differ . Sorry for not covering that earlier. I tried modifying your scripts, but no joy.

2014-09-03_194121.png

Also, I couldn't capture much about what that extra loop would do please.

[highend]

This should work for different basenames + 3digit-ids files...

Instead of creating a single criteria (the 3digit part), you have to create a combination of the basename + id.

The $baseNamePattern is necessary to use it for the $filesToKeep because the basename can contain characters that need to be escaped inside a regex (otherwise it won't match in such cases).

Code: Select all

    $crlf = "<crlf>";
    $selectedFiles = listpane(, "*.jpg", 1, $crlf);
    $resolutions = formatlist(regexreplace($selectedFiles, "^.*?-(\d+x\d+)-.*?(\r?\n|$)", "$1|"), "dents");
    $resolutionToKeep = inputselect("Choose resolution to keep", $resolutions, "|", 4);

    // Add the file's resolution as a TEMPORARY trailing parameter to allow sorting from high to low and remove it afterwards!
    $sortedFiles = regexreplace(formatlist(regexreplace($selectedFiles, "^(.*?-)(\d+x\d+)-(\d+)(\.jpg$)", "$2|$1$2-$3$4"), "dentr", $crlf), "^.*?\|");

    // Get all files for the resolution that should be kept
    $filesOnTop = regexmatches($sortedFiles, "^.*?$resolutionToKeep.*?$", $crlf);

    // Remove the files ($filesOnTop)
    $sortedFiles = regexreplace($sortedFiles, "^.*?$resolutionToKeep.*?\r?\n");

    // Rebuild the list of sorted files
    $sortedFiles = $filesOnTop . $crlf . $sortedFiles;

    // Create a list for only the first match for each unique file id (3 digit before the extension)
    $uniqueTokens = "|";
    $filesToKeep  = "";
    foreach($file, $sortedFiles, $crlf) {
        $baseName = regexreplace($file, "^(.*\\)(.*?)-(\d+x\d+-)(\d{3})(.*?$)", "$2");
        $uniqueID = regexreplace($file, "^(.*?-\d+x\d+-)(\d{3})(.*?$)", "$2");
        if (strpos($uniqueTokens, "|$baseName:$uniqueID|") != -1) { // A file with a unique token was already processed, skip this one
            continue;
        }
        $baseNamePattern = regexreplace($baseName, "(\\|\*|\^|\$|\.|\+|\(|\)|\[|\{)", "\$1");
        $filesToKeep = $filesToKeep . gettoken(regexmatches($sortedFiles, "^.*?$baseNamePattern-\d+x\d+-$uniqueID\.jpg$", $crlf), 1, $crlf) . $crlf;
        $uniqueTokens = $uniqueTokens . "$baseName:$uniqueID|"; // Add a processed file to the list of unique tokens
    }
    text $filesToKeep;

I couldn't capture much about what that extra loop would

It would shuffle additional resolutions (file names that contain one of them) to the top of the $sortedFiles variable.

Ofc the new version takes more time than the last one...

[nerdweed]

Thanks. This seems to work as required, albiet this is not as fast as your machine - it may run for around 30 mins. I have injected a minor delay of 1 msec in the for loop though. I am running for a 5400 rpm though, but since the files are not touched after reading the names, I think that shouldn't matter

I do know why you use this, but it doesn't work for me.

$baseNamePattern = regexreplace($baseName, "(\\|\*|\^|\$|\.|\+|\(|\)|\[|\{)", "\$1");

I only have to escape the backslashes. I know the below looks stupid but I have to do it and it didn't work previously because I hadn't added it.

$baseNamePattern = regexreplace($baseName, "\\", "\\");

Also, could you please explain what does this in third line mean

(\r?\n|$)

Does it mean either \r\n or \n for line feeds or line end?

All your scripts here have been really helpful.

[highend]

but it doesn't work for me

This is no error message. What exactly isn't working?

I only have to escape the backslashes

You do not have to. That line is a generic pattern to replace all metachars. The base name doesn't even contain backslashes^^

You will notice that it's needed if you e.g. have a (base)name that contains any of the chars in that list.

Does it mean either \r\n or \n for line feeds or line end?

Yeah. The ? after a char makes it optional so your assumption is true.

[nerdweed]

Correct, I had tried in just another variant of your script.

As such, it didn't give any error, but not the expected results. The final file was empty.

BTW, the script finished in 20 mins (we can say 18 mins with around 2 mins of injected delays)