Page 1 of 1
Clean out a list of dupes
Posted: 18 Apr 2017 15:35
by SkyFrontier
Is there a way I can dedupe a list of non-consecutive duplicated items, ie WITHOUT sorting it first (performance reasons), possibly using regex?
For this to perfectly work, I'd also need it to preserve a white list of exact terms (paths or phrases), ie, such terms will NOT be deduped at all.
How?
TIA, people.
Re: Clean out a list of dupes
Posted: 18 Apr 2017 16:06
by highend
The fastest way is formatlist() with format=Fdn and a filter.
This includes sorting (otherwise duplicates wouldn't be removed).
When using formatlist() you could filter with a negating expression containing your whitelist
and then add your whitelisted entries to the list again (again with the help of formatlist())
Code: Select all
$list = <<<>>>
you
people
me
you
me
people
other
>>>;
$whitelist = <<<>>>
you
me
>>>;
$deduplicated = formatlist($list, "Fdn", <crlf>, "!" . $whitelist);
$whitelisted = formatlist($list, "F", <crlf>, $whitelist);
text $deduplicated . <crlf> . $whitelisted;
All other methods will be <x> times slower (ofc depending on how many entries you have)
Re: Clean out a list of dupes
Posted: 19 Apr 2017 16:16
by SkyFrontier
Hello again, highend!
Would you mind sharing the regex-based version? I'd like to give it a try, despite de unicode remark (my inputs generally [99,99% - 0,01% being unknown/undesired] won't have it). By the way, what would happen in the presence of unicode? Silent fail? Untouched string? Total crash?
Thanks much.
Re: Clean out a list of dupes
Posted: 19 Apr 2017 16:50
by highend
Time comparison (with 5k entries):
formatlist : 46 ms
while loop: 17 seconds
Code: Select all
$list = <<<>>>
@Cyrillic-ДЖЙ-character [UTF-8]
@Chinese-漢字-character [UTF-8 BOM]
@Gotham 哥谭 (2014)
@Greek αβγδεζ.txt
@Chinese-漢字-character [UTF-8 BOM]
@Cyrillic-ДЖЙ-character [UTF-8]
@Gotham 哥谭 (2014)
@Greek αβγδεζ.txt
>>>;
$whitelist = <<<>>>
@Cyrillic-ДЖЙ-character [UTF-8]
@Greek αβγδεζ.txt
>>>;
/*
Expected list:
@Cyrillic-ДЖЙ-character [UTF-8]
@Chinese-漢字-character [UTF-8 BOM]
@Gotham 哥谭 (2014)
@Greek αβγδεζ.txt
@Cyrillic-ДЖЙ-character [UTF-8]
@Greek αβγδεζ.txt
*/
$start = now("msecs");
$final = "";
while ($i++ < gettoken($list, "count", <crlf>)) {
$line = gettoken($list, $i, <crlf>);
$pattern = regexreplace($line, "([\\^$.+*|?(){\[])", "\$1");
if ($line == "<:REMOVED:>") { continue; } // Was this line previously removed by regexreplace?
elseif !(regexmatches($whitelist, "^$pattern$", <crlf>)) { // Is the current line not in the whitelist?
$list = regexreplace($list, "^$pattern$", "<:REMOVED:>");
}
$final = $final . $line . <crlf>;
}
$duration = now("msecs") - $start;
text $duration . <crlf 3> . formatlist($final, "e", <crlf>);
Re: Clean out a list of dupes
Posted: 19 Apr 2017 17:41
by SkyFrontier
Thank you!
I'm generally aware that regex doesn't deal with unicode. But... what are the downsides? (besides the timings, of course, which is not related in case)
Re: Clean out a list of dupes
Posted: 19 Apr 2017 17:48
by highend
I'm generally aware that regex doesn't deal with unicode
Sure it does but it requires a specific notation for each character it has to deal with...
As long as full lines are captured (using ^ + $) and these patterns
are escaped for metacharacters it shouldn't matter, if it contains unicode or not