Clean out a list of dupes

[SkyFrontier]

Is there a way I can dedupe a list of non-consecutive duplicated items, ie WITHOUT sorting it first (performance reasons), possibly using regex?
For this to perfectly work, I'd also need it to preserve a white list of exact terms (paths or phrases), ie, such terms will NOT be deduped at all.
How?

TIA, people.

[highend]

The fastest way is formatlist() with format=Fdn and a filter.
This includes sorting (otherwise duplicates wouldn't be removed).

When using formatlist() you could filter with a negating expression containing your whitelist
and then add your whitelisted entries to the list again (again with the help of formatlist())

Code: Select all

    $list = <<<>>>
you
people
me
you
me
people
other
>>>;

    $whitelist = <<<>>>
you
me
>>>;

    $deduplicated = formatlist($list, "Fdn", <crlf>, "!" . $whitelist);
    $whitelisted  = formatlist($list, "F", <crlf>, $whitelist);
    text $deduplicated . <crlf> . $whitelisted;

All other methods will be <x> times slower (ofc depending on how many entries you have)

[SkyFrontier]

Hello again, highend!

Would you mind sharing the regex-based version? I'd like to give it a try, despite de unicode remark (my inputs generally [99,99% - 0,01% being unknown/undesired] won't have it). By the way, what would happen in the presence of unicode? Silent fail? Untouched string? Total crash?

Thanks much.

[highend]

Time comparison (with 5k entries):
formatlist : 46 ms
while loop: 17 seconds

Code: Select all

    $list = <<<>>>
@Cyrillic-ДЖЙ-character [UTF-8]
@Chinese-漢字-character [UTF-8 BOM]
@Gotham 哥谭 (2014)
@Greek αβγδεζ.txt
@Chinese-漢字-character [UTF-8 BOM]
@Cyrillic-ДЖЙ-character [UTF-8]
@Gotham 哥谭 (2014)
@Greek αβγδεζ.txt
>>>;

    $whitelist = <<<>>>
@Cyrillic-ДЖЙ-character [UTF-8]
@Greek αβγδεζ.txt
>>>;

/*
Expected list:
@Cyrillic-ДЖЙ-character [UTF-8]
@Chinese-漢字-character [UTF-8 BOM]
@Gotham 哥谭 (2014)
@Greek αβγδεζ.txt
@Cyrillic-ДЖЙ-character [UTF-8]
@Greek αβγδεζ.txt
*/
    $start = now("msecs");

    $final = "";
    while ($i++ < gettoken($list, "count", <crlf>)) {
        $line = gettoken($list, $i, <crlf>);
        $pattern = regexreplace($line, "([\\^$.+*|?(){\[])", "\$1");

        if ($line == "<:REMOVED:>") { continue; } // Was this line previously removed by regexreplace?
        elseif !(regexmatches($whitelist, "^$pattern$", <crlf>)) { // Is the current line not in the whitelist?
            $list = regexreplace($list, "^$pattern$", "<:REMOVED:>");
        }
        $final = $final . $line . <crlf>;
    }
    $duration = now("msecs") - $start;
    text $duration . <crlf 3> . formatlist($final, "e", <crlf>);

[SkyFrontier]

Thank you!

I'm generally aware that regex doesn't deal with unicode. But... what are the downsides? (besides the timings, of course, which is not related in case)

[highend]

I'm generally aware that regex doesn't deal with unicode

Sure it does but it requires a specific notation for each character it has to deal with...

As long as full lines are captured (using ^ + $) and these patterns
are escaped for metacharacters it shouldn't matter, if it contains unicode or not