Easier Way to Rename Lots of Files with Unknown Characters

[SkyFrontier]

">>" for current pattern and ">>>" for more complex operations? Sounds good to me. Hope I'm not the only guy in the XY universe who needs to constantly translate files from different portions of the globe into "PC friendly" language...

-and the inclusion of this mode of renaming into scripting would be of a great help considering the eventual changes to come...

Thanks for considering.

[admin]

">>" for current pattern and ">>>" for more complex operations? Sounds good to me. Hope I'm not the only guy in the XY universe who needs to constantly translate files from different portions of the globe into "PC friendly" language...

-and the inclusion of this mode of renaming into scripting would be of a great help considering the eventual changes to come...

Thanks for considering.

Just added it. Even simpler. >>> is not necessary.

Code: Select all

    + Rename Special | Search and Replace: Now you can specify  
      lists with units larger than one character. Also different 
      lengths of old a new string are now allowed in such lists.
      To enable this functionality simply separate the individual 
      units with | (pipe).
      Examples:
        Ä|Ö|Ü|ä|ö|ü>>Ae|Oe|Ue|ae|oe|ue
        &| |->>and|space|hyphen
        Copy|of>>Kopie|von
      Any excess units on either side are ignored.

[Stefan]

If there's a demand I can add the possibility to replace many chars of different lengths.
But the patterns will look a bit clumsy since we need to have a separator.
We could use ">>>" to mark this from of syntax, e.g.:

Code: Select all

ä|ö|ü>>>ae|oe|ue

I am interested.

While reading http://88.191.26.34/XYwiki/index.php/Re ... nd_Replace...
I got the idea how about this

Code: Select all

"ä>ae|ö>oe|ü>ue| on > off |2>Two| Three >3"

"ñ>n|áàâã>>>a|éèê>>>e|íìî>>>i|óòôõ>>>o|úùû>>>u|ç>c|.,[]{}()>>>_|äöü>>aou"

"ñáàâãéèêíìîóòôõúùûç.,[]{}()>>naaaaeeeiiioooouuuc________"



> means: replace whole left string with whole right
>> means: replace single left char with single right at same position
>>> means: replace each of single left char with the same one char on the right
Perhaps introducing new mode m (multi)


EDIT: i am too late.

[jeduardo]

To remove diacritics all you need is API Function FoldStringW Lib "kernel32".

It will unfold á into a´. Then remove the diacritics be removing characters &H300 To &H36F.

Sample Input
ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ

Sample Output
AAAAAAÆCEEEEIIIIÐNOOOOOaaaaaaæceeeeiiiiðnooooo÷øuuuuyþy

[admin]

To remove diacritics all you need is API Function FoldStringW Lib "kernel32".

It will unfold á into a´. Then remove the diacritics be removing characters &H300 To &H36F.

Sample Input
ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ

Sample Output
AAAAAAÆCEEEEIIIIÐNOOOOOaaaaaaæceeeeiiiiðnooooo÷øuuuuyþy

Not bad, thanks for the tip!

[SkyFrontier]

Improved dictionary:

Code: Select all

()!$'?:,&+-ŠŒŽšœžŸ¥µÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýÿ>>___________SOZsozYYuAAAAAAACEEEEIIIIDNOOOOOOUUUUYsaaaaaaaceeeeiiiionoooooouuuuyy

[SkyFrontier]

...And the last refinements (dupes removed, sync'ed minor changes, small notes, etc):

(to be used with Rename Special - Search and Replace and its scripting counterpart)

Code: Select all

FINAL (OUT:  >|<"):
?>G; !>l
Σ|•|—|¬|@|#|$|%|¨|*|{|}|[|]|\|/|=|;|.|^|~|£|₢|&|(|)|!|$|'|?|:|,|&|+|-|Š|Œ|Ž|š|œ|ž|Ÿ|¥|µ|À|Á|Â|Ã|Ä|Å|Æ|Ç|È|É|Ê|Ë|Ì|Í|Î|Ï|Ð|Ñ|Ò|Ó|Ô|Õ|Ö|Ø|Ù|Ú|Û|Ü|Ý|ß|à|á|â|ã|ä|å|æ|ç|è|é|ê|ë|ì|í|î|ï|ð|ñ|ò|ó|ô|õ|ö|ø|ù|ú|û|ü|ý|ÿ>>_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|cr|And|_|_|l|_|_|G|_|_|_|_|_|S|O|Z|s|o|z|Y|Y|u|A|A|A|A|A|A|AE|C|E|E|E|E|I|I|I|I|D|N|O|O|O|O|O|O|U|U|U|U|Y|s|a|a|a|a|a|a|a|c|e|e|e|e|i|i|i|i|o|n|o|o|o|o|o|o|u|u|u|u|y|y

?>_; !>l
Σ|•|—|¬|@|#|$|%|¨|*|{|}|[|]|\|/|=|;|.|^|~|£|₢|&|(|)|!|$|'|?|:|,|&|+|-|Š|Œ|Ž|š|œ|ž|Ÿ|¥|µ|À|Á|Â|Ã|Ä|Å|Æ|Ç|È|É|Ê|Ë|Ì|Í|Î|Ï|Ð|Ñ|Ò|Ó|Ô|Õ|Ö|Ø|Ù|Ú|Û|Ü|Ý|ß|à|á|â|ã|ä|å|æ|ç|è|é|ê|ë|ì|í|î|ï|ð|ñ|ò|ó|ô|õ|ö|ø|ù|ú|û|ü|ý|ÿ>>_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|cr|And|_|_|l|_|_|_|_|_|_|_|_|S|O|Z|s|o|z|Y|Y|u|A|A|A|A|A|A|AE|C|E|E|E|E|I|I|I|I|D|N|O|O|O|O|O|O|U|U|U|U|Y|s|a|a|a|a|a|a|a|c|e|e|e|e|i|i|i|i|o|n|o|o|o|o|o|o|u|u|u|u|y|y

Code: Select all

?>G; !>l; Æ>E; ₢>c; &>e; ">_
Σ•—¬@#$%¨*{}[]\/=;.^~£₢&()!$'?:,&+-ŠŒŽšœžŸ¥µÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýÿ">>______________________ce__l__G_____SOZsozYYuAAAAAAECEEEEIIIIDNOOOOOOUUUUYsaaaaaaaceeeeiiiionoooooouuuuyy_

?>_; !>l; Æ>E; ₢>c; &>e; ">_
Σ•—¬@#$%¨*{}[]\/=;.^~£₢&()!$'?:,&+-ŠŒŽšœžŸ¥µÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýÿ">>______________________ce__l________SOZsozYYuAAAAAAECEEEEIIIIDNOOOOOOUUUUYsaaaaaaaceeeeiiiionoooooouuuuyy_

[SkyFrontier]

Using the above formula as it is may produce files having several sequenced underscores.
This little sequence of commands (script?!) can solve the problem:

Code: Select all

   rename s, "´|Ã|Σ|•|—|¬|@|#|$|%|¨|*|{|}|[|]|\|/|=|;|.|^|~|£|₢|&|(|)|!|$|'|?|:|,|&|+|-|Š|Œ|Ž|š|œ|ž|Ÿ|¥|µ|À|Á|Â|Ã|Ä|Å|Æ|Ç|È|É|Ê|Ë|Ì|Í|Î|Ï|Ð|Ñ|Ò|Ó|Ô|Õ|Ö|Ø|Ù|Ú|Û|Ü|Ý|ß|à|á|â|ã|ä|å|æ|ç|è|é|ê|ë|ì|í|î|ï|ð|ñ|ò|ó|ô|õ|ö|ø|ù|ú|û|ü|ý|ÿ>>_|o|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|cr|And|_|_|l|_|_|G|_|_|_|_|_|S|O|Z|s|o|z|Y|Y|u|A|A|A|A|A|A|AE|C|E|E|E|E|I|I|I|I|D|N|O|O|O|O|O|O|U|U|U|U|Y|s|a|a|a|a|a|a|a|c|e|e|e|e|i|i|i|i|o|n|o|o|o|o|o|o|u|u|u|u|y|y";
   rename s, " /_"; rename s, "_____/_"; rename s, "____/_"; rename s, "___/_"; rename s, "__/_"; rename s, "_ _/_"; rename s, "_./.";

The ending rename s, "_./."; portion should be able to turn a file named Test_.txt into Test.txt, but it doesn't - XYplorer reads the ending dot as part of the extension and not of the filename. Also, some resulting files may begin with an undesired underscore or lower cased. RegEx may solve the problem. Any solution, please...?

[SkyFrontier]

Improved version of the dictionary.
Based off long time usage, I added a few tweaks to it and removed two duplicates, being one inconsistent.
The most important change: now it deals with the "_." (underscore|extension dot) problem at the end of renaming. (TheQwerty's tip: rename r, "_\. > .").

Code: Select all

//Renaming Dictionary

   rename s, "´|Σ|•|—|¬|@|#|$|%|¨|*|{|}|[|]|\|/|=|;|.|^|~|£|₢|&|(|)|!|$|'|?|:|,|+|-|Š|Œ|Ž|š|œ|ž|Ÿ|¥|µ|À|Á|Â|Ã|Ä|Å|Æ|Ç|È|É|Ê|Ë|Ì|Í|Î|Ï|Ð|Ñ|Ò|Ó|Ô|Õ|Ö|Ø|Ù|Ú|Û|Ü|Ý|ß|à|á|â|ã|ä|å|æ|ç|è|é|ê|ë|ì|í|î|ï|ð|ñ|ò|ó|ô|õ|ö|ø|ù|ú|û|ü|ý|ÿ>>_|_|_|_|_|_|_|_|Perc|_|_|_|_|_|_|_|_|_|_|_|_|_|_|cr|And|_|_|l|_|_|G|_|_|_plus_|_|S|O|Z|s|o|z|Y|Y|u|A|A|A|a|A|A|AE|C|E|E|E|E|I|I|I|I|D|N|O|O|O|O|O|O|U|U|U|U|Y|s|a|a|a|a|a|a|a|c|e|e|e|e|i|i|i|i|o|n|o|o|o|o|o|o|u|u|u|u|y|y";
   rename s, " /_"; rename s, "_____/_"; rename s, "____/_"; rename s, "___/_"; rename s, "__/_"; rename s, "_ _/_"; rename r, "_\. > .";

ToDo: When I find the time, I'll try to port the best dictionary-based solution I've built this far: ReadURL - XY Web-Text Browser and HTML Detagger (Offline Mode)

EDIT: Removed an obsolete portion of the code, too.

[admin]

You know about this, do you?

Code: Select all

v8.90.0001 - 2010-03-11 21:38
    + Rename Special: Added command
        Remove Diacritics (ä > a ...)
      Strips any diacritical marks from the letters in the selected item 
      names.

[SkyFrontier]

Yes, I tried it in the past but in the end I've preferred to do the thing as a whole (there were some characters I'd like to transliterate in a way I could have control over them).
But - perhaps trying to fine-tune the thing combining both methods I get a smaller dictionary, easier to handle (it's a pain to revise this as it currently is...).
If I'm not in mistake, the command was introduced because of a tip from another user (jeduardo, his only post), in this very thread, above.
Have to test.
Thanks for the tip, Don - it'll force me to update it sooner than I thought...

EDIT: 2nd reason for not to rely completely on it: it'll take me a second pass to remove other non-latin characters...

[TheQwerty]

Give this a try, it should give you the same results (and I think it might be easier to maintain)...

EDIT: Or don't.. I'm stupid and hardly ever use Search & Replace... give me a minute..

[SkyFrontier]

Give this a try, it should give you the same results (and I think it might be easier to maintain)...
EDIT: Or don't.. I'm stupid and hardly ever use Search & Replace... give me a minute..

Thanks. Also reminded an old trick you taught me regarding regexreplace, this may come handy too.

Code: Select all

$out = RegexReplace($out, 'Ù|Ú|Û|Ü', 'u');

Damn... it gets harder before it gets easier...

[TheQwerty]

So let me try again:

Code: Select all

"Take 2"
	//Removed "*\/?:" from the list, since they cannot be in filenames anyhow.
	//Removed "ŠŽšžŸÀÁÂÃÄÅÇÈÉÊËÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝàáâãäåçèéêëìíîïñòóôõöøùúûüýÿ" because Remove Diacritics handles them as desired.
	//Re-ordered search and replace characters (alphbetical by desired replacement.
	//Separated into a separate rename all of the items that are replaced by underscore.
	//Changed consecutive renaming to handle '_' combinations to one which will collapse any number of ' ' or '_' to a single '_'.
               
	rename s, "+|Æ|æ|&|₢|Ð|?|!|Œ|Ø|ð|œ|%|ß|µ|¥>>_plus_|AE|a|And|cr|D|G|l|O|O|o|o|Perc|s|u|Y";
	rename s, "´Σ•—¬@#$¨{}[]=;.^~£()$',->_";
	#139;   //Remove Diacritics
	rename r, '[ _]+ > _';   //Collapses one or more ' ' or '_' to a single '_'.

You could pull 'Œ|Ø|ð|œ' out of the first rename and make it two separate renames one to 'O' and one to 'o' but it doesn't seem like enough of a management nightmare to warrant that.

I'm not satisfied with this method as it's committing four different renames when it should just do string manipulation and then a single rename but I'm too lazy to write a script to manually handle the diacritics, and I don't need this thing so I'm not going to bother.

I also have a feeling you'll hate it because if you have Rename Special->Preview All enabled #139; will be show the preview dialog. Maybe you can convince Don to modify Rename() to prevent this. Options for that might be adding modes for each of the Rename Specials which don't accept pattern and ignoring pattern, or add a new mode which uses the pattern to determine which of those non-pattern renames to use, such as:
Rename 'm', 'Remove Diacritics'; //Mode m = Menu, Pattern = Menu Caption
or
Rename 'i', '139'; //Mode i = ID, Pattern = Command ID.

[jeduardo]

Don't forget that Unicode allows for combining multiple diacritics for a single base character.
To handle all the possible variations through RegExp would be a nightmare.

Here is working code for Visual Basic 6.0 that will unfold the diacritics using API FoldStringW.
Once this is done it is a simple matter to just remove the diacritic modifiers that are in the range 0x300 To 0x36F.
It should be easy enough to convert this to C++.

Sample usage:

strStripped = VBFoldString(strWithDiacritics)


Option Explicit

Private Declare Function FoldStringW Lib "kernel32" (ByVal dwMapFlags As Long, ByVal lpSrcStr As Long, ByVal cchSrc As Long, ByVal lpDestStr As Long, ByVal cchDest As Long) As Long

' Locale Independent Mapping Flags.
Public Const MAP_FOLDCZONE = &H10 'fold compatibility zone chars
Public Const MAP_PRECOMPOSED = &H20 'convert to precomposed chars
Public Const MAP_COMPOSITE = &H40 'convert to composite chars
Public Const MAP_FOLDDIGITS = &H80 'all digits to ASCII 0-9
Public Const MAP_EXPANDLIGATURES = &H200


Public Function VBFoldString(ByVal inString As String, _
Optional ByVal dwMapFlags As Long = MAP_COMPOSITE, Optional ByVal bStrip As Boolean = True) As String

Dim BufLen As Long
Dim i As Long
Dim strTemp As String

BufLen = FoldStringW(dwMapFlags, ByVal StrPtr(inString), Len(inString), ByVal 0&, 0&)

If (BufLen) Then ' Allocate buffer
VBFoldString = Space$(BufLen)
If (FoldStringW(dwMapFlags, ByVal StrPtr(inString), Len(inString), ByVal StrPtr(VBFoldString), BufLen) = 0) Then
VBFoldString = vbNullString
Else
If bStrip Then
VBFoldString = RemoveCombiningDiacritics(VBFoldString)
End If
End If
End If
End Function


Public Function RemoveCombiningDiacritics(ByVal strText As String) As String
Dim i As Long
For i = &H300 To &H36F
strText = Replace(strText, ChrW(i), "")
Next
RemoveCombiningDiacritics = strText
End Function