Page 1 of 1
Writing unicode files
Posted: 05 May 2024 15:02
by MBaas
I just noticed a problem when I tried to process that I exported from XY. For a repro, try this:
Code: Select all
writefile("foo1.txt","""Mein Behälter"" C:\Data\Mein Behälter\",,"utf8bom");
Open foo1.txt with a binary editor and examine the two "ä"s: the first one is written as "C3 A4" - the second one as "61 CC 88".
I suppose this is "somewhat legal" - but it's not entirely correct:
VS Code (which is very tolerant) has no problems with it, UltraEdit can also handle it - but its hex mode shows there's some weirdness.
05-05-2024_15-02-01.png
And the environment I want to read it from simple injects blanks - which messes up subsequent processing, ofc.
Unfortunately I don't know this stuff well enough to provide a better explanation - but I hope it is reproduceable and fixable anyway
Re: Writing unicode files
Posted: 05 May 2024 16:27
by Horst
You write UTF 8 with BOM.
So get the BOM characters, or what else do you expect ?
Re: Writing unicode files
Posted: 05 May 2024 16:46
by admin
Looks like a bug, but I ran your script and the resulting file looks as it should:
2024-05-05_164411.png
I'm using the English locale though, which might (but shouldn't!) make a difference:
Code: Select all
System / Thread Locale ID: 1033 (en-US) / 1033 (en-US)
Default ANSI Code Page: 1252 (ANSI - Latin I)
Active ANSI Code Page: 1252 (ANSI - Latin I)
Default OEM code page: 437 (OEM - United States)
Active OEM Code Page: 437 (OEM - United States)
Re: Writing unicode files
Posted: 06 May 2024 08:22
by MBaas
Just to confirm...I changed my system settings - but it did not affect the result of writefile!
My settings now match yours: (at least we know now that THIS did not cause the difference - as you said!)
Code: Select all
System / Thread Locale ID: 1033 (en-US) / 1033 (en-US)
Default ANSI Code Page: 1252 (ANSI - Latin I)
Active ANSI Code Page: 1252 (ANSI - Latin I)
Default OEM code page: 437 (OEM - United States)
Active OEM Code Page: 437 (OEM - United States)
Re: Writing unicode files
Posted: 06 May 2024 08:46
by admin
Try this:
Code: Select all
text utf8encode("""Mein Behälter"" C:\Data\Mein Behälter\", 1, 1);
Re: Writing unicode files
Posted: 06 May 2024 08:58
by MBaas
Code: Select all
"Mein Behälter" C:\Data\Mein Behälter\
Re: Writing unicode files
Posted: 06 May 2024 09:11
by admin
And this:
Code: Select all
text hexdump(utf8encode("""Mein Behälter"" C:\Data\Mein Behälter\", 0, 1), 1);
Re: Writing unicode files
Posted: 06 May 2024 09:13
by MBaas
Looks good as werll:
Code: Select all
00000000: 22 4D 65 69 6E 20 42 65 68 C3 A4 6C 74 65 72 22 "Mein Behälter"
00000010: 20 43 3A 5C 44 61 74 61 5C 4D 65 69 6E 20 42 65 C:\Data\Mein Be
00000020: 68 C3 A4 6C 74 65 72 5C hälter\
Re: Writing unicode files
Posted: 06 May 2024 09:17
by admin
Hmm. Are you sure that UltraEdit is showing the truth?
Re: Writing unicode files
Posted: 06 May 2024 09:45
by MBaas
It never lied to me (so far). i#ve attached the zipped file if you wanna check yourself.
Re: Writing unicode files
Posted: 06 May 2024 10:01
by admin
Yes, it's true. Can't explain it. Seems impossible.
What happens when you put more "ä" characters in the string? Any patterns?
Re: Writing unicode files
Posted: 06 May 2024 10:15
by MBaas
It's weird! I tried to write lots of "ä"s - they all were correct. Just my real use case (with the folder name "Gehälter") and the random repro seem to expose that behavior...
Re: Writing unicode files
Posted: 06 May 2024 10:27
by admin
It might be a different "ä" pasted from somewhere. Try to delete this character and type it again.
Re: Writing unicode files
Posted: 06 May 2024 10:48
by MBaas
Indeed - the folder name had "ä" in it (61 CC C9) which was faithfully reproduced when the file was created. It's many years old - I have no idea how it got in there. Possibly I used an inferior file mgr and for sure a different O/S. What a weird one - thanks for bearing with me and apologies for being so blind!
