writefile() & unicode

[jacky]

Little story: I have a little script that I called Quick Edit File and which I find really helpful. But something that I didn't like nor expected happened today: I edited a file, which was then used in another process and caused quite a few problems. Not because I made a mess editing it, but because XY did -- sort of.

It was a simple text file, in Ascii-DOS as XY says (in QFV). After I edited it with QFE (which is just a writefile(input(readfile())) basically) it turns out XY wrote it in UTF-16-DOS (again, from QFV), and does this conversion even if I don't change the file. I found out why it does so, because there is the euro sign (€) in that file; I also found out a temporary workaround, I use mode b (binary) with writefile (ta (text-ascii) would work as well).

Of course, as soon as I'll want to edit an unicode file with characters that need it, I'll be in a brand new mess. So I was wondering, couldn't the automatic mode (t) be changed, adding ANSI in between ASCII & UNICODE for example, so that if I use € for which ANSI is enough, it uses that, and only if I use other characters not available in ANSI (e.g. 葉) then it moves to UNICODE ?

[admin]

Issue confirmed. But I don't see a good solution with the current setup.

I could only add a new flag that would check if the text can be expressed in ANSI. But this test would take a little time so it would get slightly slower depending on the file size.

[jacky]

Issue confirmed. But I don't see a good solution with the current setup.

I could only add a new flag that would check if the text can be expressed in ANSI. But this test would take a little time so it would get slightly slower depending on the file size.

Well that would be fine by me. It would be optional so people who don't care could keep the current/fastest way, and I would be happy to know that my ANSI files will stay in ANSI, all without the risk of messing up UNICODE files... (Plus, I'm confident that "slightly slower" depending on file size won't bother me, if I even notice it that is.)

[jacky]

Adding on that, while actually being quite/completely off-topic:

- a few functions allow to specify a window-size (html, text...) which is good. However, if one writes a script that is intended to be shared, or just used on multiple computers, it can get irrelevant as not all screens have the same resolution. How about adding percentage support, e.g. 80% to get the window to be 80% of the screen size ?

- in addition, maybe getinfo() could allow one to get the screen width & height, in case one doesn't want to use a relative (%) size, yet have a few sizes available automatically chosen according to screen size?

- on that area, I would like to see QFV to adjust its size depending on the screen size. I'm running on 1440x900 here and I wouldn't mind getting a bit more text on one screen that I do now...

[admin]

Adding on that, while actually being quite/completely off-topic:

- a few functions allow to specify a window-size (html, text...) which is good. However, if one writes a script that is intended to be shared, or just used on multiple computers, it can get irrelevant as not all screens have the same resolution. How about adding percentage support, e.g. 80% to get the window to be 80% of the screen size ?

- in addition, maybe getinfo() could allow one to get the screen width & height, in case one doesn't want to use a relative (%) size, yet have a few sizes available automatically chosen according to screen size?

- on that area, I would like to see QFV to adjust its size depending on the screen size. I'm running on 1440x900 here and I wouldn't mind getting a bit more text on one screen that I do now...

All agreed. Good stuff.

[admin]

Issue confirmed. But I don't see a good solution with the current setup.

I could only add a new flag that would check if the text can be expressed in ANSI. But this test would take a little time so it would get slightly slower depending on the file size.

Well that would be fine by me. It would be optional so people who don't care could keep the current/fastest way, and I would be happy to know that my ANSI files will stay in ANSI, all without the risk of messing up UNICODE files... (Plus, I'm confident that "slightly slower" depending on file size won't bother me, if I even notice it that is.)

I thought about it and decided that the corrected behavior is nothing anybody would dislike so I just fixed it.

Now, of course, the reverse transformation can happen (but it occurs much more rarely in reality I would think): a UTF-16 input file read that's with readfile and written with writefile + "t" argument will be auto-transformed to ASCII if it contains no wide chars or only things like € sign.

[jacky]

Now, of course, the reverse transformation can happen (but it occurs much more rarely in reality I would think): a UTF-16 input file read that's with readfile and written with writefile + "t" argument will be auto-transformed to ASCII if it contains no wide chars or only things like € sign.

Yeah, I thought about that as well, but since XY has no idea to know the source of the data used in writefile() (i.e. that it comes from a readfile()), there's no real way around it.

Well, actually there could be : have a function to return whether a specified file is in ASCII or UNICODE, i.e. whether it should be used with ta or tu. Then, one could use that to determine which flag to use with writefile(), thus ensuring that there's no such change. Ideally, that function would also work on a given string, so as to allow one to ensure that the text to be written (e.g. from input()) doesn't require a change (e.g. from ASCII to UNICODE) based on its new content...

[admin]

Now, of course, the reverse transformation can happen (but it occurs much more rarely in reality I would think): a UTF-16 input file read that's with readfile and written with writefile + "t" argument will be auto-transformed to ASCII if it contains no wide chars or only things like € sign.

Yeah, I thought about that as well, but since XY has no idea to know the source of the data used in writefile() (i.e. that it comes from a readfile()), there's no real way around it.

Well, actually there could be : have a function to return whether a specified file is in ASCII or UNICODE, i.e. whether it should be used with ta or tu. Then, one could use that to determine which flag to use with writefile(), thus ensuring that there's no such change. Ideally, that function would also work on a given string, so as to allow one to ensure that the text to be written (e.g. from input()) doesn't require a change (e.g. from ASCII to UNICODE) based on its new content...

Yo, that's it. Will do.

[jacky]

Thanks! Updated my script, now it uses mode "t" again and keeps file in ANSI when e.g. "only" the euro sign is present, as well as forcing "tu" if filetype() indicated the file was Unicode, regardless of whether it is required by content or not.

[zer0]

If I may ride the wave of this wish, is it currently possible (if at all?) to convert from UTF-8 to ANSI? Some software that I use doesn't interpret Unicode, so I need the text to be in ANSI.

[jacky]

If I may ride the wave of this wish, is it currently possible (if at all?) to convert from UTF-8 to ANSI? Some software that I use doesn't interpret Unicode, so I need the text to be in ANSI.

Well, oddly enough the new functions (or writefile) do not seem to support UTF8 (neither does QFV), but there's some conversion functions available. So if you have a file that is encoded in UTF8 you can try something like this to convert it to ANSI:

Code: Select all

writefile(<curitem>,utf8decode(readfile()));

Of course, if there are characters that aren't supported/require Unicode, it won't be converted from UTF8 to ANSI, but to UTF16. You could force writefile to write in ASCII/ANSI (mode ta) but then you would screw up your file...