[GRANTED] SC function wish - Base64 encoding/decoding

[Marco]

Enternal, do you get the gibberish in the final file or in the variable that holds the content of the final file?

[Enternal]

Enternal, do you get the gibberish in the final file or in the variable that holds the content of the final file?

Oh! Never thought of that. I tested it and it's gibberish in the variable that holds the content.

After that, I tried reading that variable that READS the file and that one is gibberish. So it really is weird. I simply used:
Text $File and Text $Encoding
to read display what was read from the file and in both cases for encoding and decoding, it's messed up.

[Marco]

You shouldn't worry about the content of the variable(s). Files are read in binary mode, that's why you see gibberish. But that's the point too, as you preserve the very bytes by doing so. That's what matters, because base64 is useful for encoding bytes into text.

[Enternal]

But if it supposed to preserve the very bytes, how come the decoding process did not revert it back to the original? So basically I encode "Hello!", I should get "Hello!" back out in the process. And I do get that if it's all Roman letters in a text file. The problem occurs when I used something that's not Roman letters in a text file like: Здравствуйте!
Encoding and decoding it straight using:
Text Base64Decode(Base64Encode("Здравствуйте!", 1), 1);
works perfectly fine but it becomes a problem if I try encoding and decoding a text file.

Basically I made a text file with UTF-8 encoding. Put:

Code: Select all

Hello!
I am crazy!
I love chickens

Encoding and decoding that file comes out perfectly fine. Now if I add a line that does not have Roman letters so that the file becomes:

Code: Select all

Hello!
I am crazy!
I love chickens
Здравствуйте!

Encoding and decoding failed. In this case, all the lines with Roman letters comes out fine but the line with Russian characters become gibberish.

I have a feeling that it's because of system encoding settings again... *sigh

EDIT: AND I was right on the money about system encodings. I tested on a purely English system:

Code: Select all

System Locale ID: 1033 (en-US)
Thread Locale ID: 1033 (en-US)
Default ANSI Code Page: 1252  (ANSI - Latin I)
Active Code Page: 1252  (ANSI - Latin I)
DBCS Code Page: No

and it works as expected. On the problem system:

Code: Select all

System Locale ID: 1041 (ja-JP)
Thread Locale ID: 1033 (en-US)
Default ANSI Code Page: 1252  (ANSI - Latin I)
Active Code Page: 932   (ANSI/OEM - Japanese Shift-JIS)
DBCS Code Page: Yes

The encoding and decoding failed with gibberish.

DON! SAVE ME!! I just love how I always manage to break something whenever it comes to encoding and decoding.

[bdeshi]

readfile() has a codepage parameter. passing 1252 as codepage could help.

Code: Select all

$File = readfile($Path, b, , 1252);

[admin]

DON! SAVE ME!! I just love how I always manage to break something whenever it comes to encoding and decoding.

The next version allows to decode/encode files directly; this should fix the issues. See the new code versions inserted below.

Code: Select all

"ENCODE"
  $Path = <curitem>;
  $File = ReadFile($Path, "b");
  $Encoding = Base64Encode($File, 1);
  $Encoding = Formatlist($Encoding, "p", <crlf>, 76) . <crlf>;
  WriteFile($Path . ".Base64.txt", $Encoding, , "ta");
"ENCODE 2"
  $Path = <curitem>;
  $Encoding = Base64Encode($Path, 2);
  WriteFile($Path . ".Base64.txt", $Encoding, , "ta");
"DECODE"
  $Path = <curitem>;
  $Encoding = ReadFile($Path, "t");
  $Encoding = Regexreplace($Encoding, <crlf>);
  $File = Base64Decode($Encoding, 1);
  $Path = RegexReplace($Path, "\.Base64\.txt$");
  WriteFile($Path, $File, , "b");
"DECODE 2"
  $Path = <curitem>;
  $File = Base64Decode($Path, 2);
  $Path = RegexReplace($Path, "\.Base64\.txt$");
  WriteFile($Path, $File, , "b");

[admin]

PS: secret syntax:

Code: Select all

base64decode(string, mode)
base64encode(string, mode)

mode:
 1 = string is unicode
 2 = string points to file

[Enternal]

Awesome! Thanks!

[Enternal]

So I tested it and even tested your new code that you provided. Unfortunately the problem is still there with the gibberish.

[admin]

For a UTF-16 file that just contains こんにちはI get this:
//5TMJMwazBhMG8w

[Marco]

For a UTF-16 file that just contains こんにちはI get this:
//5TMJMwazBhMG8w

Works here. Mind, Enternal: when you decode2 the $File variable will contain gibberish, but once it is written to disk the Korean Japanese text will appear correctly (at least here on my system).

[Enternal]

Actually I was playing with Russian letters. Also, I just realized that I was using UTF-8 text files so when I switched that back to UTF-16, it works perfectly!

On the other hand, encoding a simple file and then decoding it still breaks it on my system. So why would that be the case? That's precisely why I was testing it with a text file because it's easier for me to "see" how a file changed. A text file should simply be text so if it's encoded and then decoded, no matter what it is, it should retain the information right? To note, I wasn't looking at the $File variable anymore since it's not very helpful to me. I look at the final output file and that's where I see the problem.

I might as well as attached an example of an original image (GTvmqWB-Original.jpg), then base64 encoded (GTvmqWB.jpg.Base64.txt), and then base64 decoded back (GTvmqWB-Decoded.jpg). The code used:

Code: Select all

"BASE64 ENCODE|:queue"
	// http://www.xyplorer.com/xyfc//viewtopic.php?f=5&t=11016
	// http://www.xyplorer.com/xyfc//viewtopic.php?f=5&t=11442&start=15
	$Path = <curitem>;
	$Encoding = Base64Encode($Path, 2);
	WriteFile($Path . ".Base64.txt", $Encoding, , "ta");
"BASE64 DECODE|:queue"
	$Path = <curitem>;
	$File = Base64Decode($Path, 2);
	$Path = RegexReplace($Path, "\.Base64\.txt$");
	WriteFile($Path, $File, , "b");

System locale settings:

Code: Select all

System Locale ID: 1041 (ja-JP)
Thread Locale ID: 1033 (en-US)
Default ANSI Code Page: 1252  (ANSI - Latin I)
Active Code Page: 932   (ANSI/OEM - Japanese Shift-JIS)
DBCS Code Page: Yes

As you can see, what went in did came back out the same. I hope I make sense what my concerns were. Or... I am just completely using it wrong and not the way it was intended.

[Marco]

Ok, I think that the problems lies in your system locale. In fact, the images you posted are different, but doing the encode-decode passages here generates a bit-perfect copy. Specifically, the decoding passage seems broken on your config, because your encoded file is identical to the encoding that I get here. Let's see what Don can do.

Re text: when you encode-decode a file, it doesn't matter if it's text or binary. It will always be seen as a flow of bytes.

[Enternal]

Ok, I think that the problems lies in your system locale. In fact, the images you posted are different, but doing the encode-decode passages here generates a bit-perfect copy. Specifically, the decoding passage seems broken on your config, because your encoded file is identical to the encoding that I get here. Let's see what Don can do.

Re text: when you encode-decode a file, it doesn't matter if it's text or binary. It will always be seen as a flow of bytes.

Yep! That's exactly what I was trying to get at!

For text, it's just easier for me to check how it changed because I don't have a hex editor on hand all the time. So if it's a text file, whatever I encoded BETTER be the SAME when I decode. Which in my case it did not which is why I questioned what happened. I use text files all the time to do quick checks like this including when I was playing around with encryption software. There was one software that had their encoding done wrong or something but what I encrypted and decrypted were not the same as the original. It's easier than encrypting a file, decrypt the file, and then pull out some hex editing software and compare the values. Too much work (lazy me).

After writing the above, I just realized that XYplorer has a hex viewer! Haha.

[admin]

I had a little idea in the meantime. Try the next beta...