Is there a way to detect UTF off of a URL?

SkyFrontier · Post by **SkyFrontier** » 01 Nov 2012 22:06

Title says it all.
I'm wondering on a way XY scripting can detect encoding of a url content without having to readurl() it, then writing a file beforehand so filetype() can do the job.
Thanks.

Marco · Post by **Marco** » 02 Nov 2012 00:09

Mmh, let's see if I get this correctly.
The HTML code of www.xyplorer.com page starts as follows:

Code: Select all

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"  "http://www.w3.org/TR/html4/loose.dtd">
<html><head><title>XYplorer - A Windows File Manager and Explorer Replacement</title>
  <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
  <meta http-equiv="Content-Language" content="en">

You are interested in getting the info on the third line? If so then no, the only way is reading the HTML code, since the HTTP response (which you can inspect by going here http://web-sniffer.net/ ) doesn't contain info about that.

SkyFrontier · Post by **SkyFrontier** » 02 Nov 2012 13:43

Thanks, Marco.
As XY can via writefile()

t: [default] text; auto-detects whether text can be written as ASCII or needs to be written as UNICODE

I thought it should be a way to produce sample of (online or offline) documents written in original sources' encoding as the clipped section may not contain a special character thus not triggering writefile auto-detection.

In case of online sources, writing a local file then using readfile([numbytes]), then writefile, then deleting the source document is so CPU/HDD intensive that I scrapped the whole method.

Detection of ~charset=iso-8859-1~ can be done, but that's not accurate on the scenario I describe, see?

Marco · Post by **Marco** » 02 Nov 2012 13:48

Maybe isunicode() can help you? [pag. 348 of the help]

SkyFrontier · Post by **SkyFrontier** » 02 Nov 2012 13:52

Marco wrote:Maybe isunicode() can help you? [pag. 348 of the help]

It could, but it doesnt support URLs parsing...

Marco · Post by **Marco** » 02 Nov 2012 13:57

You could do something like

Code: Select all

$code=readurl("desiredurl.com");
isunicode($code);

and check what isunicode returns. Keep in mind that the would html code must flow somewhere in the code. Even if isunicode could handle urls it still would have to make http requests and then check the whole code. As of now it simply is visible to the user.

SkyFrontier · Post by **SkyFrontier** » 02 Nov 2012 13:57

:!:A combination of redurl+isunicode may solve the problem, Marco. Ill give this one another try. Thanks!
(dumb me...)

SkyFrontier · Post by **SkyFrontier** » 02 Nov 2012 13:58

Marco wrote:You could do something like
Code: Select all
$code=readurl("desiredurl.com");
isunicode($code);
and check what isunicode returns. Keep in mind that the would html code must flow somewhere in the code. Even if isunicode could handle urls it still would have to make http requests and then check the whole code. As of now it simply is visible to the user.

Yes, just had the same idea... Will give it a go.

XYplorer Beta Club

Is there a way to detect UTF off of a URL?

Is there a way to detect UTF off of a URL?

Re: Is there a way to detect UTF off of a URL?

Re: Is there a way to detect UTF off of a URL?

Re: Is there a way to detect UTF off of a URL?

Re: Is there a way to detect UTF off of a URL?

Re: Is there a way to detect UTF off of a URL?

Re: Is there a way to detect UTF off of a URL?

Re: Is there a way to detect UTF off of a URL?