ReadURL to Clean Out HTML Code.

[SkyFrontier]

::readurl $a, "http://www.xyplorer.com/index.htm"; text $a produces an output like this:

Code: Select all

...
      <div class=listgrey><b>Operating Systems:</b> Windows 7, Windows Server 2008, Windows Vista, Windows XP, Windows Server 2003, Windows 2000, Windows NT, Windows Me, Windows 98; x64/x32.</div>
      <div class=listgrey>XYplorer is a <b>Portable File Manager</b>. It doesn't require any installation or changes to your system or registry. <b>Stickware</b>, ideal for a <b>USB drive</b>.</div>
      <div class=listgrey><b>Tabbed Browsing.</b> Cutting-edge file management using a true <a href="product.htm#p1" class="lite">multi-tabbed interface</a>.</div>
...

-wouldn't it be cool if it optionally could parse this as an output instead?

Code: Select all

...
      Operating Systems: Windows 7, Windows Server 2008, Windows Vista, Windows XP, Windows Server 2003, Windows 2000, Windows NT, Windows Me, Windows 98; x64/x32.
      XYplorer is a Portable File Manager. It doesn't require any installation or changes to your system or registry. Stickware, ideal for a USB drive.
      Tabbed Browsing. Cutting-edge file management using a true multi-tabbed interface.
...

[SkyFrontier]

Usage scenario:
One could list several news URLS, get the script running and have one file for each site updated with new information appended to previous and optionally open and ready to read on desktop.
(as XY seems to recognize file size at the server, it also could have a timer and reload script again [custom event action?] in order to download and append newest, changed version of target URL.)
KooL!

(I know, I'm dreaming a bit...)

[Stefan]

FROM:
<div class=listgrey><b>Operating Systems:</b> Windows 7, Windows Server 2008, Windows Vista, Windows XP, Windows Server 2003, Windows 2000, Windows NT, Windows Me, Windows 98; x64/x32.</div>
<div class=listgrey>XYplorer is a <b>Portable File Manager</b>. It doesn't require any installation or changes to your system or registry. <b>Stickware</b>, ideal for a <b>USB drive</b>.</div>
<div class=listgrey><b>Tabbed Browsing.</b> Cutting-edge file management using a true <a href="product.htm#p1" class="lite">multi-tabbed interface</a>.</div>

TO:
Operating Systems: Windows 7, Windows Server 2008, Windows Vista, Windows XP, Windows Server 2003, Windows 2000, Windows NT, Windows Me, Windows 98; x64/x32.
XYplorer is a Portable File Manager. It doesn't require any installation or changes to your system or registry. Stickware, ideal for a USB drive.
Tabbed Browsing. Cutting-edge file management using a true multi-tabbed interface.

You can try:

Code: Select all

readurl $a, "http://www.xyplorer.com/index.htm"; 
   $a = regexreplace($a, "<[^<]*>", ""); //drop HTML <tags>
   text $a;

EDIT:

v9.50.0001 - 2010-09-09 11:15
* SC readurl got a new StripHTML parameter which can be used to
return the (ideally) pure text without any HTML formatting.

Syntax: readurl(url, [nocookies], [StripHTML])
::text readurl("http://www.xyplorer.com/download.htm", 0, 1);

[SkyFrontier]

Amazing.
-later I'll try to get this working in "offline mode" to read & write offline saved pages.
Have TONS of good (to my taste) interviews that needs that sort of automatic cleansing (it would be humanly impossible to get the job done via conventional tools).
Will post when & if done.
Thank you very much!

[SkyFrontier]

Well.
URLdecode seems to have some sort of limitation I was unable to track - it works ok with small pieces of text, but not with larger ones (can't say how much). "My bug", some will say.
StripHTML has a severe approach dealing with excessive tabs, spaces and empty lines.
I can't make UTF8decoding pre-process the texts I'm working with, perhaps not the stuff I need, after all.
-something is broken in the commands I used, because some parts should read "Français/Português" instead of "Français/Português" (even disabling the utf8decode($a); section produced no satisfactory result, but seems to be a minor issue.

Anyway, here it comes:
CleanOutHTMLCode_p6_OfflineMode.xys will read a .txt and produce a clone out of it with extension .diz, good portions of HTML coding must be cleansed;

ReadURLToCleanOutHTMLCode_p23_FINAL.xys will pop a cool input box, suggesting as entry an URL (assuming you already have one at hand, ready at the clipboard) and then will access the internet (allow XY to pass your firewall!), retrieve the URL, clean most of HTML coding and pop a box where you can read it - at the same time it copies the output into the clipboard, getting it ready for you to edit anywhere else. Some may see this as a way to cheat bosses/network admins and read their favorite stuff at work, some will certainly use this as a productivity tool as it is intended to be.

To do (some day, one day...):
-some sort of error checking;
-multi-file processing for the offline converter;
-computer connection check for the URL reader;

[SkyFrontier]

Can someone please tell me how to get rid of sequences like the following?:
(copy and paste the sequences into a good text editor and you'll see their differences...)

Code: Select all

Code: Select all

Code: Select all

Telling the script to erase individually like

Code: Select all

Code: Select all

Code: Select all

will screw everything, completely eliminating some separators which are desired to stay.

EDIT: Forum screwed formatting. Added sample as .html file, attached.

[Jibz]

http://www.codinghorror.com/blog/2009/1 ... u-way.html

You can't parse (anything but very simple) HTML with RegExp, sorry.

[SkyFrontier]

...hmmmmm, I see.
That explains why I can't remember what I did at the recess. Nor why I haven't solved the problem after so many efforts and attempts. I felt for Cthulhu's call. Damn!!!
-thanks much for the post and for the link, Jibz!
Now I HAVE to locate Detagger's creator and thank him personally, too...

[SkyFrontier]

Updated versions;
-Support for few more characters I've came across;
-Edit Mode for Detagger - Offline:
just remove/set the comment to get the script working on clipboard content or currently selected item.
Default - work on selected (text based!) item.

Code: Select all

//   $a = <clipboard>;
   $a = readfile(<curitem>, b),,b);

Edited to work with clipboard contents:

Code: Select all

   $a = <clipboard>;
//   $a = readfile(<curitem>, b),,b);

-Other minor changes on detagging code.
-Included better sample files so one can try to help dealing with the unnecessary spacing without removing them all (aggressive method). The idea is to get rid of excessive spacing without removing them all, limiting them to at most of 2 <crlf> as per paragraph, in case they are in sequences greater or equal to 3, so they'll keep original formatting when coming as single or double CRLFs.

There's also a little code attempting to add smartness to Detagger - Offline Mode so there should be no need of prior editing the code itself , which failed. The idea was to prompt user which mode would start, making the operation on-the-fly and easier to use. I'm open to suggestions.

Code: Select all

//FAILED   input $met, "Chose and ADJUST the method to be used", "clipboard|readfile(curitem, b),,b)";
//FAILED   $a = $met;