StripHTML - Fine Tuning.

[SkyFrontier]

Some may think this is not a file manager's primary mission, some will see the potential for this regarding current readfile/readURL abilities (as stated here, for example).
So...

Let's consider this thread and StripHTML function.
-can ReadFile has the same ability for offline files? As in this script I made to get the job done (currently works one file at a time... I HATE not knowing how to script!):

Code: Select all

   $sc = readfile(<curitem>, b),,b);
   $a = regexreplace($sc, "<[^<]*>", "");
   copytext $a;
   writefile("<curitem>.diz", $a);

EDIT: even better: WRITEFILE/both having such option, as one may want to leave source, raw data intact.

-can StriptHTML deal with Unicode? If it already does, what have I done wrong to produce results like 05_ReadURL_Detagging_NNegroponte_Transcr_pgm0687.txt diplays, such:

Code: Select all

Nicholas Negroponte: Porque o retorno é muito alto. Se você está em uma posição confortável, em uma posição satisfatória, não assumirá riscos. Não tentará, quer seja na educação ou no comércio, fazer coisas que ponham em risco essa satisfação.

Correct:

Code: Select all

*Nicholas Negroponte*: Porque o retorno é muito alto. Se você está em uma
posição confortável, em uma posição satisfatória, não assumirá riscos. Não
tentará, quer seja na educação ou no comércio, fazer coisas que ponham em
risco essa satisfação.

Attached some samples of such offline source files, Detagger (best so far! - but doesn't batch operations...), PowerEdit and ReadFile+Regex/StripHTML and respective approaches/results.
Contents:

• _xyScript_XYDetagger.xys
  01_SOURCE_NNegroponte_Transcr_pgm0687.txt
  02_IDEAL_byDetagger_NNegroponte_Transcr_pgm0687_2.txt
  03_SIMILARtoCurrentReadFilePlusRegEx_PowerEDIT_NNegroponte_Transcr_pgm0687.txt
  04_ReadFilePlusRegEx_NNegroponte_Transcr_pgm0687.txt.diz
  05_ReadURL_Detagging_NNegroponte_Transcr_pgm0687.txt

Thank you!

Tag - xyScript

[Stefan]

I agree that it is an pity that StripHTML function is available with readurl() only.
I can imagine that it was some work to create that function and so
maybe it [c|sh]ould be accessible as an on script command StripHTML()?
Not often needed, but since it is coded anyway.

[admin]

I agree that it is an pity that StripHTML function is available with readurl() only.
I can imagine that it was some work to create that function and so
maybe it [c|sh]ould be accessible as an on script command StripHTML()?
Not often needed, but since it is coded anyway.

That would be no big problem, but it is really needed?

[admin]

-can StriptHTML deal with Unicode?

You might have a look at utf8decode() and utf8encode()...

[SkyFrontier]

-can StriptHTML deal with Unicode?

You might have a look at utf8decode() and utf8encode()...

I *can't understand* HOW to pre-process UTF8encoding/decoding.
-it should be a stage prior to ReadFile - not implemented (as far as I know/remember...).
-applying encoding/decoding into the output results in... nothing but small changes (unnoticed to the eye) into the document's file size.
-any help on how to apply all of this into a series (~1500) of .html files at once?
Thank you!

[SkyFrontier]

As you can see, I did a reasonable job circumventing some problems. (My limited scripting abilities, after all? Can't say for sure...)
But a thing came to mind:
-what about limiting StripHTML aggressiveness concerning the removal of excessive spacing?
I'm in touch with a few regEx experts since Jibz gave me a hint on what's going on and after some talking it seems that regEx won't deal with excessive spacing like:

Code: Select all

<space><crlf>
<space><crlf>
<space><crlf>
...

or

Code: Select all

<crlf>
<crlf>
<crlf>
...

and other (there's a 3rd character which some forum engines interpret as &nbsp;&nbsp;&nbsp;; it looks like ">>" and acts like a tab " ") - with possible variations like ">><crlf>", "<crlf><crlf>", "<space>>>" and so on.
StripHTML could deal with this like:
-ignore a maximum of 2 repeated characters in a line, whatever they are.
-ignore a maximum of 2 repeated characters in a sequence of 2 (empty) lines, whatever they are.
-reduce any number of repeated characters greater than 3 to match that maximum of 2 characters, inline or in sequence.
Then, the above sequence will look like:

Code: Select all

<space><crlf>
<space><crlf>

or

Code: Select all

<crlf>
<crlf>

Clean and readable, pleasant output.

(see the file attachment URLdecoding_Post_Samples.zip for a better idea, as Forum cannot reproduce properly what I'm trying to describe)
Thank you!