Import descript.ion files from Total Commander

Discuss and share scripts and script files...
Post Reply
highend
Posts: 9517
Joined: 06 Feb 2011 00:33

Import descript.ion files from Total Commander

Post by highend »

If anyone wants to try it...

Beware: Dupes aren't handled so it's better to start off with a clean (no user defined tags) tag.dat database!

Please report back if everything is working for you. I've never worked with TC and only did a few comments for files from it, so my descript.ion files may not show all possible cases...

It should be quite fast, because of using a search, instead of a folderreport().

Changelog (new version):
v0.2 - Fixes tag.dat changes
ImportTCComments_v0.2.zip
(1.93 KiB) Downloaded 120 times
Old version (not recommended):
ImportTCComments.zip
(1.79 KiB) Downloaded 136 times
One of my scripts helped you out? Please donate via Paypal or paypal_donate (at) stdmail (dot) de

jjk
Posts: 202
Joined: 10 Oct 2007 20:11
Location: Paris

Re: Import descript.ion files from Total Commander

Post by jjk »

Hi highend

I just saw then checked your script.
I don't know whether tag.dat format has changed since 2013 or whatelse, but I did not succeed to work with it.
Even with a single folder (without subfolders) containing only one file with descript.ion (ie folder\descript.ion contains only one line), nothing happens.

It seems that tag.dat is changed after running the script, but when I re-open XY, tag.dat seems restored to its previous state, but I am not sure.
An other possible issue is that tag.dat could must be sorted, and your script does not sort tag.dat. Here again, I am not sure.

A last question : where can I find any knowledge about tag.dat format ?

Thanks in advance.

JJK

highend
Posts: 9517
Joined: 06 Feb 2011 00:33

Re: Import descript.ion files from Total Commander

Post by highend »

Hi jjk,

This code is 15 months old and it was more "a prove of concept" (because Don asked me if I could write it). I haven't touched it since.
I'll take a look at it over the next few days (busy atm), ok?

Can you attach one of your descript.ion files (within a .zip container) to your (next) posting? Don't have TC installed here...

Knowledge about the tag.dat format...

Em... I don't work with tags, so I'm not into it atm. I've asked that Don when I wrote it but I don't have that PM anymore so somebody else should answer that...

Regards,
Highend
One of my scripts helped you out? Please donate via Paypal or paypal_donate (at) stdmail (dot) de

jjk
Posts: 202
Joined: 10 Oct 2007 20:11
Location: Paris

Re: Import descript.ion files from Total Commander

Post by jjk »

Thanks highend for your quick answer. Don't worry, I'll wait some days :) , I just dicovered your script written in january 2013, so...
About new tag.dat format, I found it in xyplorer.chm :

Code: Select all

CURRENT VERSION: File Format of the XYplorer Tag Database (version 3 -- from 13.80 onwards)
Line 1: Header and version info: XYplorer File Tags v3
Line 2: (empty)
Line 3: Labels:
Line 4: Label schemes, separated by ";". Each scheme: Name|TextColor|BackColor. Colors are in RRGGBB.
Line 5: (empty)
Line 6: Extra Tags:
Line 7-11: Extra Tag Definitions: Caption|TypeID|0|0||||| . Some fields are reserved for future use.
Line 12: (empty)
Line 13: Data:
Line 14: Tagged item #1: Full filename|LabelID|Tags|Extra1|Extra2|Extra3|Extra4|Extra5|Comment
Line 15: Tagged item #2: ... etc. ...

In case you edit tag.dat manually...
Sort Order. The tagged item lines must be in binary ascending sort order (A < B ... < a  < b ...). If you manually edit tag.dat and are unsure about the correctness of the sort order, you can have the data sorted by using "Database Clean Up..." in Configuration | Tags.
Capitalization matters. You must care for the right capitalization of the file names. Reason: For better performance, the comparisons are done by byte, not by character.
Your code was about XY 11.9. Now comment is at the end of the line 14, and we have 5 extra fields in this line before the comment field.
And now the Data: section of the tag.dat file must be sorted.
I noticed another issue : your script adds new comment lines at the end of the tag.dat file, but without the <crlf> separator.
You'll find an attached file for test.

Anyway, cheers, and I wait quietly your update if you can do it.

TIA
Attachments
test for import description files into comments.zip
(1.27 KiB) Downloaded 113 times

highend
Posts: 9517
Joined: 06 Feb 2011 00:33

Re: Import descript.ion files from Total Commander

Post by highend »

Ok, latest version is in the starting post.

It wasn't too much work.

Just changed the structure a bit to support the tag.dat changes.

The new version get's the leading part (up to "Data:" from the current tag.dat file) and writes all imported items in this Data section.

I fixed a small bug (comments were resorted but I think they should retain their original order).

I tested it with your test file and it worked flawlessly.

If there is anything more to change, feel free...
One of my scripts helped you out? Please donate via Paypal or paypal_donate (at) stdmail (dot) de

jjk
Posts: 202
Joined: 10 Oct 2007 20:11
Location: Paris

Re: Import descript.ion files from Total Commander

Post by jjk »

Many thanks, highend

I just tested. I meet some issues.

When I run script NOT by step, the script reads each descript.ion file, but I am not asked (by $saveOK = confirm...) to write tag.dat and finally XY freezes.

When I run script step by step, all works until I am asked to write tag.dat, but the next line

Code: Select all

$curTagFileLeading = regexmatches(readfile("<xydata>\tag.dat"), ".*?[\s\S]*Data:\r\n");
does not seem to be resolved correctly. Because XY freezes here.

Also, can you explain a bit the main regexp used in the script. They are difficult for me.

TIA

highend
Posts: 9517
Joined: 06 Feb 2011 00:33

Re: Import descript.ion files from Total Commander

Post by highend »

Mh, I haven't changed "really important" parts of the routine.

I just took your .zip file and extracted it again. Made 10 copies. Ran the script over their root folder.
No issues.

To see if it fails on the first part (before $saveOK = confirm...) just add

Code: Select all

text $newXYTags;
right before that line.

I see no reason why XY could freeze on
$curTagFileLeading = regexmatches(readfile("<xydata>\tag.dat"), ".*?[\s\S]*Data:\r\n");

It just reads the current tag.dat file and gets the content from the beginning up to (and including)
"Data:" with the following linebreak. On a fresh tag.dat file this would be line 1 - 13.

Which regexp is difficult for you?
.*?[\s\S]*Data:\r\n
Get everything from the beginning including all line breaks up to Data: and the following line break.

Anything else?
One of my scripts helped you out? Please donate via Paypal or paypal_donate (at) stdmail (dot) de

jjk
Posts: 202
Joined: 10 Oct 2007 20:11
Location: Paris

Re: Import descript.ion files from Total Commander

Post by jjk »

Probably regexmatches(readfile is limited in file size.
My tag.dat has 135 KB, that explains probably that xy freezes at that line.
With a fresh tag.dat, all is Ok.
Now, I will continue my tests.

BTW, is there a mean to workaround that limit ?
And are there any other users ?

Anyway, thanks for all your work and your patience.

highend
Posts: 9517
Joined: 06 Feb 2011 00:33

Re: Import descript.ion files from Total Commander

Post by highend »

It's not readfile() but regexmatches() which makes XY freeze.

I see no real workaround apart from
a: use only a fresh tag.dat file when you start the script
b: use an external regex tool (like rxrepl.exe, which does it in a blink of an eye with a 200kb tag.dat file)
One of my scripts helped you out? Please donate via Paypal or paypal_donate (at) stdmail (dot) de

jjk
Posts: 202
Joined: 10 Oct 2007 20:11
Location: Paris

Re: Import descript.ion files from Total Commander

Post by jjk »

Thank you very much

JJK

TheQwerty
Posts: 4353
Joined: 03 Aug 2007 22:30

Re: Import descript.ion files from Total Commander

Post by TheQwerty »

Ooof.. that regexmatches is quite the costly call - on my measly 48 item tag.dat that call alone takes over a minute.

I decided to make a bit of an unscientific benchmark (attached), to compare the following methods:

Code: Select all

Given:
Global $G_TAGS = ReadFile("<xytagdat>");

// RegexMatches
$temp = regexmatches($G_TAGS, ".*?[\s\S]*Data:\r\n");

// RegexReplace
$temp = RegexReplace($G_TAGS, '([\s\S]*[\r\n]+Data:[\r\n]+)[\s\S]*', '$1');

// GetTokenIndex within GetToken
$temp = GetToken($G_TAGS, GetTokenIndex('Data:', $G_TAGS, "<crlf>", 'i'), "<crlf>",, 1) . "<crlf>";

// StrPos within SubStr
$needle = "<crlf>Data:<crlf>";
$nlen = StrLen($needle);
$temp = SubStr($G_TAGS, 0, StrPos($G_TAGS, $needle) + $nlen);
I ran it over 1000 iterations, which took a little over 4 hours, and here's the results:

Code: Select all

                   Test    Count  Average (ms)  Median (ms)  Min (ms)  Max (ms)
           RegexMatches      254      60952.87     60902.50  60325.00  64163.00
           RegexReplace      264          1.07         0.00      0.00     16.00
GetToken(GetTokenIndex)      232          2.51         0.00      0.00     16.00
         SubStr(StrPos)      250          1.15         0.00      0.00     16.00

To replace this particular call I'm partial to the GetToken(GetTokenIndex) method, but it might not scale as well as SubStr(StrPos) for larger tag.dats.

In general I think I'm going to avoid regexmatches whenever possible. 8)
Attachments
RegexMatches-Benchmarking.xys
(2.15 KiB) Downloaded 199 times

fenixproductions
Posts: 131
Joined: 15 Nov 2012 22:48
Contact:

Re: Import descript.ion files from Total Commander

Post by fenixproductions »

Digging this up…

1. When XYplorer is set to not show hidden files this script find nothing. It would be better to temporary switch showing hidden ON.

2. Also with newest TC9beta it doesn't import anything.
It finds "something' but as with beta TC changed its descriptions to support unicode characters, script fails to see them properly.

It is sad because #1 could was being solved by remembering to switch view mode before running script.With older TCs it worked very well.
#2 is not I can fix :(

highend
Posts: 9517
Joined: 06 Feb 2011 00:33

Re: Import descript.ion files from Total Commander

Post by highend »

Attach a zip file that contains:

- The "descript.ion" file that you've tried to import
- Your "tag.dat" file from XYplorer after you've used the script
One of my scripts helped you out? Please donate via Paypal or paypal_donate (at) stdmail (dot) de

Post Reply