Import descript.ion files from Total Commander
Import descript.ion files from Total Commander
If anyone wants to try it...
Beware: Dupes aren't handled so it's better to start off with a clean (no user defined tags) tag.dat database!
Please report back if everything is working for you. I've never worked with TC and only did a few comments for files from it, so my descript.ion files may not show all possible cases...
It should be quite fast, because of using a search, instead of a folderreport().
Changelog (new version):
v0.2 - Fixes tag.dat changes Old version (not recommended):
Beware: Dupes aren't handled so it's better to start off with a clean (no user defined tags) tag.dat database!
Please report back if everything is working for you. I've never worked with TC and only did a few comments for files from it, so my descript.ion files may not show all possible cases...
It should be quite fast, because of using a search, instead of a folderreport().
Changelog (new version):
v0.2 - Fixes tag.dat changes Old version (not recommended):
One of my scripts helped you out? Please donate via Paypal
Re: Import descript.ion files from Total Commander
Hi highend
I just saw then checked your script.
I don't know whether tag.dat format has changed since 2013 or whatelse, but I did not succeed to work with it.
Even with a single folder (without subfolders) containing only one file with descript.ion (ie folder\descript.ion contains only one line), nothing happens.
It seems that tag.dat is changed after running the script, but when I re-open XY, tag.dat seems restored to its previous state, but I am not sure.
An other possible issue is that tag.dat could must be sorted, and your script does not sort tag.dat. Here again, I am not sure.
A last question : where can I find any knowledge about tag.dat format ?
Thanks in advance.
JJK
I just saw then checked your script.
I don't know whether tag.dat format has changed since 2013 or whatelse, but I did not succeed to work with it.
Even with a single folder (without subfolders) containing only one file with descript.ion (ie folder\descript.ion contains only one line), nothing happens.
It seems that tag.dat is changed after running the script, but when I re-open XY, tag.dat seems restored to its previous state, but I am not sure.
An other possible issue is that tag.dat could must be sorted, and your script does not sort tag.dat. Here again, I am not sure.
A last question : where can I find any knowledge about tag.dat format ?
Thanks in advance.
JJK
Re: Import descript.ion files from Total Commander
Hi jjk,
This code is 15 months old and it was more "a prove of concept" (because Don asked me if I could write it). I haven't touched it since.
I'll take a look at it over the next few days (busy atm), ok?
Can you attach one of your descript.ion files (within a .zip container) to your (next) posting? Don't have TC installed here...
Knowledge about the tag.dat format...
Em... I don't work with tags, so I'm not into it atm. I've asked that Don when I wrote it but I don't have that PM anymore so somebody else should answer that...
Regards,
Highend
This code is 15 months old and it was more "a prove of concept" (because Don asked me if I could write it). I haven't touched it since.
I'll take a look at it over the next few days (busy atm), ok?
Can you attach one of your descript.ion files (within a .zip container) to your (next) posting? Don't have TC installed here...
Knowledge about the tag.dat format...
Em... I don't work with tags, so I'm not into it atm. I've asked that Don when I wrote it but I don't have that PM anymore so somebody else should answer that...
Regards,
Highend
One of my scripts helped you out? Please donate via Paypal
Re: Import descript.ion files from Total Commander
Thanks highend for your quick answer. Don't worry, I'll wait some days , I just dicovered your script written in january 2013, so...
About new tag.dat format, I found it in xyplorer.chm :
Your code was about XY 11.9. Now comment is at the end of the line 14, and we have 5 extra fields in this line before the comment field.
And now the Data: section of the tag.dat file must be sorted.
I noticed another issue : your script adds new comment lines at the end of the tag.dat file, but without the <crlf> separator.
You'll find an attached file for test.
Anyway, cheers, and I wait quietly your update if you can do it.
TIA
About new tag.dat format, I found it in xyplorer.chm :
Code: Select all
CURRENT VERSION: File Format of the XYplorer Tag Database (version 3 -- from 13.80 onwards)
Line 1: Header and version info: XYplorer File Tags v3
Line 2: (empty)
Line 3: Labels:
Line 4: Label schemes, separated by ";". Each scheme: Name|TextColor|BackColor. Colors are in RRGGBB.
Line 5: (empty)
Line 6: Extra Tags:
Line 7-11: Extra Tag Definitions: Caption|TypeID|0|0||||| . Some fields are reserved for future use.
Line 12: (empty)
Line 13: Data:
Line 14: Tagged item #1: Full filename|LabelID|Tags|Extra1|Extra2|Extra3|Extra4|Extra5|Comment
Line 15: Tagged item #2: ... etc. ...
In case you edit tag.dat manually...
Sort Order. The tagged item lines must be in binary ascending sort order (A < B ... < a < b ...). If you manually edit tag.dat and are unsure about the correctness of the sort order, you can have the data sorted by using "Database Clean Up..." in Configuration | Tags.
Capitalization matters. You must care for the right capitalization of the file names. Reason: For better performance, the comparisons are done by byte, not by character.
And now the Data: section of the tag.dat file must be sorted.
I noticed another issue : your script adds new comment lines at the end of the tag.dat file, but without the <crlf> separator.
You'll find an attached file for test.
Anyway, cheers, and I wait quietly your update if you can do it.
TIA
- Attachments
-
- test for import description files into comments.zip
- (1.27 KiB) Downloaded 199 times
Re: Import descript.ion files from Total Commander
Ok, latest version is in the starting post.
It wasn't too much work.
Just changed the structure a bit to support the tag.dat changes.
The new version get's the leading part (up to "Data:" from the current tag.dat file) and writes all imported items in this Data section.
I fixed a small bug (comments were resorted but I think they should retain their original order).
I tested it with your test file and it worked flawlessly.
If there is anything more to change, feel free...
It wasn't too much work.
Just changed the structure a bit to support the tag.dat changes.
The new version get's the leading part (up to "Data:" from the current tag.dat file) and writes all imported items in this Data section.
I fixed a small bug (comments were resorted but I think they should retain their original order).
I tested it with your test file and it worked flawlessly.
If there is anything more to change, feel free...
One of my scripts helped you out? Please donate via Paypal
Re: Import descript.ion files from Total Commander
Many thanks, highend
I just tested. I meet some issues.
When I run script NOT by step, the script reads each descript.ion file, but I am not asked (by $saveOK = confirm...) to write tag.dat and finally XY freezes.
When I run script step by step, all works until I am asked to write tag.dat, but the next line does not seem to be resolved correctly. Because XY freezes here.
Also, can you explain a bit the main regexp used in the script. They are difficult for me.
TIA
I just tested. I meet some issues.
When I run script NOT by step, the script reads each descript.ion file, but I am not asked (by $saveOK = confirm...) to write tag.dat and finally XY freezes.
When I run script step by step, all works until I am asked to write tag.dat, but the next line
Code: Select all
$curTagFileLeading = regexmatches(readfile("<xydata>\tag.dat"), ".*?[\s\S]*Data:\r\n");
Also, can you explain a bit the main regexp used in the script. They are difficult for me.
TIA
Re: Import descript.ion files from Total Commander
Mh, I haven't changed "really important" parts of the routine.
I just took your .zip file and extracted it again. Made 10 copies. Ran the script over their root folder.
No issues.
To see if it fails on the first part (before $saveOK = confirm...) just add
right before that line.
I see no reason why XY could freeze on
$curTagFileLeading = regexmatches(readfile("<xydata>\tag.dat"), ".*?[\s\S]*Data:\r\n");
It just reads the current tag.dat file and gets the content from the beginning up to (and including)
"Data:" with the following linebreak. On a fresh tag.dat file this would be line 1 - 13.
Which regexp is difficult for you?
.*?[\s\S]*Data:\r\n
Get everything from the beginning including all line breaks up to Data: and the following line break.
Anything else?
I just took your .zip file and extracted it again. Made 10 copies. Ran the script over their root folder.
No issues.
To see if it fails on the first part (before $saveOK = confirm...) just add
Code: Select all
text $newXYTags;
I see no reason why XY could freeze on
$curTagFileLeading = regexmatches(readfile("<xydata>\tag.dat"), ".*?[\s\S]*Data:\r\n");
It just reads the current tag.dat file and gets the content from the beginning up to (and including)
"Data:" with the following linebreak. On a fresh tag.dat file this would be line 1 - 13.
Which regexp is difficult for you?
.*?[\s\S]*Data:\r\n
Get everything from the beginning including all line breaks up to Data: and the following line break.
Anything else?
One of my scripts helped you out? Please donate via Paypal
Re: Import descript.ion files from Total Commander
Probably regexmatches(readfile is limited in file size.
My tag.dat has 135 KB, that explains probably that xy freezes at that line.
With a fresh tag.dat, all is Ok.
Now, I will continue my tests.
BTW, is there a mean to workaround that limit ?
And are there any other users ?
Anyway, thanks for all your work and your patience.
My tag.dat has 135 KB, that explains probably that xy freezes at that line.
With a fresh tag.dat, all is Ok.
Now, I will continue my tests.
BTW, is there a mean to workaround that limit ?
And are there any other users ?
Anyway, thanks for all your work and your patience.
Re: Import descript.ion files from Total Commander
It's not readfile() but regexmatches() which makes XY freeze.
I see no real workaround apart from
a: use only a fresh tag.dat file when you start the script
b: use an external regex tool (like rxrepl.exe, which does it in a blink of an eye with a 200kb tag.dat file)
I see no real workaround apart from
a: use only a fresh tag.dat file when you start the script
b: use an external regex tool (like rxrepl.exe, which does it in a blink of an eye with a 200kb tag.dat file)
One of my scripts helped you out? Please donate via Paypal
Re: Import descript.ion files from Total Commander
Thank you very much
JJK
JJK
Re: Import descript.ion files from Total Commander
Ooof.. that regexmatches is quite the costly call - on my measly 48 item tag.dat that call alone takes over a minute.
I decided to make a bit of an unscientific benchmark (attached), to compare the following methods:I ran it over 1000 iterations, which took a little over 4 hours, and here's the results:
To replace this particular call I'm partial to the GetToken(GetTokenIndex) method, but it might not scale as well as SubStr(StrPos) for larger tag.dats.
In general I think I'm going to avoid regexmatches whenever possible.
I decided to make a bit of an unscientific benchmark (attached), to compare the following methods:
Code: Select all
Given:
Global $G_TAGS = ReadFile("<xytagdat>");
// RegexMatches
$temp = regexmatches($G_TAGS, ".*?[\s\S]*Data:\r\n");
// RegexReplace
$temp = RegexReplace($G_TAGS, '([\s\S]*[\r\n]+Data:[\r\n]+)[\s\S]*', '$1');
// GetTokenIndex within GetToken
$temp = GetToken($G_TAGS, GetTokenIndex('Data:', $G_TAGS, "<crlf>", 'i'), "<crlf>",, 1) . "<crlf>";
// StrPos within SubStr
$needle = "<crlf>Data:<crlf>";
$nlen = StrLen($needle);
$temp = SubStr($G_TAGS, 0, StrPos($G_TAGS, $needle) + $nlen);
Code: Select all
Test Count Average (ms) Median (ms) Min (ms) Max (ms)
RegexMatches 254 60952.87 60902.50 60325.00 64163.00
RegexReplace 264 1.07 0.00 0.00 16.00
GetToken(GetTokenIndex) 232 2.51 0.00 0.00 16.00
SubStr(StrPos) 250 1.15 0.00 0.00 16.00
To replace this particular call I'm partial to the GetToken(GetTokenIndex) method, but it might not scale as well as SubStr(StrPos) for larger tag.dats.
In general I think I'm going to avoid regexmatches whenever possible.
- Attachments
-
- RegexMatches-Benchmarking.xys
- (2.15 KiB) Downloaded 285 times
-
- Posts: 183
- Joined: 15 Nov 2012 22:48
- Contact:
Re: Import descript.ion files from Total Commander
Digging this up…
1. When XYplorer is set to not show hidden files this script find nothing. It would be better to temporary switch showing hidden ON.
2. Also with newest TC9beta it doesn't import anything.
It finds "something' but as with beta TC changed its descriptions to support unicode characters, script fails to see them properly.
It is sad because #1 could was being solved by remembering to switch view mode before running script.With older TCs it worked very well.
#2 is not I can fix :(
1. When XYplorer is set to not show hidden files this script find nothing. It would be better to temporary switch showing hidden ON.
2. Also with newest TC9beta it doesn't import anything.
It finds "something' but as with beta TC changed its descriptions to support unicode characters, script fails to see them properly.
It is sad because #1 could was being solved by remembering to switch view mode before running script.With older TCs it worked very well.
#2 is not I can fix :(
Re: Import descript.ion files from Total Commander
Attach a zip file that contains:
- The "descript.ion" file that you've tried to import
- Your "tag.dat" file from XYplorer after you've used the script
- The "descript.ion" file that you've tried to import
- Your "tag.dat" file from XYplorer after you've used the script
One of my scripts helped you out? Please donate via Paypal