Manual Tag List Cleanup

[armsys]

Currently XY does not provide function to automatically clean up the Tag List.
After a while, the Tag List may contain non-existing tags.
I find a manual way to clean up the Tag List:
1. Extract all tagged items from tag.dat.
2. Paste into a text editor (EditPad Pro in my case).
3. Use the following regex to extract the tags
Search: ^.*?\|\d\|(.*?)\|.*$
Replace: \1
4. Delete all repeat items in your text editor.
5. Sort all iems.
6. Delete all tags in the Tag List (List Management | Tag List)
7. Copy the above items into the Tag List.

Don, is there any way to automate the afore-mentioned Tag List cleanup procedure?
Alternatively, can it be accomplished by XYscript?

[binocular222]

Config (F9) > Tags > Option > Database check

[armsys]

Config (F9) > Tags > Option > Database check

Hi binocular222,
Thanks for your help.
I apologize for not stating the issue clearly.
Database check can only remove the orphaned tags.
However, XY maintains a separate Tag List (Tools | List Management | Tag List...) independant of the database per your post.
The core issue
Even though the "Database check" has cleaned up the database, all the orphaned tags remain in the Tag List.
Hope it helps your understanding.

[Marco]

Config (F9) > Tags > Option > Database check

Hi binocular222,
Thanks for your help.
I apologize for not stating the issue clearly.
Database check can only remove the orphaned tags.
However, XY maintains a separate Tag List (Tools | List Management | Tag List...) independant of the database per your post.
The core issue
Even though the "Database check" has cleaned up the database, all the orphaned tags remain in the Tag List.
Hope it helps your understanding.

Why should Tag List be cleaned? It's a list of "default" tags defined by you (unless you tick the option to memorize in the list tags added manually during tagging) - each tag should exist there regardless of its actual use on files.

[armsys]

Why should Tag List be cleaned? It's a list of "default" tags defined by you (unless you tick the option to memorize in the list tags added manually during tagging) - each tag should exist there regardless of its actual use on files.

First of all, your observation is accurate.
I guess Don's intent is to allow users remember all previously defined tags even though some tags are no longer assigned to the files/folders. That makes sense.
It will soon become a huge problem: obsolete tags grow out of control.
What if there are 10,000 tags which you won't use anymore for the rest of your life?

[Marco]

It will soon become a huge problem: obsolete tags grow out of control.
What if there are 10,000 tags which you won't use anymore for the rest of your life?

I see - good point.
I can tell you, re the procedure in the 1st post, that it can be (partly, afaik) automated. In particular steps 6 and 7:
6: you cannot delete programmatically the tags in Tag List, but you can have its window opened with command #635
7: consequently, you can't have the "clean" list of tags pasted there, however you can put it in the clipboard via scripting and you can paste it in the window with CTRL-V or the mousy right-click.

Unless
you want to go the "low-level" way and edit the XYplorer.ini directly to automate steps 6 and 7 too.

[TheQwerty]

You can modify the tag list using the taglist SC.
Combine with InputSelect and you have a nice interface to select those you want to keep or delete.

The only part that is left is determining which tags are no longer being used. Probably the most accurate way to achieve this is currently to parse tags.dat, create a list of tags in use, and then filter the result of taglist to identify those not in use.


I took a quick stab at a script to do all this:

TagListCleaner.xys

Code: Select all

"Remove Orphaned Tags from Tag List"
	// Save and then read in tags.dat.
	#195; // File | Settings Special | Save Tags
	$tagsInUse = ReadFile("<xytagdat>");

	// Ensure we're parsing version 2 of tags.dat.
	// If Don changes the format this script may need changed as well.
	$header = GetToken($tagsInUse, 1, "<crlf>", 't');
	End $header UnLikeI 'XYplorer File Tags v2', 'This script is only known to work for version 2 of tags.dat.';
	Unset $header;

	// Jump to the meat of the tagging data by skipping the first two lines.
	$tagsInUse = GetToken($tagsInUse, 2, "<crlf>", 't', 2);

	// Extract the tags from each line.
	$tagsInUse = RegexReplace($tagsInUse, '^[^\|]*\|[^\|]*\|([^\|]*)\|.*?\r?\n?$', '$1,');
	
	// Clean up the list of tags in use.
	$tagsInUse = FormatList($tagsInUse, 'dents', ',');

	// Retrieve and clean up the tag list and count.
	$taglist = FormatList(TagList(), 'dents', ',');
	$old = GetToken($taglist, 'Count', ',');

	// Determine which tags in the tag list are not currently in use.
	$taglistUnused = FormatList($taglist, 'f', ',', "!$tagsInUse");
	Unset $tagsInUse, $taglist;

	// End if there are no unused tags.
	if ($taglistUnused == '') {
		Unset $taglistUnused, $old;
		Status 'The tag list does not contain unused tags.',, 'stop';
		End true;
	}

	// Ask the user to select which tags to remove from the taglist.
	$res = InputSelect("The following tags are in the tag list but not in use.<crlf 2>Select those you wish to remove from the tag list:", $taglistUnused, ',', 2);
	Unset $taglistUnused;

	// Remove the selected tags from the taglist.
	if ($res != '') {
		TagList($res, 2);
	}
	Unset $res;

	// Calculate how many tags were removed.
	$count = $old - GetToken(TagList(), 'Count', ',');
	Unset $old;

	// Determine and display an appropriate message.
	if ($count == 0) {
		$count = 'No tags';
	} elseif ($count == 1) {
		$count = 'One tag';
	} else {
		$count = "$count tags";
	}
	Status "$count removed from tag list.",, 'ready';
	Unset $count;

[/size]

[Enternal]

You can modify the tag list using the taglist SC.
Combine with InputSelect and you have a nice interface to select those you want to keep or delete.

The only part that is left is determining which tags are no longer being used. Probably the most accurate way to achieve this is currently to parse tags.dat, create a list of tags in use, and then filter the result of taglist to identify those not in use.


I took a quick stab at a script to do all this:

TagListCleaner.xys

Code: Select all

"Remove Orphaned Tags from Tag List"
	// Save and then read in tags.dat.
	#195; // File | Settings Special | Save Tags
	$tagsInUse = ReadFile("<xytagdat>");

	// Ensure we're parsing version 2 of tags.dat.
	// If Don changes the format this script may need changed as well.
	$header = GetToken($tagsInUse, 1, "<crlf>", 't');
	End $header UnLikeI 'XYplorer File Tags v2', 'This script is only known to work for version 2 of tags.dat.';
	Unset $header;

	// Jump to the meat of the tagging data by skipping the first two lines.
	$tagsInUse = GetToken($tagsInUse, 2, "<crlf>", 't', 2);

	// Extract the tags from each line.
	$tagsInUse = RegexReplace($tagsInUse, '^[^\|]*\|[^\|]*\|([^\|]*)\|.*?\r?\n?$', '$1,');
	
	// Clean up the list of tags in use.
	$tagsInUse = FormatList($tagsInUse, 'dents', ',');

	// Retrieve and clean up the tag list and count.
	$taglist = FormatList(TagList(), 'dents', ',');
	$old = GetToken($taglist, 'Count', ',');

	// Determine which tags in the tag list are not currently in use.
	$taglistUnused = FormatList($taglist, 'f', ',', "!$tagsInUse");
	Unset $tagsInUse, $taglist;

	// End if there are no unused tags.
	if ($taglistUnused == '') {
		Unset $taglistUnused, $old;
		Status 'The tag list does not contain unused tags.',, 'stop';
		End true;
	}

	// Ask the user to select which tags to remove from the taglist.
	$res = InputSelect("The following tags are in the tag list but not in use.<crlf 2>Select those you wish to remove from the tag list:", $taglistUnused, ',', 2);
	Unset $taglistUnused;

	// Remove the selected tags from the taglist.
	if ($res != '') {
		TagList($res, 2);
	}
	Unset $res;

	// Calculate how many tags were removed.
	$count = $old - GetToken(TagList(), 'Count', ',');
	Unset $old;

	// Determine and display an appropriate message.
	if ($count == 0) {
		$count = 'No tags';
	} elseif ($count == 1) {
		$count = 'One tag';
	} else {
		$count = "$count tags";
	}
	Status "$count removed from tag list.",, 'ready';
	Unset $count;

[/size]

Wow!

[armsys]

You can modify the tag list using the taglist SC.
Combine with InputSelect and you have a nice interface to select those you want to keep or delete.

The only part that is left is determining which tags are no longer being used. Probably the most accurate way to achieve this is currently to parse tags.dat, create a list of tags in use, and then filter the result of taglist to identify those not in use.

Hi TheQwerty,
Thanks for your moving heaven and earth effort to create a textbook script.
Thanks for spending hours in abstraction, design, coding, and debugging.
Your elegant script is so heavily documented that any XY rookies can understand with ease.
Is this a typical Deutsch coding style?
I didn't know XYScript supports regex. That's very impressive!
TheQwerty, you don't just provide us a solution but also, above all, a study tool to master XY scripting.
Thanks again.

[TheQwerty]

And there I thought I was just putting off doing real work with some scripting procrastination.

Your elegant script is so heavily documented that any XY rookies can understand with ease.
Is this a typical Deutsch coding style?

I've learned that it's better for everyone if I do my best to comment code I share as if I were explaining it to a rubber duck. Not that this is any indication of my opinion on anyone's intelligence here.

Accounting for the varying skill-levels of readers in this way hopefully provides enough guidance for those wanting to learn or modify the scripts and if I'm lucky it answers many questions before they are asked.

However, of more importance, it ensures that if and when I return to a script I don't have to spend quite as long trying to figure out what the heck past-me was doing. That guy already creates enough problems for me with all his procrastinating!


I wish I could say I was always so thorough documenting my code but alas the repositories at work would show otherwise.

[j_c_hallgren]

However, of more importance, it ensures that if and when I return to a script I don't have to spend quite as long trying to figure out what the heck past-me was doing. That guy already creates enough problems for me with all his procrastinating!

Back in my prior 'lifetime' when I was a COBOL programmer, my code was likely one of the most heavily documented in our shop and others who had to read it were always happy! I often had a comment line at least every 5-8 lines even though COBOL is intended to be easy reading but I've seen some code that was almost like assembler!

But as you said, I was most concerned about being able to quickly refresh MY thoughts if there was a problem at 4am and I had to fix something when my mind was in sleep/hibernate mode.

[armsys]

Back in my prior 'lifetime' when I was a COBOL programmer, my code was likely one of the most heavily documented in our shop and others who had to read it were always happy! I often had a comment line at least every 5-8 lines even though COBOL is intended to be easy reading but I've seen some code that was almost like assembler!

That may explain why you enjoy Sunshine State now. Next time I'll remember your 4AM saga.
3 other prolific contributors of heavily-documented XY scripts are klownboy, serendipity, and Stefan.

[j_c_hallgren]

That may explain why you enjoy Sunshine State now. Next time I'll remember your 4AM saga.

It was in FL that i worked at those COBOL jobs -- after early retirement, I'm now running the family summer business in MA.

BTW, it wasn't just the 4am thing -- it was also that it could easily be 2-5 years later as well that I had to fix an issue so while you may easily recall what you coded last month, you may not 5 yrs later! Given that mainframe COBOL code tends to be quite stable and sometimes doesn't change for years or decades, it's not unusual.

I have had to work on coding done by others that was written as long as 20 yrs prior so good code doesn't die quickly in those systems.