Tag Backup - Rev. 1 / 2012/09/15

Discuss and share scripts and script files...
Marco
Posts: 2256
Joined: 27 Jun 2011 15:20

Tag Backup - Rev. 1 / 2012/09/15

Post by Marco » 12 Sep 2012 20:07

[This first post gets updated from time to time, particularly when a new revision is released. You are strongly advised to read this post carefully!]

Brand new proof-of-concept script that allows transferring tags (read: label, tags and comment) from XY database to selected items and viceversa, taking advantage of the Alternate Data Stream feature (aka forks) of the NTFS filesystem. The comments and syntax are clear enough too to give you a nice base to start from, in case you want to customize it.

General structure

The script is formed by five subscripts. Here is the purpose of each of them.

[*]Find Tagged Items: self explanatory. It finds all the tagged items inside a branch, i.e. within a folder and all its subfolders, and shows them in the List in a Find tab.
[*]Transfer Tags from XYplorer to Forks: for each of the items found with the search above, a fork named "$sig.dat" is added, which contains a dump of the tags. The fork is basically a text file in UTF-16LE containing in the first line the label name, in the second line all the tags, and from the third line down the comment.
-
[*]Find Items with Fork: the analogous of the first subscript. Things here a little trickier since there's no "direct" way to find items with a fork, so given a branch for every item the existence of a specific fork is checked via the exist() function. If an item has a fork then the item itself is tagged with a "$sig" tag. Finally a search is triggered to show all the items just tagged in this way, similarly to the first subscript.
[*]Transfer Tags from Forks to XYplorer: again, self explanatory. For each item in the List the fork is read and parsed, and the tag() function is applied accordingly to restore everything.
[*]Delete Fork from Items: does what it says. Deletes the "$sig.dat" from each file in the List.

Usage

When the script is executed you can see all the five routine, two on top and three below. This graphical grouping tells you that the two top subscripts are meant to work together (before a backup), like the three below (before or after a backup).
Typical scenario: you want to backup a whole folder and all its contents. So you browse to that folder and run the first two subscripts. Now every tagged item has its fork, and you can freely copy that folder on another NTFS partitions. The copies will contain the forks as well. In case you now want to remove the forks from the "original" files you simply need to run "Remove Fork from Items".
Another typical scenario: you need to restore those backedup items and the tags as well. In this case first restore the items as you would usually do, then go to the destination containing folder. Finally run in sequence the last three subscripts. That's it!

Tested on

* Windows 7 Ultimate SP1 x64 ITA

Requirements

* XYplorer 11.60.0004 or greater with
- scripting enabled
- permanent variables remembered across sessions
* a NTFS partition

Notes

* :!: :!: :!: I absolutely take no responsibility for whatever damage, direct or indirect, the usage of this script may cause. This code is provided as-is, use at your own risk. You are advised.
* Forks are supported only on NTFS partitions. When transferring a file with a fork from a NTFS partition to a FAT partition only the main stream of the file, i.e. "the file you actually see", is transferred, all the secondary streams (the forks) are not.
* What is $sig? $sig is the variable that contains the basename of the forks. To avoid (unlikely) collisions with other forks name or other tags you are suggested to pick a random name or some sort of hash. Random passwords from https://www.grc.com/passwords.htm are a great choice. Once you have found a suitable name (for example "iS6WykGOsrnq6oXSgESObdjgM8ZxNeqFcXWPvn3PAq6HFqVnaDau8EZpDTBs4be"), type in the Address Bar the following:

Code: Select all

perm $sig; $sig="iS6WykGOsrnq6oXSgESObdjgM8ZxNeqFcXWPvn3PAq6HFqVnaDau8EZpDTBs4be";
to initialize the $sig variable and give it a "iS6WykGOsrnq6oXSgESObdjgM8ZxNeqFcXWPvn3PAq6HFqVnaDau8EZpDTBs4be" value.
* Accessing a fork can be interpreted as a malicious behaviour by your antivirus or HIPS, and the fork itself may be considered dangerous by the mentioned softwares. Of course the forks written by this scripts are absolutely not harmful, since they consist of plain text.

Possible future updates

* -

Changelog

* Rev. 1 / 2012/09/15: the very first version.

 :D :D :D Special thanks to...

* admin/Don, for the underlying software, the scripting engine, at least two important updates to the functions used, and his corrections (in other words, everything... :biggrin:).

Code: Select all

"Find Tagged Items|:find"
 goto input ("Find Tagged Items","Find all the tagged items in the following folder and its subfolders (no trailing backslash)",<curpath>,s,,600,400)."\?:lbl:?* or tags:?* or cmt:?* /r";
 msg ("The List now shows all the items that can be processed during the next phase.<br>Please select the items that you want to be processed.<br>No processing will take place unless you actively select the next step from the script.",1);

"Transfer Tags from XYplorer to Forks|:fore"
 foreach ($item,<get selecteditemspathnames>,<crlf>) {
  $info = report ("{Label}<crlf>{Tags}<crlf>{Comment}",$item); //collects the tags of each selected item, placing a new line between them
  writefile ("$item:$sig.dat",$info,,"tu"); //writes the gathered info in the fork of each selected item, in UTF
 };
-
"Find Items with Fork|:find"
 $path = input ("Find Items with Fork","Find all the items having a fork in the following folder and its subfolders (no trailing backslash)",<curpath>,s,,600,400);
 $items = folderreport (items,r,$path,r,,);
 foreach ($item,$items,<crlf>) {
  if (exists ("$item:$sig.dat") != 0) {
   tag ($sig,$item,1,0) //add a $sig tag to each item having a fork
  };
 };
 goto "$path\?:tags:"$sig" /r"; //show every item having a fork
 msg ("The List now shows all the items that can be processed during the next phase.<br>Please select the items that you want to be processed.<br>No processing will take place unless you actively select the next step from the script.",1);

"Transfer Tags from Forks to XYplorer|:back"
 foreach($item,<get selecteditemspathnames>,<crlf>) {
  $info = readfile("$item:$sig.dat","t"); //read the info in the fork of each selected item, in UTF

  $label = gettoken($info,1,<crlf>); //puts the 1. line of the file in the label variable
  tag($label,$item,0); //attaches the content of the label variable as a label

  $tags = gettoken($info,2,<crlf>); //puts the 2. line of the file in the tags variable
  tag($tags,$item,1,1); //attaches the content of the tags variable as a tag/list of tags

  $comment = gettoken($info,3,<crlf>); //puts the 3. line of the file in the comment variable
  tag($comment,$item,2); //attaches the content of the comment variable as a comment
 };

"Delete Fork from Items|:del"
 foreach($item,<get selecteditemspathnames>,<crlf>) {
  deletefile "$item:$sig.dat"; //UNDOCUMENTED [http://www.xyplorer.com/xyfc/viewtopic.php?f=7&t=8587#p76625]: delete the fork
 };
Attachments
Tag Backup.xys
(2.43 KiB) Downloaded 292 times
Last edited by Marco on 15 Sep 2012 20:50, edited 6 times in total.
Tag Backup - SimpleUpdater - XYplorer Messenger - The Unofficial XYplorer Archive - Everything in XYplorer
Don sees all [cit. from viewtopic.php?p=124094#p124094]

admin
Site Admin
Posts: 46936
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Tag Backup 0.9

Post by admin » 13 Sep 2012 08:30

WOW. Impressive. Works great on XP. (Did not test yet on Win7, but I assume you did.) I should think about adding this to XY. It would solve the problem of taking the tags along in backups...

1. I'll think about the labels problem...

2. It works because comment line breaks are returned as "¶" characters. You don't see the difference in the Steps dialog, but you can see it in Variables dialog when you open the $info variable.

admin
Site Admin
Posts: 46936
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Tag Backup 0.9

Post by admin » 13 Sep 2012 10:11

This will work in the next beta version for Labels, Tags, and Comments:

Code: Select all

"Transfer Tags && Comment from XYplorer to ADS|:fore"
 foreach($item,<get selecteditemspathnames>,<crlf>) {
  $info = report("{Label}<crlf>{Tags}<crlf>{Comment}",$item); //collects the tags and the comment of the selected file, placing a <crlf> between them
  writefile($item:tag.dat,$info,,"tu"); //writes the gathered info in the tag.dat fork of the selected item, in UTF
 };

"Transfer Tags && Comment from ADS to XYplorer|:back"
 foreach($item,<get selecteditemspathnames>,<crlf>) {
  $info = readfile($item:tag.dat,"t"); //writes the info in the tag.dat fork of the selected item, in UTF

  $label = gettoken($info,1,<crlf>); //puts the 1. line of the file in the label variable
  tag($label,$item,0); //attaches the content of the label variable as a label

  $tags = gettoken($info,2,<crlf>); //puts the 2. line of the file in the tags variable
  tag($tags,$item,1,1); //attaches the content of the tags variable as a tag/list of tags

  $comment = gettoken($info,3,<crlf>); //puts the 3. line of the file in the comment variable
  tag($comment,$item,2); //attaches the content of the comment variable as a comment

  writefile($item:tag.dat,,,"b"); //clears the content of the tag.dat fork
 };
Note that I removed the lines where tags were removed. They were not necessary.

Marco
Posts: 2256
Joined: 27 Jun 2011 15:20

Re: Tag Backup 0.9

Post by Marco » 13 Sep 2012 11:07

admin wrote:WOW. Impressive. Works great on XP. (Did not test yet on Win7, but I assume you did.) I should think about adding this to XY. It would solve the problem of taking the tags along in backups...

1. I'll think about the labels problem...

2. It works because comment line breaks are returned as "¶" characters. You don't see the difference in the Steps dialog, but you can see it in Variables dialog when you open the $info variable.
Thank you! And yes, I tested it on 7.
Regarding the rest:
1. Wonderful!
2. Ahh, mistery solved then.
admin wrote:This will work in the next beta version for Labels, Tags, and Comments:

Code: Select all

Updated code
Note that I removed the lines where tags were removed. They were not necessary.
Ok, I wasn't totally sure that tag() always replaces the tag/label/comment. Good to know.

I'd like now to refine the code so that after transferring the tags/label/comment to the fork these data are cleared from the XY tag database, namely calling tag(,$item,[type]) with type=0,1,2 .
Now I'm asking the users: Is it an expected behaviour?
Also, for the viceversa, I currently don't delete the fork, but rather clear it's content. Don, would it be possible to tweak delete() so to act on forks too? :roll:
Tag Backup - SimpleUpdater - XYplorer Messenger - The Unofficial XYplorer Archive - Everything in XYplorer
Don sees all [cit. from viewtopic.php?p=124094#p124094]

admin
Site Admin
Posts: 46936
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Tag Backup 0.9

Post by admin » 13 Sep 2012 12:30

Marco wrote:Also, for the viceversa, I currently don't delete the fork, but rather clear it's content. Don, would it be possible to tweak delete() so to act on forks too? :roll:
It seems that it is not possible to delete individual forks, just the whole ADS (and even this is clumsy to do).

Marco
Posts: 2256
Joined: 27 Jun 2011 15:20

Re: Tag Backup 0.9

Post by Marco » 13 Sep 2012 13:43

admin wrote:It seems that it is not possible to delete individual forks, just the whole ADS (and even this is clumsy to do).
Strange, I found this page, http://www.flexhex.com/docs/articles/al ... eams.phtml , that says

Code: Select all

Deleting a Stream

The Win32 API function DeleteFile fully supports alternate streams, so deleting a stream is no more complex than deleting a file:

::DeleteFile("file.dat:alt");

As has been noted before, you can't delete the unnamed stream alone; deleting it also deletes all the alternate streams. 
Also, the utility that I regularly use for detecting and deleting ADSs, http://www.nirsoft.net/utils/alternate_ ... reams.html , supports deleting single forks.

Anyway, not an extremely big deal, but seeing that writefile supports ADS while delete doesn't breaks the simmetry :)
Tag Backup - SimpleUpdater - XYplorer Messenger - The Unofficial XYplorer Archive - Everything in XYplorer
Don sees all [cit. from viewtopic.php?p=124094#p124094]

admin
Site Admin
Posts: 46936
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Tag Backup 0.99

Post by admin » 13 Sep 2012 14:15

Confirmed, DeleteFile can delete single forks.

admin
Site Admin
Posts: 46936
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Tag Backup 0.99

Post by admin » 13 Sep 2012 14:47

BTW, if you want to preserve that dot in "tag.dat" you should quote this argument:

Code: Select all

"$item:tag.dat"
else you get ...:tagdat

admin
Site Admin
Posts: 46936
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Tag Backup 0.9

Post by admin » 13 Sep 2012 18:24

Marco wrote:I'd like now to refine the code so that after transferring the tags/label/comment to the fork these data are cleared from the XY tag database, namely calling tag(,$item,[type]) with type=0,1,2 .
Now I'm asking the users: Is it an expected behaviour?
Well, I would NOT expect them to be removed. For me you don't have to add anything. :)

Marco
Posts: 2256
Joined: 27 Jun 2011 15:20

Re: Tag Backup 0.9

Post by Marco » 13 Sep 2012 23:51

admin wrote:BTW, if you want to preserve that dot in "tag.dat" you should quote this argument:

Code: Select all

"$item:tag.dat"
else you get ...:tagdat
Oops you're totally right, I'll fix that. I also think tag.dat should be changed to something more unique to prevent name collisions (even if forks aren't that common), maybe with an hash.
admin wrote:
Marco wrote:I'd like now to refine the code so that after transferring the tags/label/comment to the fork these data are cleared from the XY tag database, namely calling tag(,$item,[type]) with type=0,1,2 .
Now I'm asking the users: Is it an expected behaviour?
Well, I would NOT expect them to be removed. For me you don't have to add anything. :)
I thought that leaving tags in XY and the fork can be confusing and could lead to "uncontrolled different evolution", alias tags could be modified from within XY interface while staying unmodified and getting not-up-to-date in the fork. Anyway, it would be just a line of code that can be commented in and out :)
Tag Backup - SimpleUpdater - XYplorer Messenger - The Unofficial XYplorer Archive - Everything in XYplorer
Don sees all [cit. from viewtopic.php?p=124094#p124094]

admin
Site Admin
Posts: 46936
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Tag Backup 0.99

Post by admin » 14 Sep 2012 08:13

I thought in a backup situation one would like to copy the tags from the sources to the ADS of the targets. The targets would not have tags anyway since XY does not copy the tags. So no need to remove any tags.

Marco
Posts: 2256
Joined: 27 Jun 2011 15:20

Re: Tag Backup 0.99

Post by Marco » 14 Sep 2012 20:52

admin wrote:I thought in a backup situation one would like to copy the tags from the sources to the ADS of the targets. The targets would not have tags anyway since XY does not copy the tags. So no need to remove any tags.
Ah gotcha, and flawless logic by the way.
Are you planning about including this routine into backupto()? Because if so I stop developing the script, else I need to sit down and figure out how I can efficiently scan all the backuped files to see if forks in destination need to be added.
Tag Backup - SimpleUpdater - XYplorer Messenger - The Unofficial XYplorer Archive - Everything in XYplorer
Don sees all [cit. from viewtopic.php?p=124094#p124094]

admin
Site Admin
Posts: 46936
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Tag Backup 0.99

Post by admin » 14 Sep 2012 21:03

Marco wrote:
admin wrote:I thought in a backup situation one would like to copy the tags from the sources to the ADS of the targets. The targets would not have tags anyway since XY does not copy the tags. So no need to remove any tags.
Ah gotcha, and flawless logic by the way.
Are you planning about including this routine into backupto()? Because if so I stop developing the script, else I need to sit down and figure out how I can efficiently scan all the backuped files to see if forks in destination need to be added.
Well, should I? This thread does not really get much response. I think your idea is totally great, but nobody is watching?

Marco
Posts: 2256
Joined: 27 Jun 2011 15:20

Re: Tag Backup 0.99

Post by Marco » 14 Sep 2012 21:22

admin wrote:Well, should I? This thread does not really get much response. I think your idea is totally great, but nobody is watching?
I'll contact two users here who showed interest in the possible development of such a feature and ask them to cooperate...
Tag Backup - SimpleUpdater - XYplorer Messenger - The Unofficial XYplorer Archive - Everything in XYplorer
Don sees all [cit. from viewtopic.php?p=124094#p124094]

admin
Site Admin
Posts: 46936
Joined: 22 May 2004 16:48
Location: Cologne, Win 8.1, Win 10
Contact:

Re: Tag Backup 0.99

Post by admin » 14 Sep 2012 21:24

Marco wrote:
admin wrote:Well, should I? This thread does not really get much response. I think your idea is totally great, but nobody is watching?
I'll contact two users here who showed interest in the possible development of such a feature and ask them to cooperate...
BTW, I added something for you:

Code: Select all

      ' UNDOCUMENTED
      ' 20120913: direct interface to API DeleteFile (can delete ADS Alternate Data Streams)
      ' syntax: deletefile file
      ' NOTE: no prompts, no recycler!
:mrgreen:

Post Reply