Tags. Hundreds of thousands of tags.

[oldtaku]

I know the new tags feature seems pretty snappy... but I was curious about something. I have ~100,000 picture files that I already have tags for - each file has 10-50 tags (if anyone cares, its danbooru pics, which come with tags already created, and my personal photos). I found the tag.dat file by tagging something manually, and it looks like the format is:

D:\news\lugaru-windows-demo.zip|0|demo, lugaru, windows||||

Now it seems to me that trying to add (with a script) 10-50 tags per file for about 100,000 files this way would just kill Xyplorer dead on load and I shouldn't even bother writing the script. Would you agree, or do you think it might actually work?

This isn't a criticism of Xyplorer, by the way; this is clearly an abnormal usage case that I'm curious about now that the possibility is open. I can already search them outside of Xyplorer, but obviously it would be nicer inside.

[SkyFrontier]

Parameters out of personal experience:

-Select Every Xth File, which requires an update* to properly work, deals with a max of 425 (447 on older versions like 9.40.0007) files. Took a time by trial-and-error to guess those values. I'd start with values near those, in your case.
-Catalog Maker, v1.0 won't work if the total of files exceeds 3047. A bug I can't fix, I'm afraid.
---
*This is the fix:
in the "_AddFile" section, the 3rd line must be:

Code: Select all

  $itemlist = $itemlist.getinfo("SelectedItemsNames", "|")."|";

[admin]

Parameters out of personal experience:

-Select Every Xth File, which requires an update* to properly work, deals with a max of 425 (447 on older versions like 9.40.0007) files. Took a time by trial-and-error to guess those values. I'd start with values near those, in your case.
-Catalog Maker, v1.0 won't work if the total of files exceeds 3047. A bug I can't fix, I'm afraid.
---
*This is the fix:
in the "_AddFile" section, the 3rd line must be:

Code: Select all

  $itemlist = $itemlist.getinfo("SelectedItemsNames", "|")."|";

These are issues because of bad scripting. This has nothing to do with the tagging feature of XYplorer.

[admin]

I know the new tags feature seems pretty snappy... but I was curious about something. I have ~100,000 picture files that I already have tags for - each file has 10-50 tags (if anyone cares, its danbooru pics, which come with tags already created, and my personal photos). I found the tag.dat file by tagging something manually, and it looks like the format is:

D:\news\lugaru-windows-demo.zip|0|demo, lugaru, windows||||

Now it seems to me that trying to add (with a script) 10-50 tags per file for about 100,000 files this way would just kill Xyplorer dead on load and I shouldn't even bother writing the script. Would you agree, or do you think it might actually work?

This isn't a criticism of Xyplorer, by the way; this is clearly an abnormal usage case that I'm curious about now that the possibility is open. I can already search them outside of Xyplorer, but obviously it would be nicer inside.

Yes, the format of the tag.da line is correct.

But you don't need it. You can easily make a test from inside XYplorer. Get your 100,000 pics into a search results list by making an unfiltered search, and apply any number of dummy tags to them. You will surely note some slowdown with this those numbers at startup and when browsing large folders with many tagged files. But it should be tolerable.

[Pagat]

These are issues because of bad scripting. This has nothing to do with the tagging feature of XYplorer.

I just wanted to say this. Since i wrote the "Select every xth file"-script you have to blame me and not XYplorer for the bad performance. I wrote this in a hurry and we had no conditional logic back then which wasn't helpful either. I'm sure with todays scripting possibilities and a well thought out script you won't have any performance issues.

[admin]

These are issues because of bad scripting. This has nothing to do with the tagging feature of XYplorer.

I just wanted to say this. Since i wrote the "Select every xth file"-script you have to blame me and not XYplorer for the bad performance. I wrote this in a hurry and we had no conditional logic back then which wasn't helpful either. I'm sure with todays scripting possibilities and a well thought out script you won't have any performance issues.

True, not necessariily bad scripting because of the author's lacking skills or effort but also because these scripts are from a state of the scripting feature where it could not do much yet.

Our friend SkyFrontier just likes to present totally out-of-date scripts from pre-historic times as if they were the state of the art. He does it again and again. Simply ignore him.

[admin]

@oldtaku

I've just been playing with 100,000 tagged files, and tag.dat is now 15 MB fat.
Startup time increased from 0.5 to 1.8 seconds. (XP SP3, Athlon 64 X2)
Browsing feels as snappy as ever.
Could be worse.

Of course, it all depends on the environment. Everybody has to find out for himself where his system hits the wall.

[oldtaku]

Hi All,

Thank you for your help. The last info in particular means it's worth trying out, because I don't mind a 2 second startup time. I just leave Xyplorer open all the time anyhow. And I have 12 gigs of RAM.

So I will try to script up an importer this week and report on how it turned out. Worst comes to worst I just nuke the tags.dat file.

For the fun of it I'll do process memory usage in the before and after cases and get an exact count of tags and unique tags.

[admin]

Hi All,

Thank you for your help. The last info in particular means it's worth trying out, because I don't mind a 2 second startup time. I just leave Xyplorer open all the time anyhow. And I have 12 gigs of RAM.

So I will try to script up an importer this week and report on how it turned out. Worst comes to worst I just nuke the tags.dat file.

For the fun of it I'll do process memory usage in the before and after cases and get an exact count of tags and unique tags.

FYI, I plan to add saving and loading of tag*.dat files similar to how it is already done with catalog*.dat files. So you can organize your tagging by project and so reduce the amount of tags in memory at a particular time.