Page 1 of 1

Off-topic : Desk Top Search vs. Databases

Posted: 27 Nov 2011 17:15
by xp10
As we know, Outliners have got that advantage to present sort of a structure, from what we can browse than if we do not want to search every time: There's subjects a, b, c, d, with their sub-subjects aa, ab, ac, etc., and ba, bb, bc, etc., etc.: If we need specific info, we'll search, but often, we just sit and have a look at the given structure (in order to get ideas, in order to change that structure, to add something into it, or whatever).

Common databases generally do not offer that leisure, every look into their content must be a (possibly stored) search, or you go from one record to another, flat view - it goes without saying that for any serious search, most databases are much more suitable than most outliners.

But there are UltraRecall, MyInfo and other applics that allow for searching by attributes; they're called "column outliners" (in UR, those attributes ain't in columns yet, but this will be introduced, and they can be entered and searched for, although in a less handy way), and sometimes, even "sorted views" are possible, but only as alternative to the sorted tree, i.e. the normal outline is not stored somewhere, then temporarily replaced by an alternative tree, sorted by some attributes, but it's a special window, with flat view of all items, at best - I prefer a database, then (and we're speaking of tens of k of records to be searched, and outliners weren't meant for this...).

Some will know askSam, a venerable database that was state of the art in the DOS years, but that had not been developed in a very smart way any further, then went almost buried, and now they promise to deliver some modern product. I mention AS here because it's got ONE very smart feature (even if poorly implemented: no storage, mandatory mouse use for settings, very slow if many k's of records): You can build up a virtual tree by the content of 2 or 3 record fields at any time, and since these fields can be freely chosen, the program will build up alternative trees, alternative views, i.e. exactly the thing current outliners cannot do. This is very clever indeed, and if they realize a crash-free version of that program, it could become a winner again.

Of course, AS offers elaborate searches (and here, also stored ones), and it offers a "table" of search results, with context of the results, or with display of some field contents of the hit records, letting you chose and arrange the order of these fields. That's very good, but, for a database, not exceptional.

Now for desktop search (dts), and that's why I've created this thread, to collect opinions, insights.

In fact, a lot of people in the web assert that nowadays, databases ain't appropriate anymore but for very specific tasks; they say dts will do almost all what databases did, and presentation of the results, in "tables", with context, is as good as in many databases, and instead of having many k of records in a proprietary format, you'll have the same number of .txt or .xml files, and it's precisely the arrival of that .xml formats that seem to have triggered this new data storage philosophy.

dts problem one

In fact, Boolean searches and all that is very well available, but then, not many dts progs will give you your hits in context (and then, in a Boolean search, WHAT's the context? but you could designate the very first line of the hit file to be displayed in the "results table", but no such dts program offers that to my knowledge - while AS does).

dts problem two

And then, you'd like to search for some number ranges, not only for straight hits: not only for "Paris", but for postal areas 75001 to 75009 e.g., or for your customers / prospects within areas 12400 and 17000, and (in the case of customers) with a turnover between x and y dollars or whatever; again, AS offers such searches, natively, whilst not a single dts program of my knowledge offers such possibility up-front. (I've been one of the severest critics of AS, this thread is not meant to tout that product, it just serves me to explain things, and I'm rather searching for NOT having to use buggy AS in my business.)

In AS, and within .txt and .xml files, such "figure" data would be stored in the form
$xy:123*
or something, and thus, searching for records / files where $xy is comprised between figure x and figure y means that for every such "field", the content is to be fetched, then to be evaluated, then to be decided upon, "yes or no" (fulfills the condition or not), i.e. a SUB-ROUTINE must process these figures, no simple evaluation "string a is equal/not equal to string b" here.

To my knowledge, no current dts program offers such functionality (whilst AS evidently has got such an inbuilt feature within its search routine, as has every other, "real" database).

In theory, you could overcome these dts limitations by using regular expressions: Your search would not be within a mask, "content of field abc between x and y", but in the search line something like
$xy:\[1]\[4-8]\?\?\?
meaning (for me, this being not a correct regex example) every number comprised between 14,000 and 18,999 would here be acceptable;

as you can see, this is not practical if you've got many such amounts/numbers to search for in a same search, even with macros, and it's not possible to search this way for numbers, in our example, between 14,500 and 18,200 e.g., since that would mean to find a regex stipulating that:

- at the third position, all digits are acceptable
- but not if position 2 digit is 4: then (at that position 3) only digits 5 to 9 are acceptable
- and not if position 2 digit is 8: then (again at that position 3) only digits 0 to 2 are acceptable

Which is to say that while in a database, a simple subroutine will do all this without any problem, in a very easy way, you cannot reach out for similar results - and without even attaining them, within a dts engine, but by atrociously convoluted regex deployment.

As for amounts, the simple way, "between 14 and 18 k" would suffice, and thus that would be doable regex-wise, whilst the postal area code example would not work that way, but then, in most cases, unwanted hits would not be too prominent. So this COULD be done, but does anybody do it in practice? Is anybody using a dts engine to search tens of k of .txt or .xml files - or did I overlook the better dts search engines that indeed do have inbuilt sub-routines for more flexible data searching, thus making it possible to use independent files instead of data records?

I think those ideas can be buried, no? And back to MS Access...

But I think it's a passionate subject to think over.

Re: Off-topic : Desk Top Search vs. Databases

Posted: 13 Jan 2012 02:44
by xp10
My mind had been occupied by other subjects in the meanwhile, but it occurs to me now that I didn't think far enough here:

Of course, the standard search tools I know (and tried) do NOT offer any computation component by which such expressions like "p284" could be compared against "price between 180 and 350", etc., hence my idea with regex.

But then, as soon as you replace the search tool with a script language, be it AutoIt processing data files in text form, be it within some of those "programmable editors" working on "their own" files, you'll get what you need, i.e. text string functions for extraction of those numbers, 284 in our example, and variables, e.g. pcurr, pmin, pmax, and comparing syntax, if pcurr (284) >= pmin (180, yes) and pcurr <= pmax (350, yes), then proceed further with the textline in question, else abort it, and you can do this with every one of thousands of textlines in ONE file, no need to have thousands of files, linefeeds, etc. could easily be replaced by a special char.

This applies to non- or semi-professional developers like me who don't like to learn VBA (because they find it as terrible as I do, perhaps, or because they think that MS isn't the future; and who aren't fond of FileMaker either), but who ain't afraid of some programming efforts otherwise. (If they are, they should look again at AS, considering that there's a "search line" into which they could enter multiple search terms in standardized encoded form, e.g. by macros - so there isn't the need to go thru multiple menus / dialogs, even for sophisticated search term combinations - those codings cannot be found in the help file but in the forum archives only).

Finally, there are some MS Excel add-ons for better text handling if you prefer that calc prog because of your preference for numbers.

So, there are some good solutions indeed for everyone needing to handle and search data incl. number comparison - but it's astonishing indeed that none of the standard search tools we all know is sophisticated enough to include number comparisons into its offerings, especially since all it would need were some dozens of additional code.

I don't want to hijack this forum for off-topic subjects, but then, file management and data management overlap, here and there, and my first post above didn't find the right answers yet to my own questions. If there are some more solutions then those mentioned here, I'd be glad to hear about them (I left out MySQL/etc. with a front end and heavy programming in C), especially in case I might have missed a more complete search tool than those we all know.

People who just need some data, with real good searching, will not complain about me sharing this link here,

http://www.mostiwant.com/blog/radiotrac ... -softcity/

I suppose, since for several thousand records (and without pictures), it'll be fine.