Off-topic : Desk Top Search vs. Databases
Posted: 27 Nov 2011 17:15
As we know, Outliners have got that advantage to present sort of a structure, from what we can browse than if we do not want to search every time: There's subjects a, b, c, d, with their sub-subjects aa, ab, ac, etc., and ba, bb, bc, etc., etc.: If we need specific info, we'll search, but often, we just sit and have a look at the given structure (in order to get ideas, in order to change that structure, to add something into it, or whatever).
Common databases generally do not offer that leisure, every look into their content must be a (possibly stored) search, or you go from one record to another, flat view - it goes without saying that for any serious search, most databases are much more suitable than most outliners.
But there are UltraRecall, MyInfo and other applics that allow for searching by attributes; they're called "column outliners" (in UR, those attributes ain't in columns yet, but this will be introduced, and they can be entered and searched for, although in a less handy way), and sometimes, even "sorted views" are possible, but only as alternative to the sorted tree, i.e. the normal outline is not stored somewhere, then temporarily replaced by an alternative tree, sorted by some attributes, but it's a special window, with flat view of all items, at best - I prefer a database, then (and we're speaking of tens of k of records to be searched, and outliners weren't meant for this...).
Some will know askSam, a venerable database that was state of the art in the DOS years, but that had not been developed in a very smart way any further, then went almost buried, and now they promise to deliver some modern product. I mention AS here because it's got ONE very smart feature (even if poorly implemented: no storage, mandatory mouse use for settings, very slow if many k's of records): You can build up a virtual tree by the content of 2 or 3 record fields at any time, and since these fields can be freely chosen, the program will build up alternative trees, alternative views, i.e. exactly the thing current outliners cannot do. This is very clever indeed, and if they realize a crash-free version of that program, it could become a winner again.
Of course, AS offers elaborate searches (and here, also stored ones), and it offers a "table" of search results, with context of the results, or with display of some field contents of the hit records, letting you chose and arrange the order of these fields. That's very good, but, for a database, not exceptional.
Now for desktop search (dts), and that's why I've created this thread, to collect opinions, insights.
In fact, a lot of people in the web assert that nowadays, databases ain't appropriate anymore but for very specific tasks; they say dts will do almost all what databases did, and presentation of the results, in "tables", with context, is as good as in many databases, and instead of having many k of records in a proprietary format, you'll have the same number of .txt or .xml files, and it's precisely the arrival of that .xml formats that seem to have triggered this new data storage philosophy.
dts problem one
In fact, Boolean searches and all that is very well available, but then, not many dts progs will give you your hits in context (and then, in a Boolean search, WHAT's the context? but you could designate the very first line of the hit file to be displayed in the "results table", but no such dts program offers that to my knowledge - while AS does).
dts problem two
And then, you'd like to search for some number ranges, not only for straight hits: not only for "Paris", but for postal areas 75001 to 75009 e.g., or for your customers / prospects within areas 12400 and 17000, and (in the case of customers) with a turnover between x and y dollars or whatever; again, AS offers such searches, natively, whilst not a single dts program of my knowledge offers such possibility up-front. (I've been one of the severest critics of AS, this thread is not meant to tout that product, it just serves me to explain things, and I'm rather searching for NOT having to use buggy AS in my business.)
In AS, and within .txt and .xml files, such "figure" data would be stored in the form
$xy:123*
or something, and thus, searching for records / files where $xy is comprised between figure x and figure y means that for every such "field", the content is to be fetched, then to be evaluated, then to be decided upon, "yes or no" (fulfills the condition or not), i.e. a SUB-ROUTINE must process these figures, no simple evaluation "string a is equal/not equal to string b" here.
To my knowledge, no current dts program offers such functionality (whilst AS evidently has got such an inbuilt feature within its search routine, as has every other, "real" database).
In theory, you could overcome these dts limitations by using regular expressions: Your search would not be within a mask, "content of field abc between x and y", but in the search line something like
$xy:\[1]\[4-8]\?\?\?
meaning (for me, this being not a correct regex example) every number comprised between 14,000 and 18,999 would here be acceptable;
as you can see, this is not practical if you've got many such amounts/numbers to search for in a same search, even with macros, and it's not possible to search this way for numbers, in our example, between 14,500 and 18,200 e.g., since that would mean to find a regex stipulating that:
- at the third position, all digits are acceptable
- but not if position 2 digit is 4: then (at that position 3) only digits 5 to 9 are acceptable
- and not if position 2 digit is 8: then (again at that position 3) only digits 0 to 2 are acceptable
Which is to say that while in a database, a simple subroutine will do all this without any problem, in a very easy way, you cannot reach out for similar results - and without even attaining them, within a dts engine, but by atrociously convoluted regex deployment.
As for amounts, the simple way, "between 14 and 18 k" would suffice, and thus that would be doable regex-wise, whilst the postal area code example would not work that way, but then, in most cases, unwanted hits would not be too prominent. So this COULD be done, but does anybody do it in practice? Is anybody using a dts engine to search tens of k of .txt or .xml files - or did I overlook the better dts search engines that indeed do have inbuilt sub-routines for more flexible data searching, thus making it possible to use independent files instead of data records?
I think those ideas can be buried, no? And back to MS Access...
But I think it's a passionate subject to think over.
Common databases generally do not offer that leisure, every look into their content must be a (possibly stored) search, or you go from one record to another, flat view - it goes without saying that for any serious search, most databases are much more suitable than most outliners.
But there are UltraRecall, MyInfo and other applics that allow for searching by attributes; they're called "column outliners" (in UR, those attributes ain't in columns yet, but this will be introduced, and they can be entered and searched for, although in a less handy way), and sometimes, even "sorted views" are possible, but only as alternative to the sorted tree, i.e. the normal outline is not stored somewhere, then temporarily replaced by an alternative tree, sorted by some attributes, but it's a special window, with flat view of all items, at best - I prefer a database, then (and we're speaking of tens of k of records to be searched, and outliners weren't meant for this...).
Some will know askSam, a venerable database that was state of the art in the DOS years, but that had not been developed in a very smart way any further, then went almost buried, and now they promise to deliver some modern product. I mention AS here because it's got ONE very smart feature (even if poorly implemented: no storage, mandatory mouse use for settings, very slow if many k's of records): You can build up a virtual tree by the content of 2 or 3 record fields at any time, and since these fields can be freely chosen, the program will build up alternative trees, alternative views, i.e. exactly the thing current outliners cannot do. This is very clever indeed, and if they realize a crash-free version of that program, it could become a winner again.
Of course, AS offers elaborate searches (and here, also stored ones), and it offers a "table" of search results, with context of the results, or with display of some field contents of the hit records, letting you chose and arrange the order of these fields. That's very good, but, for a database, not exceptional.
Now for desktop search (dts), and that's why I've created this thread, to collect opinions, insights.
In fact, a lot of people in the web assert that nowadays, databases ain't appropriate anymore but for very specific tasks; they say dts will do almost all what databases did, and presentation of the results, in "tables", with context, is as good as in many databases, and instead of having many k of records in a proprietary format, you'll have the same number of .txt or .xml files, and it's precisely the arrival of that .xml formats that seem to have triggered this new data storage philosophy.
dts problem one
In fact, Boolean searches and all that is very well available, but then, not many dts progs will give you your hits in context (and then, in a Boolean search, WHAT's the context? but you could designate the very first line of the hit file to be displayed in the "results table", but no such dts program offers that to my knowledge - while AS does).
dts problem two
And then, you'd like to search for some number ranges, not only for straight hits: not only for "Paris", but for postal areas 75001 to 75009 e.g., or for your customers / prospects within areas 12400 and 17000, and (in the case of customers) with a turnover between x and y dollars or whatever; again, AS offers such searches, natively, whilst not a single dts program of my knowledge offers such possibility up-front. (I've been one of the severest critics of AS, this thread is not meant to tout that product, it just serves me to explain things, and I'm rather searching for NOT having to use buggy AS in my business.)
In AS, and within .txt and .xml files, such "figure" data would be stored in the form
$xy:123*
or something, and thus, searching for records / files where $xy is comprised between figure x and figure y means that for every such "field", the content is to be fetched, then to be evaluated, then to be decided upon, "yes or no" (fulfills the condition or not), i.e. a SUB-ROUTINE must process these figures, no simple evaluation "string a is equal/not equal to string b" here.
To my knowledge, no current dts program offers such functionality (whilst AS evidently has got such an inbuilt feature within its search routine, as has every other, "real" database).
In theory, you could overcome these dts limitations by using regular expressions: Your search would not be within a mask, "content of field abc between x and y", but in the search line something like
$xy:\[1]\[4-8]\?\?\?
meaning (for me, this being not a correct regex example) every number comprised between 14,000 and 18,999 would here be acceptable;
as you can see, this is not practical if you've got many such amounts/numbers to search for in a same search, even with macros, and it's not possible to search this way for numbers, in our example, between 14,500 and 18,200 e.g., since that would mean to find a regex stipulating that:
- at the third position, all digits are acceptable
- but not if position 2 digit is 4: then (at that position 3) only digits 5 to 9 are acceptable
- and not if position 2 digit is 8: then (again at that position 3) only digits 0 to 2 are acceptable
Which is to say that while in a database, a simple subroutine will do all this without any problem, in a very easy way, you cannot reach out for similar results - and without even attaining them, within a dts engine, but by atrociously convoluted regex deployment.
As for amounts, the simple way, "between 14 and 18 k" would suffice, and thus that would be doable regex-wise, whilst the postal area code example would not work that way, but then, in most cases, unwanted hits would not be too prominent. So this COULD be done, but does anybody do it in practice? Is anybody using a dts engine to search tens of k of .txt or .xml files - or did I overlook the better dts search engines that indeed do have inbuilt sub-routines for more flexible data searching, thus making it possible to use independent files instead of data records?
I think those ideas can be buried, no? And back to MS Access...
But I think it's a passionate subject to think over.