XYplorer Beta Club

@Jeff

I don't know much about what you're trying to achieve in detail because you described "more of a concept" that you would like to do but not how exactly things should work. English isn't my mother language and it isn't always easy to understand if there aren't any real world example (and you didn't post your script or at least: relevant parts of it). You make it easier (for me) to understand what you really need by delivering something like this:

01. A general idea of what your trying to achieve and how you'd like to do that
02. A few examples of what is going wrong
03. "" of how it should have been
04. Maybe the relevant section of the code you're struggling with

After this, even if the reason is still unknown (that's the case for me up to now) I can deduct the logic from the input and the expected output.

I know the code looks (sometimes) complicated. It isn't At least for me (or my brain). I'm not a coder, there are always more elegant ways to express logic in scripts. I (normally) document my code but in most cases not if I only post a few lines.
01. My example directories contained linebreaks instead of pipes. I'm just converting here because... 04. will use a formatlist and it's separator argument (it's missing in this case because in defaults to "|") is used for the input param AND the pattern at the end) and I didn't want to use <crlf> as the separator for the $excludeList because of this reason. If I would have posted the example directories with "|" as the separator I would omit the 01. line completely but the output would have been difficult to read.

02. It's just the list of parts of a foldername that will be excluded later on. The pipe is used because for formatlist() patterns it means "alternation" or a logical "OR" and that's what we need.

03. I'm putting a "!* at the front and a "*" at the back the pattern and replace all pipes from the $excludeList with "*|*". In the end the pattern looks like this: "!*hits*|*the*". Why? Because "hits" and "the" can be anywhere inside the path of each directory.

04. A filtering formatlist(). The filter uses "!" to inverse all the entries in it -> So it will just give back those results that don't contain any of the entries in it hence why it delivers back only two directories at the end.

I still don't know exactly what you're trying to achieve with your script (that nobody of us have ever seen) so I was just posting a general hint of how filtering can be done. Don't know if it fits the purpose. If not tell me (with examples) why it isn't the solution...

The same thing goes for your optimized script in the thread where we discussed how to search for folders that *don't* contain a particular file or files

Do me a favor and describe what exactly you don't understand. Quote the patterns that are hard to read / understand

I don't know much about what you're trying to achieve in detail because you described "more of a concept" that you would like to do but not how exactly things should work. English isn't my mother language and it isn't always easy to understand if there aren't any real world example (and you didn't post your script or at least: relevant parts of it).

Thank you for taking the time to post a detailed and carefully thought-out response. I would never have guessed that English is not your native language -- you communicate very well in English!

I know I only provided a general overview of what I was trying to achieve, and I certainly did not expect anyone to write a perfect script based only on that general information! I recognized your script code as a general case, and I was trying to understand your logic in much the same way that you were trying to understand what I wanted from my folderreports. Your general case example using the Queen album folders is excellent. To keep this post from becoming "tldr", I will post more specifics about what I'm trying to do in another post.

03. I'm putting a "!* at the front and a "*" at the back the pattern and replace all pipes from the $excludeList with "*|*". In the end the pattern looks like this: "!*hits*|*the*". Why? Because "hits" and "the" can be anywhere inside the path of each directory.

As you deduced, I struggled with the result of 03. So following your logic, if I needed to add "Best" to the exclude list, I would add: and the exclude pattern would look like: "!*hits*|*the*|*best*". It would be interpreted as "Return results that don't have "hits", "the" or "best" anywhere in the name". Is that correct?

I have always had trouble with parsing and concatenating strings "on the fly" in scripts. I can do functions and "if-then-else" and "for each" really well. I struggle with strings, which is why regular expressions present such a challenge to me.

Do me a favor and describe what exactly you don't understand. Quote the patterns that are hard to read / understand

Again, I will do that in a separate post.

Thanks,
Jeff

I would never have guessed that English is not your native language

Anyone can see this xD Native english speakers do not use parenthesis as much as I do (we use them in german (I don't want to say: a lot)).

... Is that correct?

Absolutely.

Again, I will do that in a separate post.

And in a different thread

The best thing to do if you don't understand how a script / command works: use step / unstep in such a script. Seeing how a variable / output of a command looks like before and after executing it helps a lot to understand even complicated things.

I have my music and movies in 2 separate top-level folders and various subfolders:
Ripped Music
-- Artists Folder
---- Album Folders
iTunes Music
-- Artist Folders
---- Album Folders
---- Music Video Folders
-- Compilation Folder
---- Album Folders
-- Movies Folder
---- Movie Files
---- Movie Folders
---- Irrelevant Subfolders

iTunes Match will occasionally trash my organization and metadata in the iTunes Music folder by moving songs around and leaving empty folders behind, removing album artwork and/or adding album artwork as System files, and by changing ID3 tag information. (I keep copies of multiple tag values in the Comments field, which iTunes seems to leave intact even when it changes the values of other tags in the same file.) Unfortunately, I do need my subscription to iTunes Match. I could avoid a lot of these issues by canceling it.

My kids have access to the iTunes Music folder, so sometimes things get added, removed or moved accidentally.

The iTunes folder also contains movies, music videos, books and pdf files in various subfolders. There are many of these, so they need to be eliminated from some search results or things get very messy.

I use the Ripped Music folder (which has no interaction with iTunes at all) to create playlists for my computer, my home theater and my whole-house audio system. I need to find multichannel audio for playback in the theater, but I need to exclude it for playback on the computer or the house audio system. The multichannel audio is usually in folders that have (HR) or [HR] or [MCH] or (MCH) or (4CH) or [4CH] in the name, but not always. I can find the ones that the folder name search misses by searching for folders that contain multichannel audio files. That's usually done in Mp3tag. I have a script that generates an .m3u playlist from selected files and folders (Search Results or Paper Folder) so that I can open all of them at once in Mp3tag.

So I need to find:
1. Empty folders.
2. The names of the artist and album that the empty folders represent, because I probably want to move songs and artwork back into them.
3. Folders that contain only files with the System attribute.
4. System files posing as album artwork (will be deleted).
5. Folders that are missing album artwork completely. Artwork will be added using Mp3tag's tools.
6. Folders that do or do not contain multichannel audio.
7. ... and so on.

I have a few scripts that are designed to handle these tasks, but I ran into an issue with #2 because of the odd folder names in some of the subfolders that should be excluded from the search results. My misuse of the getpathcomponent command was also a problem. Most of the odd folder names occur in the Movies subfolders, so I explicitly assigned the results of a folderreport on that folder to a variable. Unfortunately, that method misses some subfolders elsewhere that should be found and excluded. I still need to find a way to exclude "the current folder name in the string" and "all of the subfolders of the current folder name in the string" without explicitly naming those folders at the beginning of the script. This works for the Movies folder:
I hope I've described what I want to do adequately.

Anyone can see this xD Native english speakers do not use parenthesis as much as I do

I disagree -- just read some of *my* posts!

The best thing to do if you don't understand how a script / command works: use step / unstep in such a script. Seeing how a variable / output of a command looks like before and after executing it helps a lot to understand even complicated things.

I do that all the time for my scripts. I wouldn't have gotten even this little distance without doing that! I did it for your script in another thread, but even that didn't help me figure out what you were trying to do. Needless to say I felt really dumb.

Cheers,
Jeff

Mh... The only thing you want to do with the script is filter out all directories which have something "unusual" in it?

If that's the case... Done. No foreach loop^^

But I guess I'm missing something?

The only thing you want to do with the script is filter out all directories which have something "unusual" in it?

Correct.

Now assume: Here is where I get lost. I can decode that you start with a string of folders separated by <crlf> and that you search for $metaCharacters within that string, but on which parts of the string? It seems that you look for new lines until you find the end of the string, but I'm not sure how. Stepping through doesn't help here.

I just don't know enough about Regular Expressions to figure out how that filter is supposed to work, nor how to modify (and not completely replace) the folder list using your method.

It's like if you said to me, "Ein großes Bier, bitte." I know it has something to do with beer.

You've spent enough time on this already, and I really appreciate it. Feel free to move on to other things. Sooner or later I'll figure it out on my own. Probably later. Much, much later.

Cheers,
Jeff

but on which parts of the string?

On all parts Otherwise it wouldn't make much sense, would it?

An example:

These are the folders for the $movieFolders variable.

D:\iTunes Music\Movies
D:\iTunes Music\Movies\[incorrect]
D:\iTunes Music\Movies\The World
D:\iTunes Music\Movies\The World+Ends
D:\iTunes Music\Movies\The World (today)

3 of them contain a char that appears in the $metaCharacters var:

D:\iTunes Music\Movies\[incorrect]
D:\iTunes Music\Movies\The World+Ends
D:\iTunes Music\Movies\The World (today)

So if we're filtering with a regex, it should remove all these lines so that we finally get:

D:\iTunes Music\Movies
D:\iTunes Music\Movies\The World

And we achieve that only by using: The regex:
^ = Start at the front of each line (regexes in XY are multiline aware by default)
.*? = Give back any character (zero or unlimited times) but make the match non greedy
(\^|\$|\.|\+|\(|\)|\[|\{) = Look for any symbol that's inside. It's an alternation (logical OR) and that needs to be put
into round brackets.
.*? = Same as above
(\r?\n|$) = Look for a linefeed OR the final end of the line. We need this because the last line of folders doesn't end on a newline but just with $ (the regex end of line). If we would just use "$" it would do the same but don't replace the newlines with nothing -> We get an empty line on each replaced line).

The whole thing matches ANY character in the $metaCharacters list regardless of it's position in each line of your folders list (and removes trailing newlines on matching lines as well) so you don't even need a formatlist (to delete empty lines) afterwards.

If you have enough money, either send it to me^^ or buy RegexBuddy (http://www.regexbuddy.com/). There are alternatives (cheaper ones) even online regex testers but this one helps so much in understanding (it explains what each part of a regex does) and developing regexes, it's really worth it's price. The help file alone is a wonderful resource of information how regexes work... If you get this one you won't need another tool for regexes in your life xD

(regexes in XY are multiline aware by default)

I didn't know that.

(\^|\$|\.|\+|\(|\)|\[|\{) = Look for any symbol that's inside. It's an alternation (logical OR) and that needs to be put
into round brackets.

Originally, I didn't parse this correctly. I didn't notice that the literal "\" is missing. I thought the regexreplace would find backslashes as well, and that didn't make any sense to me because every folder path would contain backslashes. One question: in the list you include a left and right round bracket, but only left square and left curly brackets. Why?

(\r?\n|$) = Look for a linefeed OR the final end of the line. We need this because the last line of folders doesn't end on a newline but just with $ (the regex end of line). If we would just use "$" it would do the same but don't replace the newlines with nothing -> We get an empty line on each replaced line).

How is "\r\n" different than "\r" or "\n" by themselves? Is it specific to XY's version of regex or is it a Windows thing? My reference documents weren't clear about that.

The whole thing matches ANY character in the $metaCharacters list regardless of it's position in each line of your folders list (and removes trailing newlines on matching lines as well) so you don't even need a formatlist (to delete empty lines) afterwards.

Very cool.

I bought RegexBuddy. It looks like exactly what I need. Thanks for the tip. Hopefully no more basic regex questions from me.

Cheers,
Jeff

OT: Does anyone know what regex engine XY uses? RegexBuddy's list of engines is intimidating and defaults to C# (.NET 2.0 -4.5).

Thanks,
Jeff

NVM. I found a post from last year where Don said it uses the VBscript engine (which doesn't support look behind).

in the list you include a left and right round bracket, but only left square and left curly brackets. Why?

Typo... The same filtering technique (without using a foreach loop) would be possible with formatlist() by supplying a pattern like "!*the*|*hits*" (you remember?^^) BUT you get a script error on using square brackets in that pattern and I was too lazy to debug that so I switched to regexreplace *g*. Forgot to readd what I took out for testing purposes. Add both escaped brackets into the list.

How is "\r\n" different than "\r" or "\n" by themselves? Is it specific to XY's version of regex or is it a Windows thing?

Linux vs. Windows newlines. Putting the ? behind \r makes it optional so it matches both versions. It the safest method (imho) to get _all_ newlines correctly. It's not XY specific.

I bought RegexBuddy.

Damn, I already saw more money flowing in xD

it uses the VBscript engine (which doesn't support look behind).

Which is a major drawback. Look behind is a powerful thing (as Look ahead is!). It lets you exclude things that your expression matches, e.g. if you want to get only the path component for a file (or better for all files in a huge variable) you use an expression like this:
^.*(?=\\)

Which matches anything from the beginning of the line up to (but NOT including) the LAST backslash in it (called: negative look ahead).

You remember what TheQwerty posted in this thread? Each folder was returned with a trailing backslash. So my last pattern would return the path for all lines regardless if it's a folder or a file and that's something what getpathcomponent() couldn't do (it returns the parent of a folder instead of the path) and you would need a foreach loop as well...