Find hex string in specific position in a file

[thecon]

Greetings. Nice software there!

I need to use XYplorer to find all files that begin with a random two hex characters (except 00) and then two specific hex characters (which are 00's by chance).

For example:

32 2A 00 00
or
11 01 00 00

Generally, anything that doesn't have hex 00 on the first and second byte of the file, but they do on the 3rd and 4th.

I've tried using "Contents", Mode = RegExp, Type = Binary and disable "Match case", "invert", "It's a hex string", and "match unicode".
Then I type on the contents box:
^[!\x00][!\x00][\x00][\x00]

However, this does not work. I know there are files with non-00 hex on the first 2 bytes and hex 00 on 3rd and 4th, and they are not captured.

If I remove the "!" on the box, it does capture all files that have all 4 first bytes of hex-00 (00 00 00 00). Therefore there must be something wrong with my
syntax, or perhaps the "!" doesn't work as intended on the original ^[!\x00][!\x00][\x00][\x00].

I could use
^..[\x00][\x00]
or even better,
^[\s\S][\s\S][\x00][\x00]
which both work, but with the drawback that they could still capture "00 00 00 00", which I don't want to (there as so many files that begin with 4 00's and
I know the first two characters are NOT 00 00).

Any ideas? I enclose two small files, one with 4 00's at the start and another one with 00's on the 3rd and 4th byte.

As an additional (but secondary) question, if a specific hex string is in a fixed position of the file (etc position 0x1003a with a hex string of 00 32 2a 00 00),
what could be done with XYplorer to find all such files? Position 0x1003a is way too big to insert .'s and I wouldn't want to capture this string in any other
fixed position.

[highend]

^[^\x00][^\x00][\x00][\x00]

Should work for the first question

[admin]

Conc. the 2nd question, here is a small script to extract a certain number of bytes from a certain position of a file, and show them in hex format.

Code: Select all

// read 4 bytes at position 1 (first position) from the currently selected file, show in hex format
  $a = readfile(,,4,,1);
  echo hexdump($a,,r);

To actually search a location for files with that sequence you need some more scripting.


BTW, a similar script can also be used to define a Custom Column that would show those 4 bytes for each file in the list.

[admin]

Idea! (where is that light bulb icon?)

I could enhance the normal Find File / Content / Hex by letting you define a position for the Hex string to match:

Code: Select all

0x1003a: 50 4B 03 04 

or optionally also decimal:

Code: Select all

65594: 50 4B 03 04 

Goody?

[TheQwerty]

Conc. the 2nd question, here is a small script to extract a certain number of bytes from a certain position of a file, and show them in hex format.

Code: Select all

// read 4 bytes at position 1 (first position) from the currently selected file, show in hex format
  $a = readfile(,,4,,1);
  echo hexdump($a,,r);

To actually search a location for files with that sequence you need some more scripting.


BTW, a similar script can also be used to define a Custom Column that would show those 4 bytes for each file in the list.

I was going to suggest a similar script, but then in testing I got the following error when it hit a 3.7 GB file:

Error 7

Error: 7 (0x00000007)
Desc: Out of memory
Dll: 0
Proc: script_Process: return

Source: XYplorer
XY ver: XYplorer 15.90.0020
OS: Windows 7 Professional (Service Pack 1), 64-bit
Locale: 1033
ANSI: 1252
ACP: 1252 (ANSI - Latin I)
DBCS: No
DPI: 96 (100%)

Date: 2015-11-09 16:26:03

Removing the sanity checks from the below script triggers the error.

Working around that led me to cap it at 1 GB which worked okay here:

Code: Select all

$bytes = 5;          // Bytes to return.
$address = 0x1003a;  // Starting address (1-based)

// Some sanity checks to speed it up.
$size = filesize(<cc_item>);
// 1 GiB still feels too big.
// Setting it too high can cause an Out of Memory error.
if ($size > 1074000000*1) { return 'too big'; } // 1 GiB
if ($size < $address) { return 'too small'; }

// Read file and convert to hex.
return hexdump(readfile(<cc_item>, 't', $bytes,, $address), 0, 'r');

Your suggested improvements to Find Files sound great to me!

[thecon]

^[^\x00][^\x00][\x00][\x00]

Should work for the first question

It DOES work, however, with a wrong side effect.
It captures files such as
1A EE 00 00 ....

but it doesn't capture those rare files which have
00 01 00 00 .....
or
BB 00 00 00 .....

So, in short, i need the first two bytes to be NOT "00 00" and the 3rd and 4th to be "00 00"

Thanks!

[thecon]

Idea! (where is that light bulb icon?)

I could enhance the normal Find File / Content / Hex by letting you define a position for the Hex string to match:

Code: Select all

0x1003a: 50 4B 03 04 

or optionally also decimal:

Code: Select all

65594: 50 4B 03 04 

Goody?

That could be an _EXCELLENT_ idea.

Also, you could implement a wildcard hex implementation. For example, winhex when you want to search for something in a specific file,
you could tell to to look for "00 00 AA EE AA 1F" and you could tell it that AA is a wildcard (anything can be specified to be a wildcard).
So, winhex would capture
00 00 00 EE 1E 1F
00 00 AB EE 1A 1F
00 00 BA EE FF 1F
etc...

XYplorer is the greatest

[admin]

@TheQwerty: Thanks, saw that bug and fixed it. Next version will run bug free and much faster on the same job. No sanity checks required anymore.

[admin]

Also, you could implement a wildcard hex implementation. For example, winhex when you want to search for something in a specific file,
you could tell to to look for "00 00 AA EE AA 1F" and you could tell it that AA is a wildcard (anything can be specified to be a wildcard).
So, winhex would capture
00 00 00 EE 1E 1F
00 00 AB EE 1A 1F
00 00 BA EE FF 1F
etc...

_COOL_ idea!

Coming with next beta...

[admin]

And? Any reaction? Is it cool?

[highend]

I could now write my own Universal Extractor tool easily...

[xman]

This is definitely useful and can replace regex in many situations!

So instead of, say, \x11\x12[\d\D]{3}\x23, I can now simplify it to just 11 12 ?? ?? ?? 23

[admin]

And it's MUCH faster.