Categorize web and script files

[Czatar]

Summary

Categorize web files and scripts for later analysis. Related to issue #75.

If merged this pull request will

Proposed changes

Added branches to magic bytes that cover python, bash, html, css, and js. The bash and html branches use magic bytes in case of a missing file type extension.

[Czatar]

Wondering if:

• File type extension should be made into its own function or if it's fine to have it where it is
• There are any file types to include or exclude for this PR

[nightlark]

* File type extension should be made into its own function or if it's fine to have it where it is

I was thinking it could be made into its own function, since it isn't really a check based on magic bytes. Using the #! line is also maybe more of a heuristic than magic bytes?

[nightlark]

Something I wanted to make a note on regarding the regex checks -- I came across a few #! lines in a stackoverflow post:

#!/usr/bin/env bash
#!/bin/bash
#!/bin/sh
#!/bin/sh -

In particular, the last one stands out where the end of the line is passing a - as an argument to sh. I think in theory the shebang line could have any set of arguments passed to the given command, so to recognize some corner cases it might be necessary to split the line similar to how the OS would to identify the command to run (+ a special case for using the first argument if env is the command).

This sounds like a lot of work though for an uncommon case, so I'm leaning towards adding that as a GitHub issue for the backlog.

[Czatar]

The issue with having arguments in the shebang line is tricky. From what I know:

• It's possible for the path to the binary to have spaces, so we can't just split by spaces
• It's possible for the arguments to have paths, so just splitting it by slashes also won't help

My next thought would be to check for single/double quotes or backslashes that include the space in the path. Windows doesn't acknowledge shebang lines, so maybe backslash checking won't be a huge issue. For the GitHub issue, Is there anything else that would complicate this?

[nightlark]

Forgot to make these comments visible.

[nightlark]

I'm thinking that instead of mixing the suffix and #! checks, let's move the suffix related parts of this to after the try/except block (and remove the return None that's right above the except line.

This will also give priority to whatever the #! line says, which should be more reliable than the file suffix, which could "lie" a bit more easily.

[nightlark]

I think this section should also be wrapped in a try/except block to catch any UnicodeDecodeError that could get thrown, and log a warning with some details on what happened. (Or add a 2nd except block after the existing one -- that then falls through to the file suffix check).

[nightlark]

Any idea how the performance is for re.search vs just using the in keyword?

[nightlark]

Okay, so I put together some quick scripts to test with time python3 <script>. I got around 2.3s using re.search, and 1.2s using in.

import re

interpreters = {
    b"sh": "SHELL",
    b"bash": "BASH",
    b"zsh": "ZSH",
    b"php": "PHP",
    b"python": "PYTHON",
    b"python3": "PYTHON",
}

candidate = b"#!/bin/fsdafsdf/local/bas/bas/bas/bas/pytho/python3"

for i in range(0, 1000000):
    for interpreter, filetype in interpreters.items():
        if re.search(interpreter, candidate):
            pass

interpreters = {
    b"sh": "SHELL",
    b"bash": "BASH",
    b"zsh": "ZSH",
    b"php": "PHP",
    b"python": "PYTHON",
    b"python3": "PYTHON",
}

candidate = b"#!/bin/fsdafsdf/local/bas/bas/bas/bas/pytho/python3"

for i in range(0, 1000000):
    for interpreter, filetype in interpreters.items():
        if interpreter in candidate:
            pass

Out of curiosity, I tried out using Aho-Corasick (on strings though... using the pyahocorasick library) to see how fast it could be if we really wanted to get a faster search; with this, the test took 0.35s to run:

import ahocorasick

automaton = ahocorasick.Automaton()

interpreters = {
        "sh": "SHELL",
        "bash": "BASH",
        "zsh": "ZSH",
        "php": "PHP",
        "python": "PYTHON",
        "python3": "PYTHON3",
}

for interpreter, filetype in interpreters.items():
    automaton.add_word(interpreter, filetype)

automaton.make_automaton()

candidate = "#!/bin/fsdafsdf/local/bas/bas/bas/bas/pytho/python3"

for i in range(0, 1000000):
    for item in automaton.iter_long(candidate):
        pass

At some point, it might be nice to switch some of the different file type ID checks over to a faster method (magic bytes, srec/hex, etc) -- but saving a second or two isn't that much compared to the time it takes to generate hashes and analyze large binaries.