Created ordered list for HTTP headers

[nakulbajaj]

Modified ZGrab2's HTTP package to extract headers as a list of
HeaderField objects and then extract as a map via the textproto package.

Created new HeaderField object with UTF-8 string for header names
paired with a byte slice for header values.

[mzpqnxow]

@nakulbajaj does this change how the headers are represented in the NDJSON output?

(Aside from ordering, of course)

[nakulbajaj]

@mzpqnxow This doesn't change the "headers" dictionary as part of the JSON output at all. Instead, it creates a new field in the JSON output called "header_list," which composed of a list of header objects (which include a name and a value string).

[phillip-stephens]

Hey @nakulbajaj, thanks for the PR and apologies it's taken 4 years to get any eyes on it!

I'm curious though what the value-add is for this ordered list. From a quick http scan, it looks like header_list's fields are the same as headers but having the values be base-64 encoded rather than decoded into human-readable strings as they are in headers.
Are you wanting a way to store the exact order of header fields sent by the server, or something else? From initial impressions, this seems to require twice the output for the one benefit of preserving the header order sent by the server, which I'd imagine we could do in a less verbose way: "header_ordered_list": ["Server", "Date",...] .

          "headers": {
            "content_length": [
              "102"
            ],
            "content_type": [
              "text/html"
            ],
            "date": [
              "Mon, 27 Jan 2025 19:40:23 GMT"
            ],
            "last_modified": [
              "Mon, 27 Jan 2025 19:38:52 GMT"
            ],
            "server": [
              "SimpleHTTP/0.6 Python/2.7.9"
            ]
          },
          "header_list": [
            {
              "name": "Server",
              "value": "U2ltcGxlSFRUUC8wLjYgUHl0aG9uLzIuNy45"
            },
            {
              "name": "Date",
              "value": "TW9uLCAyNyBKYW4gMjAyNSAxOTo0MDoyMyBHTVQ="
            },
            {
              "name": "Content-Type",
              "value": "dGV4dC9odG1s"
            },
            {
              "name": "Content-Length",
              "value": "MTAy"
            },
            {
              "name": "Last-Modified",
              "value": "TW9uLCAyNyBKYW4gMjAyNSAxOTozODo1MiBHTVQ="
            }
          ],

[phillip-stephens]

I'm going to close this due to the length of time this has been opened. Please feel free to re-open if the PR still feels justified.

[Seanstoppable]

So, my understanding is that in some scenarios header order matters for things like fingerprinting some services/systems. So returning them in a map, where ordering is non-deterministic + concatenated, prevents this.
My org maintains a fork of zgrab2 with some modifications to the http module to capture headers in their more 'raw' representation (though as a block of strings, akin to curl output) to make sure we can do similar fingerprinting.

[Seanstoppable]

This is probably still obsoleted by #349