Add dumpAvroRecords method and start method error checking #9
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add support for reading flowtuple4 files
Incomplete, but most of the way there.
* explicitly declare certain variables as unsigned and const * use boundscheck and wraparound decorators to try and reduce wasteful index checking on our bytearrays * use cdef for internal functions to save python overhead
* explicitly declare certain variables as unsigned and const * use boundscheck and wraparound decorators to try and reduce wasteful index checking on our bytearrays * use cdef for internal functions to save python overhead
* avoid passing parsedString struct into functions, as the scope change can completely break the contents if intentional null bytes are present * change parseString to allocate memory for string contents directly and copy the string into it * null characters are appending to the end of the string in read_string() itself * added parameter to read_string() to force null termination to NOT be added to the read string * ensure entire initial packet contents are returned by getRsdosPacketString(), not just up to the first null byte
Code to support 'rb' (or non-utf8 decoding of 'r' files) is not yet available in pywandio publicly, which was causing issues for external users.
Add code to support reading RS-DOS attack avro data
Improve docs and licensing
All AvroReaders will now save the schema from the Avro header so that they can check for the presence of potentially missing data fields.
Also added a getRsdosPacketSize() method that accounts for any tag header that may be present.
Add support for upcoming new RSDOS data schema
* Don't include fields in the string that don't exist in schema version 1. * Decode string fields to 'utf-8' before adding to str() result. * Rename 'attacker_ip_count' to 'attacker_slash16_count' for v2 asDict() result.
Update rsdos parser to handle both the original schema and the new version just added to corsaro3
Schema version is now declared in the baseavro class, as it is used by multiple data formats.
Now the perAvroRecord and dumpAvroRecords commands check to make sure that the reader has had the start method called on it before attempting to process any files
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
dumpAvroRecordsmethod is nearly a clone of theperAvroRecordmethod except for the fact that it's a generator function instead of one that uses callbacks. This makes it easier to iterate over all records in an avro file using something like afor inloop rather than an asynchronous callback. Basic checking for whether the start method was called yet is also something that is implemented.