diff --git a/docs/new_connector.md b/docs/new_connector.md index 382afc7..827db14 100644 --- a/docs/new_connector.md +++ b/docs/new_connector.md @@ -11,11 +11,11 @@ Regardless of the connector actions, it will need to somehow save the following | Project name | PROJECT-X | | Expiration days (or date) | 100 (or 01/01/2024) | -# Download Processing +## Download Processing File processing is the one flow that differs from other structured data ingestion. First, the file bytes need to be uploaded to Nemesis, and second, a metadata message needs to be posted to kick off processing. -## Step 1 - File Upload +### Step 1 - File Upload For a file to be processed, the raw file bytes first need to be posted to the correct API route for storage in the data lake. This is accomplished by POSTing the file bytes to the `https:///api/file` which returns a simple JSON response with an `object_id` field containing a UUID that references the uploaded file. For example, to do this in Python (as shown in [mythic-connector](https://github.com/SpecterOps/Nemesis/blob/main/cmd/connectors/mythic-connector/sync.py)), you would run something like this: @@ -32,7 +32,7 @@ curl -H "Content-Type: application/octet-stream" -v --user 'nemesis:Qwerty12345' The `nemesis_file_id` is used in the `file_data` message in Step 2 below. This UUID is the unique reference for the file in Nemesis. -## Step 2 - File Data Message +### Step 2 - File Data Message After the file is uploaded to Nemesis, a [file_data](odr/references/file_data.md) ODR message needs to be posted with file metadata information. The example from the [mythic-connector](https://github.com/SpecterOps/Nemesis/blob/main/cmd/connectors/mythic-connector/sync.py) is: @@ -62,7 +62,7 @@ r = requests.request("POST", f"{NEMESIS_URL}/data", auth=basic, data=data, heade *Note that timestamps need to be in ISO 8601 UTC form, e.g., 2023-08-01T22:51:35* -# Other Structured Data +## Other Structured Data For other types of structured data, only a single message needs to be posted to the `http:///api/data` API route, e.g. Step 2 in the downloading processing example. The `metadata["data_type"]` field should be one of the types defined in the [ODR](odr/references/). The appropriate ODR document will also define the fields and structure needed for the datatype.