Feature Request: TAXII Services - STIX Upload Avoid Creation of Identical Events

[packet-rat]

When importing NCCIC STIX Packages with STIX Upload two nearly identical Events are created from a single STIX Package. The only difference between the duplicate Events seems to be the Description.

Can't share the files or screenshots publicly due to TLP, but the problem can be locally recreated using NCCIC STIX Package: MAR-455057_stix.xml.

This specific Malware Analysis Report was chosen because it only has a few objects making it much easier to analyze than the majority of NCCIC STIX Packages with 100's of Objects and Relationships.

(1) STIX Upload MAR-455057_stix.xml

(2) Two separate events are created with the following Titles:

MAR-455057 -- This Event has a blank Description
455057 -- This Event has the full Description

@brlogan : If you don't have these locally, all of these NCCIC STIX Packages are "up on the portal" in ZIP Archives attached to Message Board Posts - just search on NCCIC, or STIX, or US-CERT

[brlogan]

I took a look at the XML, and this is expected behavior.
The first CRITs Event (MAR-455057) is constructed from the package header. As you already know, you can control whether or not this happens via the Pkg Header Events TAXII service configuration option.
The second CRITs Event (455057) is constructed from the STIX Incident.

This is desired behavior because sometimes a single STIX package will contain multiple Incidents with relationships to different subsets of objects within the package. The package-level Event will have relationships to every object in the package (including the incident-level Events), while the incident-level Events will only relate to those TLOs specified within its respective Incident. In the case of this specific STIX document, the Incident relates to all 3 Indicators in the package, so the two Events have the same relationships.

This might be a good justification for being able to configure the Pkg Header Events option on a per-feed basis.

[packet-rat]

Thinking Out Loud: Multiple Events for the latter scenario makes perfect sense (multiple incidents with different Objects and Relationships). For the scenario where essentially identical Events will be created, wouldn't it make sense to check for this condition before creating "duplicate" events? In the current behavior: (1). 100s of "duplicates" are created. When Relationships to other Events and Objects (from this Source or correlations to other Intel and Analysis Sources) the Relationship Graphs become very convoluted. (2) Bifurcation can quickly emerge for workflows where disposition, annotation, status, attribution, correlation etc. are managed and tracked in the Event Object container. Thoughts?

[mgoffin]

This is yet another reason why I don't like STIX. People can abuse content structure horribly and create custom objects which require custom code to parse and it makes ingestion systems practically impossible to develop without some level of compromise and frustration.

[brlogan]

@packet-rat: Checking for duplication sounds nice, but what qualifies as a duplicate? In the example you provided, the Package header and Incident differed in both title and description. If the two were truly identical I might be comfortable dropping one of them, but in many (if not most) cases they won't be. What if a Package contained two (or more) Incidents that collectively duplicated the package, but individually only covered a subset of the Package contents? In that case it becomes more difficult to determine if the Package-level Event is a duplicate, and the Package-level Event may even be desired as a way to group the Incident-level Events together. It seems to me that it would be difficult to determining if a Package header is a duplicate of any related Incidents in a consistent and predictable way that would make sense in all use cases.

The easier (and possibly better) solution might be to allow feed-specific configuration of Package header import. That way, if the Packages in a particular feed always include useful Incidents, the Package header can simply be dropped.