Issue warning when converting composite expressions about cti-stix-elevator HOT 8 CLOSED

I would say that if there is a composite indicator with AND, we should use ALONGWITH regardless of whether it's the same type or not.

from cti-stix-elevator.

how about:

file:name = 'fred.pdf' AND file:hashes:md5 = 'a4754be7b34ed55faff832edadac61f6

That seems to be a reasonable observable expression.

Also, are you assuming Composite Indicator Expressions and Composite Observable Expressions should be handled the same way or differently??

from cti-stix-elevator.

In general, a pattern in an Composite_Indicator_Expression should be handled the same as a pattern in an Observable_Composition. I'm still not sure how to handle multiple nested compositions (either an Observable_Composition in an Indicator in a Composite_Indicator_Expression, or an Composite_Indicator_Expression in an Indicator in an Composite_Indicator_Expression), but I think we can defer that for now.

My point about AND vs. ALONGWITH (or, if we change ALONGWITH to AND, AND between Comparison Expressions vs. AND between Observation Expressions) was just that you should have a single Observation Expression per CybOX 2.X Observable. So if you have:

<cybox:Observable>
  <cybox:Object>
    <cybox:Properties xsi:type="FileObj:FileObjectType">
      <FileObj:File_Name condition="Equals">fred.pdf</FileObj:File_Name>
      <FileObj:Hashes>
        <cyboxCommon:Hash>
          <cyboxCommon:Type condition="Equals">MD5</cyboxCommon:Type>
          <cyboxCommon:Simple_Hash_Value condition="Equals">a4754be7b34ed55faff832edadac61f6</cyboxCommon:Simple_Hash_Value>
        </cyboxCommon:Hash>
      </FileObj:Hashes>
    </cybox:Properties>
  </cybox:Object>
</cybox:Observable>

then it should be represented as [file:name = 'fred.pdf' AND file:hashes:md5 = 'a4754be7b34ed55faff832edadac61f6'].

But if you have the following two Observables, no matter how they are composed, you should use
[file:name = 'fred.pdf'] ALONGWITH [file:hashes:md5 = 'a4754be7b34ed55faff832edadac61f6'] (substituting AND for ALONGWITH if necessary).

<cybox:Observable>
  <cybox:Object>
    <cybox:Properties xsi:type="FileObj:FileObjectType">
      <FileObj:File_Name condition="Equals">fred.pdf</FileObj:File_Name>
    </cybox:Properties>
  </cybox:Object>
</cybox:Observable>
...somehow composed with...
<cybox:Observable>
  <cybox:Object>
    <cybox:Properties xsi:type="FileObj:FileObjectType">
      <FileObj:Hashes>
        <cyboxCommon:Hash>
          <cyboxCommon:Type condition="Equals">MD5</cyboxCommon:Type>
          <cyboxCommon:Simple_Hash_Value condition="Equals">a4754be7b34ed55faff832edadac61f6</cyboxCommon:Simple_Hash_Value>
        </cyboxCommon:Hash>
      </FileObj:Hashes>
    </cybox:Properties>
  </cybox:Object>
</cybox:Observable>

from cti-stix-elevator.

I think you overestimate the ability of users to specify observable patterns. See the indicator with the title "GDOCUPLOAD (Family)" in APT1/Appendix_G_IOCs_full.xml.

Look at the inner "AND" composition - it corresponds to the following pattern expression:

((file:file_name = 'gdocs.exe' OR file:file_name = 'hotmail.exe' OR file:file_name = 'hotmail.exe' OR file:file_name = 'sg.exe') AND (file:size = '104448' OR file:size = '104449' OR file:size = '98304' OR file:size = '113664') AND (file:extended_properties.windows_pebinary_ext.file_header.time_date_stamp = '2011-03-18T02:41:49Z' OR file:extended_properties.windows_pebinary_ext.file_header.time_date_stamp = '2011-07-29T00:57:16Z' OR file:extended_properties.windows_pebinary_ext.file_header.time_date_stamp = '2011-03-31T03:16:31Z'))

These should all match the same file observation, and therefore need to be in ONE observable expression.

from cti-stix-elevator.

Ok, I see what you're saying. But that falls into the "multiple nested compositions" area (I should have included "Observable_Composition in Observable in Observable_Composition" in my examples above). The OR is being used to define multiple possible values for the same field. I would expect that it's hard to infer the exact intent of nested compositions in an algorithmic way, so I'd rather just emit a warning and give up.

Granted, part of the reason for this is that it was translated somewhat-literally from OpenIOC, when in fact an optimal representation in STIX 1 would have used condition="Equals" apply_condition="ANY" on a single WinExecutableFileObject, using a list of values delimited by ##comma##:

<cybox:Observable id="mandiant:observable-3cfaf45b-31a1-4f1e-a690-09f132e5c612">
  <cybox:Object>
    <cybox:Properties xsi:type="WinExecutableFileObj:WindowsExecutableFileObjectType">
      <FileObj:File_Name condition="Equals" apply_condition="ANY">gdocs.exe##comma##hotmail.exe##comma##hotmail.exe##comma##sg.exe</FileObj:File_Name>
      <FileObj:Size_In_Bytes condition="Equals" apply_condition="ANY">104448##comma##104449##comma##98304##comma##113664</FileObj:Size_In_Bytes>
      <WinExecutableFileObj:Headers>
        <WinExecutableFileObj:File_Header>
          <WinExecutableFileObj:Time_Date_Stamp condition="Equals" apply_condition="ANY">2011-03-18T02:41:49Z##comma##2011-07-29T00:57:16Z##comma##2011-03-18T02:41:49Z</WinExecutableFileObj:Time_Date_Stamp>
        </WinExecutableFileObj:File_Header>
      </WinExecutableFileObj:Headers>
    </cybox:Properties>
  </cybox:Object>
</cybox:Observable>

(notwithstanding hotmail.exe being included twice).

This would be naturally translated to a STIX 2 pattern like

[file:file_name IN ('gdocs.exe', 'hotmail.exe', 'hotmail.exe', 'sg.exe') AND 
file:size IN (104448, 104449, 98304, 113664) AND 
file:extended_properties.windows_pebinary_ext.file_header.time_date_stamp IN ('2011-03-18T02:41:49Z', '2011-07-29T00:57:16Z', '2011-03-31T03:16:31Z')]

But it's hard to convince the elevator to do that automatically from the current content in APT1 Appendix G.

I suppose we could look for the "conjunction of disjunctions" pattern -- (A or B or C) AND (D or E) AND (F or G or H) -- where all items in a given OR set use the same single field name, and collapse those into an IN Comparison Expression. Then, if all the ANDed expressions have the same root object type, combine the Comparison Expressions into the same Observation Expression.

The more I think about it, I bet it would be possible to develop some decent heuristics, but I'm still not convinced it's worth the effort.

I'm curious if @chisholm has any thoughts.

from cti-stix-elevator.

For the first 1.0 release of the elevator I would prefer to keep it simple - and not try to use a fancy heuristic to determine what the producer REALLY wants. The dumb heuristic I'm planning is to keep as many comparison expressions within an observation expression as possible - based on the object path root.

from cti-stix-elevator.

I'm fine with that, as long as it won't lead to false negatives. False positives are OK, as is just bailing on any patterns we don't feel confident enough to convert (unless the user passes the equivalent of a --best-effort flag).

from cti-stix-elevator.

The elevator always does the "best effort", but will also always print the warning message

from cti-stix-elevator.