oasis-open / cti-stix-elevator Goto Github PK

If there’s enough data in the Package to populate a Report (a title, mostly) then create a Report. If it’s a bare package (no title), then don’t.

When description fields contain multi-line/formatted text (including HTML), whitespace present in the XML may not lead to "pretty" descriptions in JSON. This would essentially require doing description = ' '.join(description.split())

This should NOT be the default behavior.

Note that should not apply to whitespace between fields that get added to the STIX 2.0 description because there is no equivalent field in STIX 2.0 for some STIX 1.x field; those should continue to be separated by newlines.

relationships are missing target_ref and bundles are missing spec_version.

The status refers to the coverage of the AIS profile

related to #6

When a STIX 1.x observation was intended to be used as a pattern, the observed data object produced can be omitted from the final STIX 2.0 bundle. A command line option should control this. If some object is omitted, there would be a messages indicating that.

Handle the negate attribute of indicators and observations

• patterns don't have /s
• no buckets for bundle
• remove version property
• remove precision?

Handle when valid_from is not given, but value_until is

Many of the indicators are never accessed directly. They exist as a subexpression of some other indicator. Because we don't do expression substitution in STIX 2.0 (all pattern expressions are "flat") these indicators are extraneous, and make the result much larger (and complex) than it needs to be.

This would probably be something we can turn on or off using a command line argument.

This code is highly fragile, since the correct order of the tasks it performs is critical.

However, it must make multiple passes over the full content of the bundle - perhaps it could be more efficiently done

If left running the memory footprint will grow indefinitely. I investigated the issue with heapy.

The type taking most of the space is dictionaries:

	 Partition of a set of 513745 objects. Total size = 130115120 bytes.                                                   
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)                                             
     0  41241   8 63649752  49  63649752  49 dict (no owner)                                                          
     1 153068  30 12261576   9  75911328  58 str           
     2  88333  17  7873408   6  83784736  64 tuple         
     3  14425   3  6690568   5  90475304  70 list          
     4  24848   5  3180544   2  93655848  72 types.CodeType                                                           
     5   2422   0  3025168   2  96681016  74 dict of type  
     6  24896   5  2987520   2  99668536  77 function      
     7   9417   2  2636760   2 102305296  79 dict of 0x23d2080                                                        
     8   2423   0  2186576   2 104491872  80 type          
     9    774   0  2088336   2 106580208  82 dict of module

Investigation the dictionaries show that most were cybox objects:

>>> h[0].byrcs                                             
Partition of a set of 41241 objects. Total size = 63649752 bytes.                                                     
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)                                
     0      1   0 25166104  40  25166104  40 dict of cybox.utils.caches.DictCache                                     
     1   2350   6  7877200  12  33043304  52 dict of cybox.objects.file_object.File                                   
     2   5140  12  5386720   8  38430024  60 dict of cybox.core.observable.Observable                                 
     3   4397  11  4608056   7  43038080  68 dict of cybox.common.properties.HexBinary                                
     4   4397  11  4608056   7  47646136  75 dict of cybox.common.vocabs.HashName                                     
     5   3695   9  3872360   6  51518496  81 dict of cybox.common.properties.String                                   
     6   2047   5  2145256   3  53663752  84 dict of cybox.common.properties.UnsignedLong                             
     7    318   1  1969488   3  55633240  87 dict of module                                                           
     8   4397  11  1231160   2  56864400  89 dict of cybox.common.hashes.Hash                                         
     9   1124   3  1177952   2  58042352  91 dict of cybox.common.properties.AnyURI 

Code:

import stix2elevator

results = stix2elevator.elevate_file('Mandiant_APT1_report.xml')
print(results)

Traceback:

  File "/opt/venv_elevator/lib/python3.6/site-packages/stix2elevator/utils.py", line 21, in warn
    if options.msg_id_enabled(ecode):
  File "/opt/venv_elevator/lib/python3.6/site-packages/stix2elevator/options.py", line 137, in msg_id_enabled
    return msg_id in get_option_value("enabled")
TypeError: argument of type 'NoneType' is not iterable

• Python version == 3.6
• Stix 2 elevator version == 0.2.0
• STIX version of file being elevated == 1.2

• add ALONGWITH (and maybe new OR observation operator)
• insure that observation expressions contain comparison expressions that use AND are all related to the same object path root.
• use [] as appropriate

Convert to 'full' or omit (because 'full' is the default)

Lower case with -s. One specific example is kill chain names.

In version 1.0 of the elevator, we only implemented the conversion of a subset of the CybOX 2.x objects that could be represented in STIX 2.0. This subset was informed by the DHS NCCIC AIS profile for STIX 1.x.

For instance, X509 Certificates and UserAccounts are not handled by the elevator.

Some options are already implemented via a Boolean flag:

• retain the STIX 2.0 observed data object, even if the purpose of the STIX1.x observable was to specify an indicator pattern
• include STIX 1.x content that cannot be represented directly in STIX 2.0 in the description property
• use Incident and Infrastructure stubs that weren’t included in 2.0

Some other ones we may want to implement:

• turn on or off certain errors or warnings
• default identity for created_by_ref
• default timestamp
• how to handle CybOX 2.0 objects in STIX 1.x patterns that cannot be represented in STIX 2.0

According to the spec - the following is ok:

[ipv4-addr:value = '121.41.129.223' OR file:hashes:md5 = 'a4754be7b34ed55faff832edadac61f6']

but this is not:

[ipv4-addr:value = '121.41.129.223' AND file:hashes:md5 = 'a4754be7b34ed55faff832edadac61f6']

We need to check if when a indicator composite expression's operator is an AND that it doesn't contain multiple types.

Need to convert 1,000s of legacy STIX packages. Is this supported OOTB or do we have to iterate through each with a wrapper?

They are no longer part of STIX 2.0

This will affect both convert_pattern.py and convert_cybox.py

These are a priority since they appear in APT1.

Depends on #4.

Sometimes a ttp can't be converted to a STIX 2.0 object, which may cause a Relationship object to be missing a source or target ref. Maybe help users fix the issue by using the old STIX 1.x ttp id

clear_pattern_mapping has been renamed to clear_pattern_cache but the references in init.py @ 79 & 120 have not been updated.

Here is a snippet from fireeye pivy:

<stixCiqIdentity:Specification>
    <xpil:OrganisationInfo xpil:IndustryType="Finance Sector,Economic Sector,Trade Policy Sector,ISP/Telco Sector,Government Sector,Defense Contractor Sector"/>
</stixCiqIdentity:Specification>

which is converted to:

"sectors": [
    "finance-sector,economic-sector,trade-policy-sector,isp/telco-sector,government-sector,defense-contractor-sector"
]

Can we do better??

In STIX 1.x, observable objects can be related to other object which appear in other Observables.

In general, relationships between cyber observables are not supported in STIX 2.0, but there are some cases (domain_name related to an ip_address via the "resolved_to" relationship) that we might want to not just ignore

Most of this has been done, but a few fields still remain.

parameter observables, maybe turn into pattern expressions and put in description???

Currently, the elevator does the same thing, regardless of the correctness of the conversion - it generates the json, and a log of messages.

We could include command line arguments so the elevator fails, for instance, if the output doesn't pass the stix validator, or if certain elevator messages are produced.

In #4 we completed captured most of the CybOX 2.x types that are used in the AIS profile. However, there are several additional CybOX 2.x types that are convertible to STIX 2.0 that we haven't covered: e.g., various file extension, artifact, AS, X509, etc.

Hi everyone,

I encountered the following error while updating a STIX 1.2 report:

File "/home/XXX/.local/lib/python2.7/site-packages/stix2elevator/convert_pattern.py", line 621, in convert_windows_executable_file_to_pattern
s.entropy.condition,
AttributeError: 'Entropy' object has no attribute 'condition'

These are my installed pip packages:

• stix (1.2.0.4)
• stix-validator (2.5.0)
• stix2-elevator (1.0.0)
• stix2-patterns (0.4.1)
• stix2-validator (0.4.0)
• pycountry (17.5.14)
• lxml (3.8.0)
• cybox (2.1.0.14)

The part in the STIX report that causes the error (without it updates just fine):

<WinExecutableFileObj:Entropy> <WinExecutableFileObj:Value>7.74202363178</WinExecutableFileObj:Value> </WinExecutableFileObj:Entropy>

even if I change these lines and add a condition it won't succeed.

<WinExecutableFileObj:Entropy> <WinExecutableFileObj:Value condition="Equals">7.74202363178</WinExecutableFileObj:Value> </WinExecutableFileObj:Entropy>

Which Python style guide is this project following? Are there any particular PEP-8 exceptions that are being allowed? (Extra credit: Which pylint or flake8 codes are ignored?)

I know this is a proof-of-concept, and as such it might not need to have an answer to the above questions. However, I wanted to open the discussion, because PoCs sometimes get "promoted" to official releases, or "borrowed from" by other projects.

Also, as a potential contributor to this codebase, I'd like to follow whatever style guide is in force.

Thanks!

Any global variables should be a field in a singleton environment object - maybe working off of ElevatorOptions.

Love this project. Awesome, guys!

I was wondering, if I were to try to use this as a library, would it be thread-safe? I see some module-level dictionaries, which is why I ask.

If thread-safety isn't a goal for this project, I understand.

Thanks!

Only remove indicators that were involved ONLY as sub-indicators within composite indicator expressions

In my testing it seems that indicators don't elevate correctly and are output containing PLACEHOLDER.

Input:

<stix:STIX_Package xmlns:cyboxCommon="http://cybox.mitre.org/common-2" xmlns:cybox="http://cybox.mitre.org/cybox-2" xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2" xmlns:marking="http://data-marking.mitre.org/Marking-1" xmlns:simpleMarking="http://data-marking.mitre.org/extensions/MarkingStructure#Simple-1" xmlns:tlpMarking="http://data-marking.mitre.org/extensions/MarkingStructure#TLP-1" xmlns:TOUMarking="http://data-marking.mitre.org/extensions/MarkingStructure#Terms_Of_Use-1" xmlns:opensource="http://hailataxii.com" xmlns:edge="http://soltra.com/" xmlns:indicator="http://stix.mitre.org/Indicator-2" xmlns:ttp="http://stix.mitre.org/TTP-1" xmlns:stixCommon="http://stix.mitre.org/common-1" xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1" xmlns:stix="http://stix.mitre.org/stix-1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:taxii="http://taxii.mitre.org/messages/taxii_xml_binding-1" xmlns:taxii_11="http://taxii.mitre.org/messages/taxii_xml_binding-1.1" xmlns:tdq="http://taxii.mitre.org/query/taxii_default_query-1" id="edge:Package-0d6e49eb-35e8-43c7-824d-5aefbb387b40" version="1.1.1" timestamp="2016-02-09T12:35:51.135458+00:00">
    <stix:STIX_Header>
        <stix:Handling>
            <marking:Marking>
                <marking:Controlled_Structure>../../../../descendant-or-self::node()</marking:Controlled_Structure>
                <marking:Marking_Structure xsi:type="tlpMarking:TLPMarkingStructureType" color="WHITE"/>
                <marking:Marking_Structure xsi:type="TOUMarking:TermsOfUseMarkingStructureType">
                    <TOUMarking:Terms_Of_Use>www.malwaredomainlist.com | Malware Domain List - is a non-commercial community project. Our list can be used for free by anyone. Feel free to use it. 
</TOUMarking:Terms_Of_Use>
                </marking:Marking_Structure>
                <marking:Marking_Structure xsi:type="simpleMarking:SimpleMarkingStructureType">
                    <simpleMarking:Statement>Unclassified (Public)</simpleMarking:Statement>
                </marking:Marking_Structure>
            </marking:Marking>
        </stix:Handling>
    </stix:STIX_Header>
    <stix:Indicators>
        <stix:Indicator id="opensource:indicator-4c72329f-dc3a-4582-b872-3e6f469b2260" timestamp="2015-02-05T11:34:30.219739+00:00" xsi:type="indicator:IndicatorType" version="2.1.1">
            <indicator:Title>Compromised Site: : qwmlad.xyz:9290/openvpnadmin/popular.php?shopping=26</indicator:Title>
            <indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.1">IP Watchlist</indicator:Type>
            <indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.1">Domain Watchlist</indicator:Type>
            <indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.1">URL Watchlist</indicator:Type>
            <indicator:Description>This domain qwmlad.xyz:9290 (91.209.77.124) located in RU (Russia), has been identified as Malious [exploit kit] by www.malwaredomainlist.com. For more detailed infomation about this indicator go to [CAUTION!!Read-URL-Before-Click] [http://www.malwaredomainlist.com/mdl.php?search=qwmlad.xyz:9290].</indicator:Description>
            <indicator:Observable idref="opensource:Observable-0bc3ed1b-6ea7-4662-af0c-b6aef817cc42">
            </indicator:Observable>
            <indicator:Indicated_TTP>
                <stixCommon:TTP idref="opensource:ttp-e49fe46a-c86d-4b60-b619-2efef53073a4" xsi:type="ttp:TTPType"/>
            </indicator:Indicated_TTP>
            <indicator:Producer>
                <stixCommon:Identity id="opensource:Identity-f465129c-0151-4f74-8913-b80f6a1516a9">
                    <stixCommon:Name>www.malwaredomainlist.com</stixCommon:Name>
                </stixCommon:Identity>
                <stixCommon:Time>
                    <cyboxCommon:Produced_Time>2015-02-04T13:35:00+00:00</cyboxCommon:Produced_Time>
                    <cyboxCommon:Received_Time>2015-02-05T11:34:30+00:00</cyboxCommon:Received_Time>
                </stixCommon:Time>
            </indicator:Producer>
        </stix:Indicator>
    </stix:Indicators>
</stix:STIX_Package>

Output:
{u'valid_from': u'2015-02-05T11:34:30.219739Z', u'created_by_ref': u'identity--f465129c-0151-4f74-8913-b80f6a1516a9', u'description': u'This domain qwmlad.xyz:9290 (91.209.77.124) located in RU (Russia), has been identified as Malious [exploit kit] by www.malwaredomainlist.com. For more detailed infomation about this indicator go to [CAUTION!!Read-URL-Before-Click] [http://www.malwaredomainlist.com/mdl.php?search=qwmlad.xyz:9290].', u'created': u'2015-02-05T11:34:30.219Z', 'pattern': u'PLACEHOLDER:opensource:Observable-0bc3ed1b-6ea7-4662-af0c-b6aef817cc42', u'labels': [u'ip-watchlist', u'domain-watchlist', u'url-watchlist'], u'modified': u'2015-02-05T11:34:30.219Z', u'object_marking_refs': [u'marking-definition--77ff791e-6cb4-4f03-be04-6c748c453ef3', u'marking-definition--fc1c40e3-9ffd-45b4-8a25-c504898e6414', u'marking-definition--db89b6c0-a0db-4fec-b0d7-d3388b243b27'], u'type': u'indicator', u'id': u'indicator--4c72329f-dc3a-4582-b872-3e6f469b2260', u'name': u'Compromised Site: : qwmlad.xyz:9290/openvpnadmin/popular.php?shopping=26'}

This feature is not in STIX 2.0 as it was in STIX 1.x. Need to figure out how to compensate for this.

STIX 1.x allows "use" before "definition", in other words, using idrefs before the object has been defined and its id made available.

This causes much "post" processing to resolve idrefs once the content has been scanned.

Two pass processing might be more efficient

Right now get_name just takes the first name in a list of names. What should we do if there are others?