oasis-open / cti-python-stix2 Goto Github PK

Found the following SDOs not enforcing required properties:

• ObservedData
  • 'objects' is required
• Report
  • 'published' is required
  • 'object_refs' is required

One additional change:

• Indicator
  • 'labels' not in correct order.

Something that was changed in the Bundle() function between v0.2 and v0.3 is introducing large delays in creating a bundle for very large bundles (greater than a few hundlred STIX objects).

In v0.2, building a bundle containing 1000 STIX objects would take a few seconds. In v0.3, it's taking 10+ minutes. It might be related to the Bundle() function trying to convert things to an OrderedDict.

As we move forward with the new STIX 2.1 objects. He have to make sure they work as intended and also keep our code coverage scores in par.

lang is a highly-suggested property for objects that contain strings meant for humans. Investigate automatic language detection, and build high level helpers.

Some libraries (If you find a better candidate, feel free to use it):

• https://github.com/saffsd/langid.py
• https://github.com/Mimino666/langdetect

I'm guessing this line should raise an instance of the exception class, not the class itself?

From #58

The API should throw an error if a granular marking is added to an object but one of its selectors references a property or list item that is not present on that object. Or if it doesn't throw an error, it should at least omit the invalid selector.

The culprit seems to be code that is intended to check for None, but is too general. See here.

That will prevent any property from being settable to False, empty lists, dicts, strings, zero, etc. It should check specifically for None if that's the intent.

We need to determine how to support STIX 2.0, 2.1 and later versions using the same library. Please use this issue to track ideas and progress surrounding this problem.

The transformations from different Confidence scales have been implemented. It would be nice to have higher level methods that could,

filter objects based on:

• A confidence integer value or interval. (55, 70-100, 60>)
• A string scale value. ("Med", "2 - Probably True")
  • Provide the name of the scale or function to use??

Note that, string scale values were implemented as case-sensitive strings.

Methods:

• Given a list of objects, return objects that matched the given parameters
  • Using the first method as building block, provide support for dict and Bundles

Other ideas?

bundle is not defined in OBJ_MAP, so calling stix2.parse() on a dict/JSON string representing a Bundle fails.

Also, the objects property of a Bundle does not convert dictionaries into actual python-stix2 objects. It should likely use a ListProperty(STIXObject) (where STIXObject is a class we haven't yet defined), or maybe a custom Property subclass, and re-use some of the logic from parse to detect the correct object type to construct.

We want to be consistent on the terminology.

Per conversation in #79 we should start by updating the current package structure for STIX 2.0

This refers to adding new extension types to existing observable object types.

Hi,

When attempting to generate a bundle, a failure message is created when passing an SDO with custom objects even with allow_custom=True set on the SDO object.

example:

v = factory.create( Vulnerability, name="Test Vulnerability", custom_field = "This is custom", allow_custom=True )

print Bundle(v)

Will result in the following output:

File "stix.py", line 142, in <module> print Bundle(v) File "/usr/local/lib/python2.7/dist-packages/stix2/core.py", line 51, in __init__ super(Bundle, self).__init__(**kwargs) File "/usr/local/lib/python2.7/dist-packages/stix2/base.py", line 121, in __init__ self._check_property(prop_name, prop_metadata, setting_kwargs) File "/usr/local/lib/python2.7/dist-packages/stix2/base.py", line 55, in _check_property kwargs[prop_name] = prop.clean(kwargs[prop_name]) File "/usr/local/lib/python2.7/dist-packages/stix2/properties.py", line 115, in clean valid = self.contained.clean(item) File "/usr/local/lib/python2.7/dist-packages/stix2/core.py", line 28, in clean parsed_obj = parse(dictified) File "/usr/local/lib/python2.7/dist-packages/stix2/core.py", line 94, in parse return obj_class(allow_custom=allow_custom, **obj) File "/usr/local/lib/python2.7/dist-packages/stix2/base.py", line 104, in __init__ raise ExtraPropertiesError(cls, extra_kwargs) stix2.exceptions.ExtraPropertiesError: Unexpected properties for Vulnerability: (custom_field).

The reason is that _STIXBase.__getattr__() returns None if the property doesn't exist. That's saying that the property exists, with value None. Instead, it needs to raise AttributeError.

Fixing this will reveal another bug here about the "granular_markings" attribute not existing. Attribute existence should be checked with hasattr().

Right now, custom types (custom STIX Objects, custom Cyber Observables, etc.) are global to the library.

Part of the conversation in #52 was whether we should have environment-specific custom types, but we didn't want to wait to merge that PR while we were discussing this. Hence this issue 😉 .

We should decide:

1. whether this is a good idea.
2. if so, how to implement it.

If this is a feature that's important to you, please give it a 👍 reaction so it can be prioritized accordingly. If you want to add more details in a comment, feel free to do that as well.

The localization in the except part of the try-except isn't assigned to the zoned variable. That means that the next line calling strftime() will error out since the variable hasn't been assigned. See here.

The Workbench API is designed to be a high-level interface to querying, creating, and interacting with STIX data. It builds on the explicit "Environment" concept by exposing a set of module-level functions that operate on an implicit, module-level Environment, without requiring users to be aware of the underlying Environment. (This encapsulation allows applications that need to support multiple Environments to do so, but does not require individual users to manage their own Environment).

Specifically for adding/pushing STIX objects to these DataStores/Sinks as they can now be supplied with STIX content of a variety of forms

From #58

The Bundle class doesn't perform any checks on what is added to its objects property.

We need to implement this check from the spec:

Objects in this list MUST be a STIX Object (SDO, SRO or Custom Object) or a Marking Definition object.

This may mean we need to refactor more code like we did for #23.

Hi,
As an exercise I'm starting a similar Scala library for STIX 2.1, at:

scalastix

According to the specs and as part of the common properties, labels is optional.
However, there are SDOs (e.g. Indicator) where it is required. Is this correct?
labels are optional but not when required!

I see that labels are not part of the python COMMON_PROPERTIES.
Should I do the same?

Hi! If I install stix2 with pip and try to parse json data from here https://github.com/oasis-open/cti-stix2-json-schemas/tree/master/examples/threat-reports I've got an error ValueError: Can't parse unknown object type 'bundle'!, but if I install this package with cloning from git and using python setup.py install everything works fine. So I wonder what is the problem here? Is it an old version on PyPi or what? Thanks!

Found a couple of Objects with missing properties or the property is not enforced:

• AutonomousSystem

  • number is required

• SocketExt

  • socket_descriptor is missing
  • socket_handle is missing

The STIX specification says:

The created timestamp MUST be precise to the nearest millisecond (exactly three digits after the decimal place in seconds).
The modified timestamp MUST be precise to the nearest millisecond (exactly three digits after the decimal place in seconds).

This is to avoid ambiguity in whether one timestamp equals another for purposes of versioning.

python-stix2 doesn't seem to always do this (unless I'm missing something).

This is an abstraction that will encompass a data source, data sink, and object factory, and potentially other state, in a unified interface for interacting with external STIX repositories.

When reading the standard (from this location), labels is a required property. The library enforced it which makes sense but we would recommend to allow an empty array/list. Currently, you need to create at least an empty label to make it valid. Could you update the library to allow empty array/list? Thank you.

Currently the code examples have output that is not correct due to errors with the external TAXII server we are using for tests. Currently we are holding off as author of external TAXII server was notified of the errors and has stated they will address them; and implied sooner rather than later. (they should just need to be re-executed within jupyter notebook and output should correct when the TAXII server is patched)

Create helpers for the Location object.

My thoughts include using the contents of a Location object to generate URLs that could be used on a map engine. As long as we know the format the engine uses it would be possible to support it.

For example,

• Google Maps: https://stackoverflow.com/questions/6018060/google-maps-url-i-want-to-create-a-url-to-google-maps-i-want-to-show-search-ne
  • https://developers.google.com/maps/documentation/urls/guide
• Bing Maps: https://msdn.microsoft.com/en-us/library/dn217138.aspx

This would let users specify defaults for some common properties and easily create STIX objects without needing to specify them every single time; the objects would automatically be created with the default value.

For example, you could specify a default created_by_ref value to point to the Identity SDO for your organization.

This one should be relatively simple. Given an arbitrary number of SDO/SROs, create helpers that would allow the creation of Opinions or Note. The helper needs to accept identifier strings or just passing Objects and extracting their id to populate the object_refs property.

Some ideas I had while writing documentation, for how to make data markings a little easier to use:

• [ ] Maybe StatementMarking() should return a MarkingDefinition object and not require the user to create both a MarkingDefinition and StatementMarking object. Would a user ever create a StatementMarking object without also creating a MarkingDefinition?
• We should add a shortcut to be able to call add_markings() (and the other marking functions) on STIX objects. So for example
  indicator.remove_markings('marking--id')
  instead of
  markings.remove_markings(indicator, 'marking--id')
  As we implement these, be explicit that they return new versions of the objects since they are immutable.
• stix2/__init__.py should import the functions from stix2.markings so they're in the top-level
  namespace.
• add_markings() and remove_markings() should support passing in a MarkingDefinition object, not just an ID string.

To reproduce:

 def test_custom_property_in_bundled_object():
        identity = stix2.Identity(
            name="John Smith",
            identity_class="individual",
            x_foo="bar",
            allow_custom=True,
        )
        bundle = stix2.Bundle(identity, allow_custom=True)

        assert bundle.objects[0].x_foo == "bar"
        assert '"x_foo": "bar"' in str(bundle)

Results in an error like this:

TypeError: '<' not supported between instances of 'NoneType' and 'int'

I believe this is because stix2.utils.find_property_index returns None for custom properties, which we should probably put at the end.

While working #98 I noticed even further problems with how the stores work. In addition to the fixes, I will adding:

• Proper ABC for DataStore, DataStore, and DataSink.
• The version positional argument was missing in a lot of places.

We have several list comprehensions in the form:

[x for x in <iterable>]

These can be replaced with just list(<iterable>)

There are some more complex ones such as

[x for x in <iterable> if <some predicate on x>]

which can be replaced by list(filter(lambda x: <predicate on x>, <iterable>))

The first example in https://github.com/oasis-open/cti-python-stix2/blob/master/README.rst does not verify with https://github.com/oasis-open/cti-pattern-validator/blob/master/README.rst

I suppose the correct example should be this.

indicator = Indicator(name="File hash for malware variant",
labels=['malicious-activity'],
pattern="[file:hashes.md5 = 'd41d8cd98f00b204e9800998ecf8427e']")

Add simple sanity check for the TAXIICollectionSource.get()/all_versions() calls, where the id of the STIX objects returned from the TAXII endpoint are checked to match

We should have links in the guide documentation (the Jupyter notebooks) to appropriate sections in the API reference documentation. So for example, when we mention MemoryStore, we can link to the API docs for that class.

In testing, I can get the links to work but sphinx spits out warnings.

See also: spatialaudio/nbsphinx#130.

In preparation for the Workbench layer, the Environment API should have some functions to explore/retrieve/dereference related objects. This would include something like created_by() (for created_by_ref) and relationships(). relationships() might have parameters like source_only, target_only, and relationship_type.

We will need to modify Filters to allow filtering on non-common properties first, however, as we cannot currently filter on source_ref and target_ref.

Steps:

• created_by()/creator_of()
• filters
• relationships()
• related_to()

example usage:

bundle      = Bundle(*bArgs, allow_custom=True) 
store.add(bundle)

Traceback (most recent call last):
  File "stix.py", line 166, in <module>
    store.add(bundle)
  File "/usr/local/lib/python2.7/dist-packages/stix2/environment.py", line 147, in add
    return self.sink.add(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/stix2/sources/filesystem.py", line 86, in add
    _check_path_and_write(self._stix_dir, stix_data)
  File "/usr/local/lib/python2.7/dist-packages/stix2/sources/filesystem.py", line 82, in _check_path_and_write
    f.write(str(Bundle(stix_obj)))
  File "/usr/local/lib/python2.7/dist-packages/stix2/core.py", line 62, in __init__
    super(Bundle, self).__init__(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/stix2/base.py", line 121, in __init__
    self._check_property(prop_name, prop_metadata, setting_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/stix2/base.py", line 57, in _check_property
    raise InvalidValueError(self.__class__, prop_name, reason=str(exc))
stix2.exceptions.InvalidValueError: Invalid value for Bundle 'objects': This property may not contain a Bundle object

FYI i'm using the patched code submitted in #73

Currently if you pass a list to the Bundle constructor it creates a list within a list:

        {
          "objects":
             [
               [
                   { },
                   { }
               ]
             ]
        }

If the first argument is a list, the constructor should just assign it to objects.

When this library outputs STIX objects as strings, the properties are put in alphabetical order. Their order should match that in the spec, with type as the first property, followed by id, etc.

We should pass patterns through the pattern validator to make sure users cannot create indicators with invalid patterns.

This was a holdover from when the DataStores did validation.

This is a meta-issue for various documentation improvements

• Add API documentation using Sphinx autodoc
• Ensure docstrings are aligned between classes like DataSource, DataSink, ObjectFactory and their "wrapper" classes like DataSource and Environment.