threatgrid / ctim Goto Github PK

In ctim.schemas.common, rename ctia-schema-version to ctim-schema-version

Flanders should be able to generate model.dot, and running "lein doc" should trigger an update of the PNG file.

In support of threatgrid/ctia#212

Doesn't look like it made it into code.
intial_compromoise

Coordinate with Emmett Koen on enriching the intended effect vocabulary that we use in TTPs, Actors, etc.

Need to allow people to make relations between objects (source, relation, destination)

Should contain:

• Source: URI to origin object
• Relation: a new vocabulary
• Destination: The subject object

Iterate on improving the markdown that is generated by the flanders type definitions

• Nested maps should be documented un-nested with links in the top level map
• References should be markdown links, not just URIs
• All maps should have a name, and most should have descriptions
• Any URL in a description should be a markdown link
• Add :comments to the nodes (developer notes in the documentation)
• Truncate long schemas, like "(enum a b c ...", which can be done since the values are listed

Also do some simple refactoring

• Standardize map definitions (syntax evolved during conversion of CTIM)
  • Use [Type] in entries instead of (seq-of Type)
  • Use map/map-of/def-map-type consistently
  • Use #{Type ...} in entries instead of (enum #{Type ...})
  • def-entity-type should allow references to be defined

As was done in threatgrid/ctia#322.

It needs to use long IDs for references. There is new code in CTIA that does this, but it won't work in CTIB (because it depends on properties). Refactor and fix.

CTIM TTPs currently reference a vector of Indicators:
https://github.com/threatgrid/ctim/blob/master/src/ctim/schemas/ttp.clj#L112

STIX TTPs have no such reference:
http://stixproject.github.io/data-model/1.2/ttp/TTPType/

The TG TTP data currently creates empty vectors for all of its new TTPs in threatbrain-engine:
https://github.com/threatgrid/threatbrain-engine/blob/master/src/threatbrain/engine/iocs.clj#L226

Should we remove this reference from our model?

Based on the spike-schema-tree branch.

• Generate plumatic schemas on-demand
• Generate model documentation

Probably need to setup a clojars group.

https://github.com/clojars/clojars-web/wiki/Groups

The ctia.schemas.common/relations-map should be alphabetized by key. It is hard to find things when manually searching the code.

We need to be judicious how we use nested documents for performance reasons. Craig's asked us to review our current use of nested documents to represent observables within the sightings, to make sure it's the best approach.

#27 had some comments that should be looked at (to address or ignore)

Add cyber kill chain to TTP schema.

• relationships.cljc should be references.cljc
• observed_relationship.cljc should be for observable relationships
• the entity ObservedRelationship should just be Relationship in relationship.cljc

Since conditional schema are "hard" in swagger, and static types languages, I am going to suggest we use per specification keys:

:specification [{:type "threatbrain" :threatbrain_specification ....}] as opposed to:

:specification [{:type "threatbrain" :specification ...}]

Update doc/graph/model.dot and generate the the associated PNG file.

Depends on #65 (which will copy it over from CTIA).

Print out example JSON formatted examples on demand

Not under /test, because they will be used by CTIA.

Convert the whole project to cljc so it can be used from clojurescript

When describing generated schemas (as in flanders.schema/describe), clean up the descriptions for swagger (because descriptions are markdown and it looks ugly in swagger UIs).

At a minimum...

• Truncate at some configured length (default 50?)
• Replace markdown links with just the text of the link

Incorrect type used Indicator -> CompositeIndicatorExpression.

On line 46 of Indicator, there is a wrong type. It should be f/enum, not f/eq.

Intended effects are often multidimensional. Ours should be a vector.

We want a way to explain how many times we "saw" an observable or indicator, even if we just create once object. This is important with really high volume indicators, like C&C infrastructure.

The value should be coerced to 1 if not provided. We want to always store it with a value.

In flanders, support #{Str} type schemas

Iterate on the CTIM documentation

The ctim.schemas.identity ns is specific to CTIA.

In ctim.schemas.common/stored-schema, we add common fields that should be in all StoredEntity type schemas. The :version field, required on all stored entities, should be set in this fn.

• rename observed_relationship to Relationship
• rename the current relationship namespace to "Reference"
• put the ObservableRelations back into Sightings (that is observabe to observable relations with that fixed vocab

(s/defschema NewSighting
  (st/merge
   Sighting
   c/NewBaseEntity
   (st/optional-keys
    {:count s/Int
     :confidence v/HighMedLow})))

as a result generative tests fail.

It is removed from STIX 2.0, from all objects.

(s/defschema Resource
  "See http://stixproject.github.io/data-model/1.2/ttp/ResourceType/"
  (st/optional-keys
   {:tools (describe c/Tool "The tool leveraged by this TTP")
    :infrastructure (describe
                     Infrastructure
                     "infrastructure observed to have been utilized for cyber attack")
    :personas c/Identity}))

Here the :toolskey is described as a single item, should we rename the key to toolor change c/Toolto [c/Tool] ?

It needs a description, proper copyright, etc.

We need to test CTIM on travis:

• make a travis config file that runs the tests for both clj and cljs.

Some basic examples that test the schemas. When the test fail, that should indicate compatibility breakage and hint that a version bump will be required.

It is currently :ttp_id, which doesn't match the naming convention.

Matching URL IDs with hyphens fails.

For example, in ctim.domain.id:

(re-matches long-id-re "http://localhost:3001/ctia/exploit-target/exploit-target-d51dfc7b-df40-46a4-9b06-c396e3dfdbcf")

Results in nil.

Currently it only generates them for maps

Both JSON and CURL examples should be generated by flanders. Each type could have a default way of generating its example value (like the string types could be the 'str of the key), but it could also be overridden with an :example value on the type instance.

We want to return an error when references are not URLs in CTIA. Tests that use generated entities (that contain references) get random strings in the reference and ID fields. The current generators are not specific enough to create sufficiently accurate examples of our domain objects.

In other words, don't just take a schema and generate samples.

Related to threatgrid/ctia#212

Indicator schema change

related to #378

Currently our entity fixture generators only output the simplest scenario to match a schema, thus we can't simulate any of the optional keys.

Make it so that our entity schema generators output entities with most of their optional keys

• To version 0.22.11
• Remove the CTIA exclusion