Creation of statements

As a user that found information about a particular vulnerability, I want to create a corresponding statement; after having checked if other statements about that vulnerability exist already in one of my sources, I want the statement to be validated and then published on my own vulnerability repository.
After that, I also want to produce a script to import to my Steady backend this new information, reconciled with other statements about the same vulnerability (if they exist).

Examples (tentative):

kaybee lookup <VULN_ID>
kaybee create <VULN_ID>
kaybee push --validate
kaybee export  --target steady   # this produces steady.sh
tail -n 200 .kaybee/mergelog
bash ./steady.sh

Automated aggregation of vulnerability data

In an automated script, run periodically, all sources are to be pulled and merged using a suitable policy; for the statements that can be automatically reconciled, a script to import into Steady must be generated and run automatically. For the statements that could not be reconciled, a notification must be produced to trigger manual intervention.

# in crontab
kaybee pull && kaybee merge
kaybee prune -r 30   # purge stale local clones
kaybee export --target steady && bash ./steady.sh

Manual reconciliation

After having received a notification that a certain group of statements could not be reconciled, I want to inspect the MergeLog to determine what was the reason. I can then craft a new statement to override the necessary elements of the existing (3rd-party) statements and push to my own statement repository.

tail -n 200 .kaybee/mergelog
kaybee merge -p manual
kaybee push --validate

Does this project have any relationship with the OWASP dependency check project? See:

• https://www.owasp.org/index.php/OWASP_Dependency_Check

Specifically:

1. Functionally, is it possible to leverage this information when performing a check using the aforementioned OWASP tooling, and

2. technically, can this type of data store be accessed from the OWASP dependency check tool?

The -f flag of the setup command is simply ignored.

We could use something like: https://github.com/git-chglog/git-chglog

As per the doc - https://github.com/package-url/purl-spec, PURL should be something like - pkg:maven/org.apache.xmlgraphics/[email protected] but it is generated as pkg:maven://org.apache.xmlgraphics/[email protected] which is invalid.

For some reason, some (transitive?) dependencies are failing the build. Note: I cannot reproduce locally.

The NVD does contain fix-commits for some CVEs: extract them and represent them as statements.

Extraction of the existing commit pre-processing (feature extraction) code and its restructuring into a separate module.

https://github.com/SAP/project-kb#installing-the-kaybee-tool , the link mentioned there is not working.

Re-use this implementation that is part of eclipse/steady to have a local replica of the NVD feeds.

Heads-up: most likely, steady will drop that module in the future, we can just clone it in project-kb and continue from there.

A comand like create-statement should provide a UI to guide the user in entering the necessary data. This could be implemented as a multi-step form, on the CLI, in a browser-based app, or both.

It is not possible to generate the correct steady.sh file when using the attached statement.yaml file.
Exports does not raise any error but CVE data are not included as expected.

steady.sh.txt

statement.yml.txt

• go version go1.14.2 darwin/amd64
• Using the test file from KaybeeConf with backend URL http://localhost:8033/backend/

Command with error -

kaybee import -c kaybeeconf.yaml -v
Using config file: /Users/I334616/fork/project-kb/kaybee/testdata/conf/kaybeeconf.yaml
Importing vulnerability data from http://localhost:8033/backend/ (using 0 workers)
2020/06/26 09:37:13 json: cannot unmarshal object into Go value of type []*model.Bug

At each PR, PUSH: make sure that the documentation (gh-pages) are built automatically correctly (Optional: check that they contain no dead links).

kaybee config for steady generates java properties metadata file for commits rather than json file
Example -
kaybee export -c kaybeeconf_steady.yaml -t steady -f .kaybee/imported/HTTPCLIENT-1803/statement.yaml -o test2.sh
./test2.sh
cd HTTPCLIENT-1803
tree -L 2

.
 ├── 0554271750599756d4946c0d7ba43d04b1a7b2
  │   ├── after
  │   ├── before
  │   └── meta.properties //this should be a json file
 └── metadata.json

Scenario

A statement s_1 and a statement s_2, from sources S_1 and S_2 respectively, are conflicting. With some policy (or via manual intervention) they are reconciled and the result is statement s_3.

The next day, kaybee merge is run again, how to deal with the same conflict? Shall we keep a pointer to the last commit that was considered from each source so that reconciliation is not done again on the exact same set of conflicting statements?

Kaybee setup fails with error -
Tested on MacOS

kaybee setup
Running setup...
Non-interactive mode
[+] Running Setup task
2020/06/26 14:15:52 stat /home/*******/devel/project-kb/kaybee/internal/tasks/data/default_config.yaml: no such file or directory

Please refer to the diagram below:

Note: the label on the database at the bottom of the diagram is wrong: its contents are pre-processed commits, not vectorial representations of commits (which, for most features, depend on the query).

Kaybee statements should be validated automatically when we create a pull request
I can think of these basic validations -

1. Validate Git Commit Id
2. Validate Git branch
3. Validate PURL for affected artifacts
4. Validate Git/Svn Repository

Command kaybee setup is supposed to be used to generate a config file, however it won't run complaining that no config file exists (unless there is one).

I was checking the statement file for vulnerability CVE-2018-11248 and I noticed it points to commit lingochamp/FileDownloader@ff240b8 while the vulnerability seems to be fixed at lingochamp/FileDownloader@b023cc0.

There are several steps not documented (including installation of dependencies that are not in the requirements.txt file):

• sklearn
• spacy (+language model to be installed separately)
• matplotlib
• streamlit
• (....)

The strict policy refuses to reconcile statements about the same vulnerability that come from different sources.

We need another policy that is able to automatically reconcile statements about the same vulnerability.

What is the expected behaviour of such policy?

The current bottleneck in Prospector runtime is parsing the .git folder to an SQL database. We might take a look at other solutions, which parse .git folders and try them and learn from them to improve the processing speed of this initial step. Example projects:

• gitbase seems abandoned
• gitql looks good, but probably not optimized
• askgit seems promising, adds github API support (in beta)

Generate license and copyright information with the 'Reuse' tool of the Free Software Foundation Europe (FSFE)
To adopt external standards and best practices, the 'Reuse' tool of FSFE must be used to generate copyright and license information

TODOs:

• Generate the required information with the 'Reuse' tool as described here.
• Remove the notices file which was used in the past to store the SAP copyright (the file might be named NOTICES or NOTICES.txt).
• Remove the License section from the README.md file.

kaybee could check if a newer version exists on GH.
Separate command? Should also download?
Of course, the user should be able to opt-out/disable.

Currently a source is a 3-tuple:

• repository url
• branch
• rank

It should become a 4-tuple and include also:

• path

Example:

sources:
  - repo: https://github.com/sap/project-kb
    path: vulnerability-data
    branch: master
    rank: 10

This will allow to point to specific folders in the git repository and ahve separate "sources" in the same branch of the same repo.

If the binary files of the sqlite DB are not present (they should not be in Git), Prospector fails to create them.

When processing the same CVE twice in a row, the second run aborts with the following:

Traceback (most recent call last):
  File "main.py", line 220, in <module>
    plac.call(main)
  File "/home/i064196/.local/share/virtualenvs/prospector-_0gS2ZiV/lib/python3.6/site-packages/plac_core.py", line 367, in call
    cmd, result = parser.consume(arglist)
  File "/home/i064196/.local/share/virtualenvs/prospector-_0gS2ZiV/lib/python3.6/site-packages/plac_core.py", line 232, in consume
    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
  File "main.py", line 153, in main
    advisory_record.gather_candidate_commits()
  File "/home/i064196/devel/project-kb/prospector/rank.py", line 781, in gather_candidate_commits
    self.validate_database_coverage()
  File "/home/i064196/devel/project-kb/prospector/rank.py", line 751, in validate_database_coverage
    database.add_repository_to_database(self.connection, self.repo_url, self.project_name, verbose=self.verbose)
  File "/home/i064196/devel/project-kb/prospector/database.py", line 439, in add_repository_to_database
    {'repo_url':repo_url, 'project_name':project_name})
sqlite3.IntegrityError: UNIQUE constraint failed: repositories.repo_url

Add a new target to kaybee export e.g., create-changed-source-code, to generate a script that

• clones the repositories and created the before and after folders for each vulnerability
• compresses the created folders into an archive 'changed-source-code.tar.gz'

Note: the commit folders containing before and after should be removed after being compressed.

Compared to the existing steady target, no metadata.json nor invocation of kb-importer are required.

This will allow us to keep the general notes field for text that concerns the vulnerability itself, whereas information about where and how the fix commits were found can be kept in the new field.

When merge cannot reconcile conflicting statements, the user will be asked to reconcile them manually.
This could be a separated command, such as: kaybee reconcile <vuln-id> or just a special policy that can be called as usual:
kaybee merge -p interactive.

Clean up the top-level folder and make it independent of kaybee and prospector (no go or python stuff).
Currently the go.mod file of kaybee is in the root folder.

When we do a kaybee export for steady with a statement containing python Vulnerability, steady also needs __init__ files to be checkout as part of the bash generated script. These __init__ files are then used in Steady to create valid python constructs.

The temporary cloned repositories take a lot of space. Delete it as soon it is consumed by KB-Importer
https://github.com/SAP/project-kb/blob/master/kaybee/internal/tasks/data/default_config.yaml

We could keep in the config file just a configuration that says in which folder the tool can find "exporter definitions"; these will be separated one-file-per-target, so that the main configuration is much leaner. Also, the "standard" exporters could be packaged with the binary itself or created upon kaybee setup

For certain vulnerabilities, the branch info appears in the commit id field (concatenated to the commit, sha:branch)
e.g.:
id: 3.0.x
commits:

• id: aaa97cb1:3.0.x

CVE-2016-6812 CVE-2016-8739 CVE-2016-1000031

OS: macOS
kaybee merge take long even with one single repository
kaybee version 0.6.15

I used the git cloned file system to do the tests. kaybeeconf.yaml contains one single repo -
repo: file:///*********

Git cloned file system with tar containing 577 vulnerabilities -

kaybee pull
date && kaybee merge && date   
Fri Nov  6 11:20:33 CET 2020
......
Fri Nov  6 11:45:36 CET 2020

It took ~25mins

Git cloned file system without tar containing 720 vulnerabilities -

kaybee pull
date && kaybee merge && date   
Fri Nov  6 12:04:08 CET 2020
......
Fri Nov  6 12:17:47 CET 2020

It took ~13mins

kaybee version does not show any info.
The Makefile needs an update.

This component is supposed to abstract the storage of pre-processed commits so that they can be retrieved efficiently
when predicting.

Right now this is achieved with the bash script generated by kaybee export -t steady-with-changed-source-code
but it's more logical to allow the user to achieve the same with something like: kaybee import --make-tarballs
or kaybee push --make-tarballs.

https://github.com/SAP/project-kb/blob/master/kaybee/internal/tasks/data/default_config.yaml#L221

Certain commit features can only be computed based on a particular query (=they depend on the vulnerability
advisory record) and as such cannot be pre-computed and persisted in the DB.

We need a module that contains functions to extract such features and to augment datamodel.Commit instances,
so that filter_rank can consider them all.

This functionality could be in the commit-preprocessor package or in the filter_rank (i would prefer the former,
although in that case a more general name for the package would be advised).

A few good first example features:

• does the commit contain a reference to the vuln id in the commit message?
• does the commit changes a path that matches the "filepath" entities in the AdvisoryRecord?
• distance in time between commit message and advisory publication date
• commit falls in the commit interval determined based on the tags that (seem to) correspond to the versions mentioned in the advisory (yes, this is the most complex :-P )

See prospector legacy for more ideas

In view of an architectural overhaul of prospector, we need to clarify which use cases it is meant to serve.
This will guide the design of the APIs of a prospector backend component.

There can be a scenario where a user wants to import specific vulnerabilities.
I saw that there is no import capability where users can import specific vulnerabilities. It has -n parameter where I can specify the number of vulnerabilities to be imported but cannot specify one or more vulnerability IDs.

When I run kaybee setup, a default conf file is generated. The new changes are not seen as part of the conf file - d6a59122b21a615e2455815952ba2f3c91d7cff7

One additional source of statements could be the list of known malicious packages maintained at https://github.com/dasfreak/Backstabbers-Knife-Collection.

It contains a file package_index.csv with the following columns: Type, Package Name, Affected Version, Published, Reported, Sample, Injection Component, Obfuscation, Trigger, Conditional, Targeted OS, Objective, Details, Source, Comment, Typo Target, Campaign, Location of malicious snippet. See here for a detailed description of those columns.

The first three columns can be used to create one or more PURLs for artifacts, (some of) the other columns can be used for the description and references.