opensbom-generator / parsers Goto Github PK

Assess the parser to ensure the data looks as expected

We should write a couple of conformance tests to ensure the parsers are following the design we want to apply to all of them. I will start dumping some thoughts about what I think should be important to capture in the conformance suite:

Tests Across All Ecosystems:

• Ensure Uniform Package Representation
  We should create test repositories consisting of a simple project with a fixed set of dependencies. Maybe two direct dependencies one of them with a transient one. Once we replicate
• Ensure Uniform License and Copyright Detection
  License data is often found in code comments. We need to make sure all ecosystems are extracting the same data from their own language.
• Ensure Consistent Hashing
  One of the common problems in SBOMs is the wrong hashing of files. We should ensure all ecosystems produce the same hashes when looking at the same file while expressing their own ecosystem hashes correctly.
• Common Errors for Repeatable Failures
  While I'm not a fan of predefining errors, I think it is useful when dealing with plugin-like projects to factor out common errors. Things like emitting errors when the build environment is not ready or complete, execution errors when shelling out, etc are good candidates to unify.

It looks like the github workflow requires installation of all the package managers and using package managers to build applications. Add installation of package managers (eg: yarn) and building of test package (yarn install in the right location).

Assess the parser to ensure the data looks as expected

Assess the parser to ensure the data looks as expected

Related issues:
#20

Is it possible to generate SBOM from vcpkg package manager?

Assess the parser to ensure the data looks as expected

The current npm JSON parser implements JSON reading using ioutil.ReadJson which is deprecated. Plus, some considerations with package-json.lock file is not supported.

See opensbom-generator/spdx-sbom-generator#252 and opensbom-generator/spdx-sbom-generator#242 for more details.

Assess the parser to ensure the data looks as expected

Assess the parser to ensure the data looks as expected

Right now there is no PR template for contributors, should we add that?
If yes, I will make a PR for this.

After code migration we should assess the quality of the generated data in each ecosystem.

Let's use this issue to discuss metodology and track progress,

• #2
• #3
• #4
• #5
• #6
• #7
• #8
• #9
• #10

I think we should replace the command execution library with the command package in kubernetes-sigs/release-utils. That library is under constant maintenance and has more features and control.

➜  lsif-node git:(main) ✗ ./sbomgen -o . -f JSON
INFO[2023-08-27T14:53:17+08:00] Starting to generate SPDX ...                
INFO[2023-08-27T14:53:18+08:00] Using npm, current Language Version 6.14.11  
INFO[2023-08-27T14:53:18+08:00] Global Setting File path                     
INFO[2023-08-27T14:53:18+08:00] Parsing . for packages                       
FATA[2023-08-27T14:53:29+08:00] error creating SBOM, err: writing serialized document: json: error calling MarshalJSON for type *common.Supplier: failed to marshal invalid Supplier: {Supplier: SupplierType:Organization} 

The process will break if either the SupplierType or Supplier field is empty.
Maybe we need some default values (both name and type) for avoiding this, not only assign a name. Like:

Assess the parser to ensure the data looks as expected

Right now, the parsers try to make external network calls for fetching package data.

nuget

https://github.com/opensbom-generator/parsers/blob/main/nuget/helpers.go#L14

Used to fetch:

• nuget spec
• checksum

pip

https://github.com/opensbom-generator/parsers/blob/main/pip/worker/pypi.go#L85

Used to fetch:

• author details
  • There's no need to make an external network call here since this information is available with pip's metadata.
• checksum
• download url

Bonus:

For all dependency managers, pip offers a command pip inspect to get metadata
for the current environment which contains package metadata, platform information(this can be used for #25) and a lot more.

Assess the parser to ensure the data looks as expected

When analyzing the dependencies of projects, the effective dependencies can change based on what platform the build is targeting. In order to cover those cases, we need to be able to do two things from any of the parsers:

1. Give the parsers a unified way of knowing which platform they are running on and
2. Create an option to specify the target platform so that each parser can pass a platform slug to the underlying tooling

This umbrella issue tracks the progress of the code migration from the generator repository to
the new standalone repository according to the plan agreed upon in the Oct 19th community meeting.

• Import relevant ecosystem files from the original codebase preserving their history
• Initialize new go module
• Update parsers to new import path
• Import rest of required files
• Update packages to use the newly migrated general modules

Assess the parser to ensure the data looks as expected

The reader package seems to be a little bit overkill, maybe we should deprecate it.

/cc @cpanato

Assess the parser to ensure the data looks as expected

The data model for the OpenSBOM parsers is broken If you have a project with more than one top-level module/package, it cannot be represented in the current model :(

Say for example this one: https://github.com/rust-secure-code/cargo-geiger/blob/master/Cargo.toml

I don't know if we can fix this with the current model, it will require a v2 API.

Assess the parser to ensure the data looks as expected