Duplicate extracted licenses when parsing from RDF about tools-python HOT 5 CLOSED

@xavierfigueroav Please attach the full output and the input files to the issue.

from tools-python.

@goneall I've added a link to the input file now.
I had already included the full output produced by parse_rdf.py. What do you mean exactly?

from tools-python.

In the input file, there are 4 extracted licenses inside some <licenseConcluded> tags. The same 4 extracted licenses are referenced (<member rdf:nodeID="ID"/>) inside a <licenseDeclared> tag.
Is that something wrong in the rdf file, something that should not happen? If yes, there was never any error in parsers. If not...
Both extracted licenses in <licenseConcluded> and extracted licenses in <licenseDeclared> are being added to Document.extracted_licenses by the following code (specifically, line 265):

That code can be split into a separate method with some changes for it to add the extracted licenses children of <SpdxDocument>. Maybe:

for lic, _, _ in self.graph.triples((None, None, self.spdx_namespace['ExtractedLicensingInfo'])):
    self.handle_extracted_license(lic)

I am no so familiarized with the SPDX RDF representation (e.g., I don't know whether all extracted licenses are always direct children of <SpdxDocument> or not), so this may be working on this and some files, but it may not be a general solution. If you think it is OK, I can fix it that way and submit a PR.

from tools-python.

I took a look at the RDF file and it looks valid.

The 4 extracted licenses are only defined once in the RDF and the same license is referenced in other parts of the document (the ID is used to reference the same license definition).

@xavierfigueroav It looks like you found the source of the problem above - it should not add the extracted license info on each reference, it should only add it once.

Your proposed solution is very similar to the approach we take in the Java tools - adding the extracted license information as a separate method.

Just a bit more context if interested (probably not needed to fix this specific bug) - for the RDF, there are 3 ways of referencing an object - a literal value (like a string or number), a URI to point to a predefined object and Anonymous Nodes which are references generated and are local to the RDF graph. For extractedLicenseingInfo definitions Anonymous Nodes are generated and referenced throughout the RDF graph. A good overview can be found at https://www.w3.org/TR/rdf-concepts/

from tools-python.

@goneall Thank you for the link, it was and will be helpful.

from tools-python.