Code Monkey home page Code Monkey logo

go-license-detector's Introduction

WE CONTINUE THE DEVELOPMENT AT go-enry/go-license-detector. This repository is abandoned, and no further updates will be done on the code base, nor issue/prs will be answered or attended.

go-license-detector GoDoc Build Status Build status codecov Go Report Card

Project license detector - a command line application and a library, written in Go. It scans the given directory for license files, normalizes and hashes them and outputs all the fuzzy matches with the list of reference texts. The returned names follow SPDX standard. Read the blog post.

Why? There are no similar projects which can be compiled into a native binary without dependencies and also support the whole SPDX license database (≈400 items). This implementation is also fast, requires little memory, and the API is easy to use.

The license texts are taken directly from license-list-data repository. The detection algorithm is not template matching; this directly implies that go-license-detector does not provide any legal guarantees. The intended area of it's usage is data mining.

Installation

export GO111MODULE=on
go mod download
go build -v gopkg.in/src-d/go-license-detector.v3/cmd/license-detector

Contributions

...are welcome, see CONTRIBUTING.md and code of conduct.

License

Apache 2.0, see LICENSE.md.

Algorithm

  1. Find files in the root directory which may represent a license. E.g. LICENSE or license.md.
  2. If the file is Markdown or reStructuredText, render to HTML and then convert to plain text. Original HTML files are also converted.
  3. Normalize the text according to SPDX recommendations.
  4. Split the text into unigrams and build the weighted bag of words.
  5. Calculate Weighted MinHash.
  6. Apply Locality Sensitive Hashing and pick the reference licenses which are close.
  7. For each of the candidate, calculate the Levenshtein distance - D. the corresponding text is the single line with each unigram represented by a single rune (character).
  8. Set the similarity as 1 - D / L where L is the number of unigrams in the quieried license.

This pipeline guarantees constant time queries, though requires some initialization to preprocess the reference licenses.

If there are not license files found:

  1. Look for README files.
  2. If the file is Markdown or reStructuredText, render to HTML and then convert to plain text. Original HTML files are also converted.
  3. Scan for words like "copyright", "license" and "released under". Take the neighborhood.
  4. Run Named Entity Recognition (NER) over that surrounding context and extract the possible license name.
  5. Match it against the list of license names from SPDX.

Usage

Command line:

license-detector /path/to/project
license-detector https://github.com/src-d/go-git

Library (for a single license detection):

import (
    "gopkg.in/src-d/go-license-detector.v3/licensedb"
    "gopkg.in/src-d/go-license-detector.v3/licensedb/filer"
)

func main() {
	licenses, err := licensedb.Detect(filer.FromDirectory("/path/to/project"))
}

Library (for a convenient data structure that can be formatted as JSON):

import (
	"encoding/json"
	"fmt"

	"gopkg.in/src-d/go-license-detector.v3/licensedb"
)

func main() {
	results := licensedb.Analyse("/path/to/project1", "/path/to/project2")
	bytes, err := json.MarshalIndent(results, "", "\t")
	if err != nil {
		fmt.Printf("could not encode result to JSON: %v\n", err)
	}
	fmt.Println(string(bytes))
}

Quality

On the dataset of ~1000 most starred repositories on GitHub as of early February 2018 (list), 99% of the licenses are detected. The analysis of detection failures is going in FAILURES.md.

Comparison to other projects on that dataset:

Detector Detection rate Time to scan, sec
go-license-detector 99% (897/902) 13.5
benbalter/licensee 75% (673/902) 111
google/licenseclassifier 76% (682/902) 907
boyter/lc 88% (797/902) 548
amzn/askalono 87% (785/902) 165
LiD 94% (847/902) 3660
How this was measured
$ cd $(go env GOPATH)/src/gopkg.in/src-d/go-license-detector.v3/licensedb
$ mkdir dataset && cd dataset
$ unzip ../dataset.zip
$ # src-d/go-license-detector
$ time license-detector * \
  | grep -Pzo '\n[-0-9a-zA-Z]+\n\tno license' | grep -Pa '\tno ' | wc -l
$ # benbalter/licensee
$ time ls -1 | xargs -n1 -P4 licensee \
  | grep -E "^License: Other" | wc -l
$ # google/licenseclassifier
$ time find -type f -print | xargs -n1 -P4 identify_license \
  | cut -d/ -f2 | sort | uniq | wc -l
$ # boyter/lc
$ time lc . \
  | grep -vE 'NOASSERTION|----|Directory' | cut -d" " -f1 | sort | uniq | wc -l
$ # amzn/askalono
$ echo '#!/bin/sh
result=$(askalono id "$1")
echo "$1
$result"' > ../askalono.wrapper
$ time find -type f -print | xargs -n1 -P4 sh ../askalono.wrapper | grep -Pzo '.*\nLicense: .*\n' askalono.txt | grep -av "License: " | cut -d/ -f 2 | sort | uniq | wc -l
$ # LiD
$ time license-identifier -I dataset -F csv -O lid
$ cat lid_*.csv | cut -d, -f1 | cut -d"'" -f 2 | grep / | cut -d/ -f2 | sort | uniq | wc -l

Regenerate binary data

The SPDX licenses are included into the binary. To update them, run

make bindata.go

go-license-detector's People

Contributors

adracus avatar aperezg avatar campoy avatar dsymonds avatar erizocosmico avatar johanbrandhorst avatar lafriks avatar marclop avatar mcuadros avatar sebbonnet avatar smola avatar tevino avatar vmarkovtsev avatar zurk avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.