I get the following error when running python atarashii.py -a wordFrequencySimilarity <file>:

Traceback (most recent call last):
  File "atarashii.py", line 213, in <module>
    main()
  File "atarashii.py", line 167, in main
    result = run_scan(scanner_obj, inputPath)
  File "atarashii.py", line 116, in run_scan
    return scanner.scan(inputFile)
  File "/home/xavierfigueroav/Documents/atarashi-project/atarashi/atarashi/../atarashi/agents/wordFrequencySimilarity.py", line 41, in scan
    processedData = super().loadFile(filePath)
  File "/home/xavierfigueroav/Documents/atarashi-project/atarashi/atarashi/../atarashi/agents/atarashiAgent.py", line 44, in loadFile
    self.commentFile = CommentPreprocessor.extract(filePath)
  File "/home/xavierfigueroav/Documents/atarashi-project/atarashi/atarashi/../atarashi/libs/commentPreprocessor.py", line 131, in extract
    data = json.loads(data_file)
  File "/usr/lib/python3.8/json/__init__.py", line 341, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not dict

I got this error when running the wordFrequencySimilarity agent, but since commentPreprocessor is used by all the others, this may be affecting the whole package.

The error occurs because the result of the function extract from Nirjas is passed in the method loads of the module json, but extract returns a dictionary and loads expects a string. See lines 129 and 130 in commentPreprocessor.py.

So the fix consists of removing the line 130 and passing data_file in licenseComment, instead of data.

Description

argparse is a parser for command-line options, arguments, and sub-command.
Read Docs: https://docs.python.org/3/library/argparse.html
Currently, argparse is used as command line parser in atarashi and we're planning to shift it to plac.
plac does the same thing with very less line of code.
Repo: https://github.com/micheles/plac

How to Solve

Read the plac documentation: http://micheles.github.io/plac/

Files to be changed

• atarashii.py
• build_deps.py
• imtihaan.py
• cosineSimNgram.py
• dameruLevenDist.py
• tfidf.py
• wordFrequencySimilarity
• evaluator.py
• commentPreprocessor.py
• license_clustering.py
• ngram.py
• licenseDownloader.py
• licensePreprocessor.py
• license_merger.py

Description

There is a script to evaluate the algorithms for Atarashi: evaluator.py

Currently, it scans the test files sequentially (One by One).
We have to parallelize the script by using multiprocessing, multithreading or something else to reduce the effective time of scanning.

How to solve

Use multiprocessing, multithreading in the main loop of evaluator.py

I tried running it after installing it with pip on python 3.6.9. Then, the following error occurs.
Are there additional modules I need to install?

$pip install atarashi 
$atarashi -h
Traceback (most recent call last):
  File "/home/soimkim/test/venv/bin/atarashi", line 5, in <module>
    from atarashi.atarashii import main
  File "/home/soimkim/test/venv/lib/python3.6/site-packages/atarashi/atarashii.py", line 26, in <module>
    from atarashi.agents.cosineSimNgram import NgramAgent
  File "/home/soimkim/test/venv/lib/python3.6/site-packages/atarashi/agents/cosineSimNgram.py", line 30, in <module>
    from atarashi.agents.atarashiAgent import AtarashiAgent
  File "/home/soimkim/test/venv/lib/python3.6/site-packages/atarashi/agents/atarashiAgent.py", line 27, in <module>
    from atarashi.libs.commentPreprocessor import CommentPreprocessor
  File "/home/soimkim/test/venv/lib/python3.6/site-packages/atarashi/libs/commentPreprocessor.py", line 23, in <module>
    import code_comment  # https://github.com/amanjain97/code_comment/
ModuleNotFoundError: No module named 'code_comment'

Create a unified entry point for every file using command line arguments instead of adding __main__ to every point. This will help in keeping code concise, easy to maintain and readable.

Add Exceptional Licenses from SPDX to license list of Atarashi.

The evaluation script should

• Allow to print a comparison table with all the algorithms supported by atarashi. You can find examples of comparison tables in #95 and #65.
• Allow to print a confusion matrix so that we can easily do error analysis to make decisions on how to improve current agents or implement new ones.

After the merge of #3 , the license findings are not correct. Need to be fixed later.

Currently atarashi can scan only files. If a directory is provided as input, it should be able to find all files under it and run the selected agent on them.
The results of each scan can be stored in a list and printed as a JSON array.

It will be preferred, however, to print results as they come maintaining the validity of the JSON array. So if someone is running a scan in interactive terminal, it should not give a feeling as nothing is happening.
It can be emulated as printing a starting [ followed by printing of scan result object {...} and a ,. The last result will not have a trailing , and a ] can be printed at the end of scan. This approach will eliminate the need of additional list to hold temporary results.

Generally, license contained in the source code file is either is short license itself or a block of large license which becomes difficult for the information retrieval algorithms and similarity finding algorithms to classify efficiently.

Please suggest how this should be resolved before implementing other IR (Information retrieval) algorithms.

We can improve the performance of atarahi, nirjas, and others by using Numba and RAPIDS by Nvidia. Regular NumPy, pandas, and other libraries are slow. Maximum amount of time is wasted in serialization, deserialization, pre-processing, transfer of memory between CPU and others. We can make it fast using Numba's parallel processing, JIT, and in built features which can even work on CPU. Also, most of the programs can be made even faster using RAPIDS' cuML, cuDF, dask, etc by executing everything through a GPU like pre-processing, vectorization, database query, serialization, deserialization, parallel processing, etc. The entire codebase can be translated without much hassle resulting in computational efficiency, higher accuracy, and lower memory usage.
This can ensure Atarashi's integration with FOSSology.
I have somewhat started with the work. Can I proceed with the same??
@hastagAB @GMishx

According to the code guidelines, it should follow the PEP8 coding style but the code doesn't follow PEP8 standards.
Thus, make the codebase PEP8 compliant.

Hi, I'm getting an error when trying to build (master branch) on Python 3.7:

Installing collected packages: code-comment
  Running setup.py develop for code-comment
    Complete output from command /home/rob/projects/atarashi/.venv/bin/python -c "import setuptools, tokenize;__file__='/home/rob/projects/atarashi/.venv/src/code-comment/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help
    
    error: option --user not recognized

I'm able to build when I run the command with sudo, but this shouldn't be necessary per my understanding.

We can migrate our current requirement.txt file to pipfile due to several reasons like:

• TOML syntax for declaring all types of Python dependencies.
• One Pipfile (as opposed to multiple requirements.txt files).
• A Pipfile is inherently ordered.

or, Refer: Why?

WHAT

Atarashi can use a lot of agents and can use a lot of similarity types.
But according to the previous tests, we observed that there is a need for improvement in speed of scanning the licenses.

Proposal

to be decided

I get the following error:

Traceback (most recent call last):
  File "setup.py", line 26, in <module>
    from atarashi.build_deps import download_dependencies
  File "/home/xavierfigueroav/Documents/atarashi-project/atarashi/atarashi/build_deps.py", line 32, in <module>
    from atarashi.license.licensePreprocessor import LicensePreprocessor
  File "/home/xavierfigueroav/Documents/atarashi-project/atarashi/atarashi/license/licensePreprocessor.py", line 31, in <module>
    from atarashi.libs.commentPreprocessor import CommentPreprocessor
  File "/home/xavierfigueroav/Documents/atarashi-project/atarashi/atarashi/libs/commentPreprocessor.py", line 30, in <module>
    from nirjas import extract as commentExtract, LanguageMapper
ImportError: cannot import name 'LanguageMapper' from 'nirjas' (/home/xavierfigueroav/Documents/atarashi-project/atarashi/.env/lib/python3.8/site-packages/nirjas/__init__.py)

This happens when running python setup.py build, after cloning the repository and installing the dependencies.

small issue: spelling maybe can get corrected at early stages https://www.thefreedictionary.com/deprecate

Thanks @mcjaeger for pointing the issue.

• Add comments to code to make it more readable
• Update README.md

nltk is used:

https://github.com/siemens/atarashi/blob/master/requirements.txt

but unfortunately not compatible with a GPL-2.0-only project:

https://www.apache.org/licenses/GPL-compatibility.html
https://www.gnu.org/licenses/license-list.html

Curling quotes (“,”, ‘, ’) are not filtered in comment extractor which results in some wrong results.

A more extensive listing of problematic word characters:

Whenever a invaid file path is provided to atrashi it generate the following error:
env) akshay@akshay-VirtualBox:~/atarashi/atarashi/evaluator$ atarashi -a tfidf Testfiles/APSL-style.html
Traceback (most recent call last):
File "/home/akshay/atarashi/env/bin/atarashi", line 8, in

sys.exit(main())

File "/home/akshay/atarashi/env/lib/python3.8/site-packages/atarashi/atarashii.py", line 123, in main
result = atarashii_runner(inputFile, processedLicense, agent_name, similarity, ngram_json, verbose)
File "/home/akshay/atarashi/env/lib/python3.8/site-packages/atarashi/atarashii.py", line 83, in atarashii_runner
result = scanner.scan(inputFile)
File "/home/akshay/atarashi/env/lib/python3.8/site-packages/atarashi/agents/tfidf.py", line 140, in scan
return self.__tfidfcosinesim(filePath)
File "/home/akshay/atarashi/env/lib/python3.8/site-packages/atarashi/agents/tfidf.py", line 112, in __tfidfcosinesim
processedData1 = super().loadFile(inputFile)
File "/home/akshay/atarashi/env/lib/python3.8/site-packages/atarashi/agents/atarashiAgent.py", line 44, in loadFile
self.commentFile = CommentPreprocessor.extract(filePath)
File "/home/akshay/atarashi/env/lib/python3.8/site-packages/atarashi/libs/commentPreprocessor.py", line 129, in extract
data1 = licenseComment(data)
File "/home/akshay/atarashi/env/lib/python3.8/site-packages/atarashi/libs/commentPreprocessor.py", line 42, in licenseComment
for id, item in enumerate(data[0]["multi_line_comment"]):
IndexError: list index out of range
(env) akshay@akshay-VirtualBoxatarashi/atarashi/evaluator$

### Instead It should Generate a simple error msg that the file path that was provided was wrong

Multi line comment extraction for following file types are still not working JS, PHP, and Python.

For example in Python

print """Some long
print message
"""
print 'Some different message'
"""
Actual comment
"""

In this case, the script returns print 'Some different message' as a comment and leave the Actual comment.

Description

• The link of Reference in Readme.md is broken
• In the about section of Readme.md the link of Atarashi webpage is given as http://fossology.github.io/atarashi which should be changed to https://fossology.github.io/atarashi (changing http to https )

How to reproduce

Visit the Reference link in Readme.md

Screenshots

If applicable, add screenshots to help explain your problem.

When Running atarashi -a {Any agent } -s {Similarities} following error produce:

Take a reference from here#80

Steps To Reproduce

--> Previously @Aman-Codes make changes in atarahii.py file #L83
->which was working fine but after This PR it again started creating the same error
-> or I think we need to implement the scan function inside the scan function like [here] or update the atarashii.py file

Comment extraction is not working for .h and .hpp. It is just copying the file content to the temporary files.

.h files can be parsed just like .c files and .hpp files like .cpp.

Description

The Evaluator commands are set for both two parameters i.e agent_name and similarity but some agents runs without similarity type also.

Example: for tfidfagent there are three commands :

1. With cosine similarity : atarashi -a tfidf -s CosineSim /path/to/file.c
2. With Score similarity : atarashi -a tfidf -s ScoreSim /path/to/file.c
3. Without any similarity : atarashi -a tfidf /path/to/file.c

The evaluator covers the first two cases and not the third one. The same goes for other agents.

How to fix

1. Goto the getCommand function of the evaluator
2. Write separate conditions as desired or manipulate the existing ones.
3. Test and verify if it's working.

Right now atarashi is using damerau_levenshtein_distance imported from pyxdameraulevenshtein in dameruLevenDist agent. The function is not that long and we do not if it will get removed. So, We can remove it from atarashi and write our own damerau_levenshtein_distance function to increase the overall speed of dameruLevenDist agent and make atarashi less dependent on other repository.
I have already started working on it. Can i proceed further?

Hello.

I've been playing around with some parameters of the TF-IDF agent.

I've found that if we stop using a threshold (cosine similarity >= 0.30) to filter the match results, the accuracy improves up to 3 points. However, filtering helps to reduce the compute time, since at the end of the search the results are sorted. See the piece of code I am talking about (specially lines 126 and 133):

Using the evaluation.py script, I've carried out some experiments:

• Row 1 shows the performance (speed and accuracy) of the current configuration of the TF-IDF agent using CosineSim as similarity measure.
• Row 7 shows how we can reach an accuracy of 62.% just by removing the threshold (cosine similarity >= 0.00). However, just removing the threshold makes the agent 2x slower, so I continued tuning the threshold holding the last value that produces 62.0% of accuracy, which is 0.16, showed in row 4.
• In order to continue decreasing the excecution time and increasing the accuracy, I tuned some parameters of the TfidfVectorizer. Setting max_df to 0.10 (default is 1.0) keeps the accuracy equal to 62.0%, but makes the agent 1.1x faster, showed in row 3.
  • Why does decreasing the max_df value increase the speed? It increases the speed because the vectorizer ignores all the terms that appear in more than the max_df percent of the documents (see docs), i.e., it ignores more frequent terms, so each document vector is shorter, making the cosine similarity easier to compute.
  • Why does decreasing the max_df value keeps the accuracy high? My explanation is that the terms that appear in most licenses do not help the algorithm distinguish licenses; rare terms are the ones that make licenses different between each other, so they are enough for the algorithm to do a good job.

I will be opening a PR for you to reproduce the results in row 3 and merge the changes if you consider them relevant.

Important notes:

• I've left out the speed times for all the other algorithms, because I ran those experiments in another context, so the comparison of time wouldn't be fair.
• All the results differ from the last report I could find out there. I do not fully understand why some of them are so different; probably changes in the test files or changes in the algorithms. Anyway, 62.0% is the new best result in both reports.
• My findings may help improve other agents that use thresholds, such as Ngram.
• This new state-of-atarashi performance 😅 may also push the goals of future agents implementations, since it would be the new baseline.