Code Monkey home page Code Monkey logo

fossdriver's Introduction

FOSSology

Gitpod ready-to-code GPL-2.0 CII Best Practices Coverage Status Slack Channel GitHub release (latest by date) YouTube Channel REUSE status

About

FOSSology is an open source license compliance software system and toolkit. As a toolkit, you can run license, copyright, and export control scans from the command line. As a system, a database and web UI are provided to give you a compliance workflow. In one click you can generate an SPDX file or a ReadMe with all the copyrights notices from your software. FOSSology deduplication means that you can scan an entire distro, rescan a new version, and only the changed files will get rescanned. This is a big time saver for large projects.

Check out Who Uses FOSSology!

FOSSology does not give legal advice. https://fossology.org/

Requirements

The PHP versions 7.3 and later are supported to work for FOSSology. FOSSology requires Postgresql as the database server and apache httpd 2.6 as the web server. These and more dependencies are installed by utils/fo-installdeps.

To install Python dependencies, run install/fo-install-pythondeps.

Installation

FOSSology should work with many Linux distributions.

See https://github.com/fossology/fossology/releases for source code download of the releases.

For installation instructions see Install from Source page in Github Wiki

Docker

FOSSology comes with a Dockerfile allowing the containerized execution both as a single instance or in combination with an external PostgreSQL database. Note: It is strongly recommended to use an external database for production use since the standalone image does not take care of data persistency.

A pre-built Docker image is available from Docker Hub and can be run using the following command:

docker run -p 8081:80 fossology/fossology

The docker image can then be used using http://IP_OF_DOCKER_HOST:8081/repo user fossy password fossy.

If you want to run Fossology with an external database container, you can use Docker Compose, via the following command:

docker-compose up

Docker Compose is a tool that allows you to define and run multi-container applications using a YAML file. FOSSology provides a docker-compose.yml file that defines three services: scheduler, web, and db.

The scheduler service runs the FOSSology scheduler daemon, which handles the analysis tasks. The web service runs the FOSSology web server, which provides the web interface. The db service runs a PostgreSQL database server, which stores the FOSSology data.

The docker-compose up command starts all the three services at once.

The FOSSology web service allows you to configure its database connection using some environment variables. These variables are defined in the docker-compose.yml file under the environment key.

  • FOSSOLOGY_DB_HOST: Hostname of the PostgreSQL database server. An integrated PostgreSQL instance is used if not defined or set to localhost.
  • FOSSOLOGY_DB_NAME: Name of the PostgreSQL database. Defaults to fossology.
  • FOSSOLOGY_DB_USER: User to be used for PostgreSQL connection. Defaults to fossy.
  • FOSSOLOGY_DB_PASSWORD: Password to be used for PostgreSQL connection. Defaults to fossy.

You can change them if you want to use a different database server or credentials.

Vagrant

FOSSology comes with a VagrantFile that can be used to create an isolated environment for FOSSology and its dependencies.

Pre-requisites: Vagrant >= 2.x and Virtualbox >= 5.2.x

Steps:

git clone https://github.com/fossology/fossology
cd fossology/
vagrant up

The server must be ready at http://localhost:8081/repo/. The login credentials are:

user: fossy
pass: fossy

Test Instance

For trying out FOSSology quickly, a test instance is also available at https://fossology.osuosl.org/. This instance can be deleted or reinstalled at any time, thus it is not suitable for serving as your productive version. The login credentials are as follows:

Username: fossy
Password: fossy

Note: The test instance is not up to date with the latest release. The instance will reset every night at 2 am UTC and all the user uploaded data will be lost.

Quick dev prototype with GitPod.io

FOSSology is ready to be coded on GitPod.io. To use it, you would need to setup an account. You can directly use the following button to launch the project on GitPod.io: Link to Gitpod

Once in, you should see 2 terminals, one running FOSSology scheduler and one running the installation.

Handy scripts/aliases

For the ease of usability, following aliases/scripts have been defined and can be used:

  • conffoss: This will reconfigure cmake with all variables
  • buildfoss: This will build the FOSSology using cmake
  • installfoss: This will install FOSSology
  • fossrun: Run the FOSSology scheduler
  • pg_stop: Stop PostgreSQL server
  • pg_start: Start PostgreSQL server

Documentation

We are currently migrating our documentation to Github. At this stage, you can find general documentation at: https://www.fossology.org/get-started/basic-workflow/ and developer docs on Github Wiki and https://fossology.github.io/

Support

Mailing lists, FAQs, Release Notes, and other useful info are available by clicking the documentation tab on the project website. We encourage all users to join the mailing list and participate in discussions. There is also a #fossology IRC channel on the freenode IRC network if you'd like to talk to other FOSSology users and developers. See Contact Us

Contributing

We really like contributions in several forms, see CONTRIBUTING.md

Licensing

The original FOSSology source code and associated documentation including these web pages are Copyright (C) 2007-2012 HP Development Company, L.P. In the past years, other contributors added source code and documentation to the project, see the NOTICES file or the referring files for more information.

Any modifications or additions to source code or documentation contributed to the FOSSology project are Copyright (C) the contributor, and should be noted as such in the comments section of the modified file(s).

FOSSology is licensed under GPL-2.0

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; version 2 of the License

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

Exception:

All of the FOSSology source code is licensed under the terms of the GNU General Public License version 2, with the following exceptions:

libfossdb and libfossrepo libraries are licensed under the terms of the GNU Lesser General Public License version 2.1, LGPL-2.1.

This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License.

This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
02110-1301  USA

Please see the LICENSE file included with this software for the full texts of these licenses.

fossdriver's People

Contributors

goneall avatar mcjaeger avatar sineadtl avatar smcv avatar swinslow avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fossdriver's Issues

Add support to detect Fossology server version

As part of moving towards supporting later versions of Fossology, it will likely be necessary to detect which version of Fossology is running.

Since fossdriver currently aims to maintain compatibility with Fossology 3.3.0 servers, the version check cannot count on using the REST API. Instead we will parse the version from the Version: [X.Y.Z] header on the Fossology server main page.

Code Formatting and Linting is required

Currently, fossdriver codebase is not following any specific Linters and formatters. Having a linter is useful for uniformity in coding style.

We can add black code formatter.

Errors when folder contains single-file uploads

Many of the tasks and server functions call GetUploadNum. This function scans through the uploads in the specified folder. It appears to assume that all items in the folder will be links to unpacked uploads, and therefore appears to fail if they aren't -- for instance, if a single non-archive file has been uploaded. For example:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/steve/programming/lftools/fossdriver/fossdriver-dev/fossdriver/fossdriver/tasks.py", line 372, in run
    uploadNum = self.server.GetUploadNum(folderNum, self.uploadName)
  File "/Users/steve/programming/lftools/fossdriver/fossdriver-dev/fossdriver/fossdriver/server.py", line 157, in GetUploadNum
    u = self._getUploadData(folderNum, uploadName, exact)
  File "/Users/steve/programming/lftools/fossdriver/fossdriver-dev/fossdriver/fossdriver/server.py", line 138, in _getUploadData
    parsedUploads = fossdriver.parser.parseAllUploadDataForFolder(uploadData)
  File "/Users/steve/programming/lftools/fossdriver/fossdriver-dev/fossdriver/fossdriver/parser.py", line 86, in parseAllUploadDataForFolder
    u = parseUploadDataForFolderLineItem(lineItem)
  File "/Users/steve/programming/lftools/fossdriver/fossdriver-dev/fossdriver/fossdriver/parser.py", line 63, in parseUploadDataForFolderLineItem
    href = aTag.get("href", None)
AttributeError: 'NoneType' object has no attribute 'get'

One of these functions (perhaps parseUploadDataForFolderLineItem) should probably check to see whether the upload is actually an unpacked archive rather than a single file, and then act accordingly.

Fossology REST api

Fossdriver was probably started when the REST API of Fossology was not available

Are there any plans regarding an integration of the REST endpoints into this library?

/cc @mcjaeger

Add documentation on bulk text matches

The Usage section in the README file currently describes most of the available functionality.
However, it doesn't describe how to use the monk bulk text match functionality (beyond just running a standard Monk scan). This should be described.

Can't identify whether server version 'unknown' is recent

The demo instance at https://fossology.osuosl.org/repo/?mod=browse identifies itself as:

 Version: [unknown], Branch: [master], Commit: [#b04657] 2022/01/07 08:35 UTC built @ 2022/01/07 08:42 UTC 

Similarly, the fossology/fossology:4.0.0-rc1 Docker image on Dockerhub identifies itself as:

 Version: [unknown], Branch: [unknown], Commit: [#unknown] unknown built @ 2021/12/22 06:06 UTC 

and other recent Docker images are similar.

As a result, the version checking in fossology.server cannot identify whether to use old or new APIs:

Traceback (most recent call last):
[local code redacted]
  File "/usr/local/lib/python3.9/dist-packages/fossdriver-0.0.3-py3.9.egg/fossdriver/tasks.py", line 73, in run
  File "/usr/local/lib/python3.9/dist-packages/fossdriver-0.0.3-py3.9.egg/fossdriver/server.py", line 492, in WaitUntilAgentIsDone
  File "/usr/local/lib/python3.9/dist-packages/fossdriver-0.0.3-py3.9.egg/fossdriver/server.py", line 297, in _getMostRecentAgentJobNum
  File "/usr/local/lib/python3.9/dist-packages/fossdriver-0.0.3-py3.9.egg/fossdriver/server.py", line 278, in _getJobsForUpload
  File "/usr/local/lib/python3.9/dist-packages/fossdriver-0.0.3-py3.9.egg/fossdriver/server.py", line 94, in IsAtLeastVersion
  File "/usr/local/lib/python3.9/dist-packages/version_parser-1.0.0-py3.9.egg/version_parser/version.py", line 23, in __init__
  File "/usr/local/lib/python3.9/dist-packages/version_parser-1.0.0-py3.9.egg/version_parser/version.py", line 139, in _parse
ValueError: Could not parse unknown

Retrieve counts of stats from upload summary

On the left side of the Fossology screen, there is a "Summary" window with various statistics:

image

It would be useful for fossdriver to be able to retrieve these stats for generating metrics and the like.

Warrior - issue & Dunfell takes longer time to generate SPDX

I am planning to use meta-spdxscanner layer to generate the SPDX docs in my poky warrior distro. I have run the fossology/fossology:3.5.0 docker image and have installed fossdriver. I am noticing this error with do_spdx()


ERROR: linux-libc-headers-5.0-r0 do_spdx: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:do_spdx(d)
     0003:
File: '/home/yocto/output/poky_warrior/poky/meta-spdxscanner/classes/fossdriver-host.bbclass', lineno: 137, function: do_spdx
     0133:    bb.note(' run fossdriver ...... ')
     0134:    if not os.path.isfile(tar_name):
     0135:        bb.warn(info['pn'] + "has no source, do nothing")
     0136:        return
 *** 0137:    invoke_fossdriver(tar_name,sstatefile)
     0138:    if get_cached_spdx(sstatefile) != None:
     0139:        write_cached_spdx( info,sstatefile,cur_ver_code )
     0140:        ## CREATE MANIFEST(write to outfile )
     0141:        create_manifest(info,sstatefile)
File: '/home/yocto/output/poky_warrior/poky/meta-spdxscanner/classes/fossdriver-host.bbclass', lineno: 237, function: invoke_fossdriver
     0233:    if (Reuse(server, tar_file, "Software Repository", tar_file, "Software Repository").run()  != True):
     0234:        bb.note("This OSS has not been scanned. So upload it to fossology server.")
     0235:        i = 0
     0236:        while i < 5:
 *** 0237:            if (Upload(server, tar_file, "Software Repository").run() != True):
     0238:                bb.warn("%s Upload failed, try again!" %  tar_file)
     0239:                time.sleep(delaytime)
     0240:                i += 1
     0241:            else:
File: '/home/fossdriver/fossdriver/tasks.py', lineno: 73, function: run
     0069:        logging.info("Upload complete, {} upload ID number is {}".format(self.filePath, newUploadNum))
     0070:
     0071:        # and wait until upload finishes unpacking
     0072:        logging.info("Waiting for upload {} to unpack".format(newUploadNum))
 *** 0073:        self.server.WaitUntilAgentIsDone(newUploadNum, "ununpack", pollSeconds=5)
     0074:        self.server.WaitUntilAgentIsDone(newUploadNum, "adj2nest", pollSeconds=5)
     0075:
     0076:        return True
     0077:
File: '/home/fossdriver/fossdriver/server.py', lineno: 457, function: WaitUntilAgentIsDone
     0453:            - agent: name of agent to check for.
     0454:            - pollSeconds: number of seconds to wait between polling. Defaults to 10.
     0455:        """
     0456:        # FIXME consider adding a max # of tries before returning
 *** 0457:        jobNum = self._getMostRecentAgentJobNum(uploadNum, agent)
     0458:        while not self._isJobDoneYet(jobNum):
     0459:            time.sleep(pollSeconds)
File: '/home/fossdriver/fossdriver/server.py', lineno: 297, function: _getMostRecentAgentJobNum
     0293:            - agent: name of agent to check for.
     0294:        Returns job ID number or -1 if not found.
     0295:        """
     0296:        # FIXME given _getJobsForUpload, currently retrieves just first page
 *** 0297:        jobs = self._getJobsForUpload(uploadNum)
     0298:        if jobs is None or jobs == []:
     0299:            return -1
     0300:        # will be returned in reverse chrono order, so we can just loop through
     0301:        # and stop on the first one we come to
File: '/home/fossdriver/fossdriver/server.py', lineno: 280, function: _getJobsForUpload
     0276:        # response format changed from XML to JSON on or around 3.5.0
     0277:        # see https://github.com/fossology/fossdriver/issues/17
     0278:        if self.IsAtLeastVersion("3.5.0"):
     0279:            # parse json
 *** 0280:            jobData = fossdriver.parser.parseJSONShowJobsData(results.content)
     0281:            return jobData
     0282:        else:
     0283:            # decode and parse XML
     0284:            decodedContent = fossdriver.parser.decodeAjaxShowJobsData(results.content)
File: '/home/fossdriver/fossdriver/parser.py', lineno: 209, function: parseJSONShowJobsData
     0205:    jobData = []
     0206:    js = json.loads(content.decode('utf-8'))
     0207:    showJobsData = js.get("showJobsData", [])
     0208:    for sjd in showJobsData:
 *** 0209:        jd = sjd.get("job", None)
     0210:        if jd is not None:
     0211:            jq = jd.get("jobQueue", None)
     0212:            if jq is not None:
     0213:                for k, v in jq.items():
Exception: AttributeError: 'str' object has no attribute 'get'

ERROR: linux-libc-headers-5.0-r0 do_spdx: 'str' object has no attribute 'get'
ERROR: linux-libc-headers-5.0-r0 do_spdx: Function failed: do_spdx
ERROR: Logfile of failure stored in: /home/yocto/output/poky_warrior/build/tmp/work/armv7vet2hf-neon-poky-linux-gnueabi/linux-libc-headers/5.0-r0/temp/log.do_spdx.27653
ERROR: Task (/home/yocto/output/poky_warrior/poky/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_5.0.bb:do_spdx) failed with exit code '1'

Can you please help me to solve this. ?

Initial release and PyPI packaging

I gather that fossdriver is seeing at least some usage out in the community! It would be great to be able to put together an initial release (even if that's still something like a 0.1.0 release).

I'd say a minimal piece of this will be to make it possible to install fossdriver via pip / PyPI. Rather than having to manually retrieve and unpack the code.

If others have input / experience or can help with moving to an initial point release, or with instrumenting fossdriver to work with PyPI, all help is welcome!

BulkTextMatch task does "add" when "remove" is requested in 3.6.0

When running a BulkTextMatch task, the user can call .add() to instruct the Fossology server to add a license, or .remove() to instruct it to remove a license.

When calling .remove() on a Fossology 3.6.0 server, it is incorrectly treating it as an .add() action instead, and adding the license.

This behavior was not observed on a Fossology 3.3.0 server when fossdriver was originally written. I haven't tested it (and likely won't) on 3.4.0 or 3.5.0, but would welcome reports from anyone who is using those servers.

Change format of stats returned

Currently, stats are returned as an array of tuples, with the first element a string with the stat name and the second as the integer count. This would be more useful as a dictionary. I've already implemented this previously, just submitting an issue here to track the commit/PR.

Add test scripts and CI

Currently, the fossdriver repo does not include any test scripts or CI integration. Prior to an official release, it should include:

  • scripts to enable spinning up a test Fossology server (presumably using the Fossology docker images)
  • CI integration (perhaps Travis CI?)
  • testing scripts to run through fossdriver functionality and ensure it works against the Fossology test server)

As a bonus, this could also be set up to test against multiple versions of Fossology, to help ensure that fossdriver stays current if/when the Fossology interface changes.

Should be modified to work with versions of Python prior to 3.6

Replacing f-strings with .format() should (hopefully) enable fossdriver to work with versions of Python older than 3.6.

Additionally, some users may still be using Python 2. It would be ideal if fossdriver were modified to the extent needed to work with Python 2.

Can't work with latest fossology

fossdriver can't with the fossology after 3.5.0. The following error will be output:
......
Upload(server, "xz-5.2.1.tar.gz", "Software Repository").run()
/fossdriver/fossdriver/tasks.py", line 73, in run
self.server.WaitUntilAgentIsDone(newUploadNum, "ununpack", pollSeconds=5)
/fossdriver/fossdriver/server.py", line
410, in WaitUntilAgentIsDone
jobNum = self._getMostRecentAgentJobNum(uploadNum, agent)
/fossdriver/fossdriver/server.py", line
250, in _getMostRecentAgentJobNum
jobs = self._getJobsForUpload(uploadNum)
/fossdriver/fossdriver/server.py", line
237, in _getJobsForUpload
decodedContent = fossdriver.parser.decodeAjaxShowJobsData(results.content)
/fossdriver/fossdriver/parser.py", line
147, in decodeAjaxShowJobsData
b1 = s1.encode("utf-8")
AttributeError: 'list' object has no attribute 'encode'

It is because the data from fossology changes from xml to json format. So, there are many functions to update.

Running bulk text match task fails if repo is empty

Appears to fail because no files means it crashes when it can't retrieve license data:

Traceback (most recent call last):
  File "upload-**********.py", line 256, in <module>
    runTasksForUpload(server, projectName, zipPath, uploadName)
  File "upload-**********.py", line 214, in runTasksForUpload
    retval = t.run()
  File "/Users/steve/programming/lftools/fossdriver/fossdriver/tasks.py", line 260, in run
    a = self._makeRealAction(licenseName, actionType)
  File "/Users/steve/programming/lftools/fossdriver/fossdriver/tasks.py", line 234, in _makeRealAction
    licenseId = self._findLicenseID(licenseName)
  File "/Users/steve/programming/lftools/fossdriver/fossdriver/tasks.py", line 224, in _findLicenseID
    self.parsedLicenses = self.server.GetLicenses(u._id, u.topTreeItemId)
  File "/Users/steve/programming/lftools/fossdriver/fossdriver/server.py", line 235, in GetLicenses
    licenses = fossdriver.parser.parseAllLicenseData(results.content)
  File "/Users/steve/programming/lftools/fossdriver/fossdriver/parser.py", line 127, in parseAllLicenseData
    options = sel.find_all("option")
AttributeError: 'NoneType' object has no attribute 'find_all'

(transferred from https://gitlab.com/swinslow/fossdriver/issues/1)

Exception: TypeError: the JSON object must be str, not 'bytes'

Hello,

I tried to install fossdriver (master branch: b950917).
My environment:

  • Ubuntu: 16.04
  • Python: 3.5

I get this error below

Exception: TypeError: the JSON object must be str, not 'bytes'

In parser.py (func: parseSingleJobData) and server.py (func: decodeAjaxShowJobsData),

I have added ".decode('utf-8')" to fix this error.
E.g.
from

rj = json.loads(content)

to

rj = json.loads(content.decode('utf-8'))

Can you check it?

Best regards,
Thuy

Add command-line interface

Currently, fossdriver provides functions to be called from a Python script. It should be fairly trivial (hopefully!) to add a basic command-line interface as well.

Make tasks more configurable

Tasks should be more configurable, either via keywords or via function calls after creation.

In particular, it would be helpful to be able to configure a task for (1) whether or not to wait for the task to finish, before returning; and (2) if waiting, how long to wait between each re-check.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.