This framework enables to prepare and perform a graph-based analysis on the graph topology of RDF datasets. One of the main goals were to do that on large-scale and with focus on performance, i.e., with large state-of-the-art RDF graphs (hundreds of millions of edges) and in parallel, with many datasets at once.

A recent analysis on 280 datasets from the LOD Cloud 2017 has been conducted with this framework. Please find here the results on 28 graph measures as a browsable version of the study. Also, the results are available as a citable resource at Zenodo.

RDF data dumps are preferred (so far). The framework is capable of dealing with the following:

• Automatic downloading of the RDF data dumps before preparation.
• Packed data dumps. Various formats are supported, like bz2, 7zip, tar.gz, etc. This is achieved by employing the unix-tool dtrx.
• Archives, which contain a hierarchy of files and folders, will get scanned for files containing RDF data. Files which are not associated with RDF data will be ignored, e.g. Excel-, HTML-, or text-files.
• The list of supported RDF media types is currently limited to the most common ones for RDF data, which are N-Triples, RDF/XML, Turtle, N-Quads, and Notation3. Any files containing these formats are transformed into N-Triples while graph creation. The transformation is achieved by employing the cli-tool rapper.

Further:

• The framework is implemented in Python. The list of supported graph measures is extendable.
• There is a ready-to-go docker-image available, with all third-party libraries pre-installed.

Currently ongoing and work in progress:

• Query instantiation from graph representation, and
• Edge- and vertex-based graph sampling.

Installation instructions can be found in INSTALL.

In each of the subpackages you will find a detailed README file. The following table gives you an overview of the most important subpackages.

Executable code can be found in each of the corresponding *.tasks.* subpackages, i.e.,

Please find more detailed instructions in the README files of the corresponding packages.

The software is suppossed to be run from command-line on a unix-based system.

$ python3 -m graph.tasks.prepare --from-db core education-data-gov-uk webisalod --threads 3

This command will (1) download (if not present), (2) transform (if necessary), and (3) prepare an RDF dataset as an edgelist, ready to be instantiated as graph-object.

• --from-db used to load dataset URLs and available formats from an sqlite-database configured in db.sqlite.properties.
• --threads indicates the number of datasets that are handled in parallel.

$ python3 -m graph.tasks.analysis.core_measures --from-file core education-data-gov-uk webisalod --threads 2 --threads-openmp 8 --features diameter --print-stats

This command instantiates the graph-objects, by loading the edgelists or the binary graph-objects, if available. After that, the graph measure diameter will be computed in the graphs.

• --from-file used here, so measure values will be printed to STDOUT.
• --threads indicates the number of datasets that are handled in parallel.

This package is licensed under the MIT License.

Please refer to the DOI for citation. You can cite all versions of this project by using the canonical DOI 10.5281/zenodo.2109469. This DOI represents all versions, and will always resolve to the latest one.