hyperlearningai / ontopop Goto Github PK

OntoPop is a collection of event-driven data pipelines and APIs that enable the visualisation, search, exploration and management of version-controlled ontologies. The OntoPop backend, which includes its event-driven data pipelines and APIs, is an open-source software project (the source code of which is available in this repository). The OntoPop frontend app is a proprietary closed-source software project.

Website · App · Docs

The OntoPop app

1. Introduction
    1.1. Vision
    1.2. Sponsors
    1.3. Open Frameworks
2. Design
    2.1. Microservices
    2.2. Design Principles
    2.3. Logical System Architecture
3. Getting Started
    3.1. Build from Source
    3.2. Deployment
4. License
5. Acknowledgements
6. Contact and Further Information

OntoPop is a collection of event-driven data pipelines and APIs that enable the visualisation, search, exploration and management of version-controlled ontologies. The OntoPop backend, which includes its event-driven data pipelines and APIs, is an open-source software project (the source code of which is available in this repository). The OntoPop frontend app is a proprietary closed-source software project.

The goal of OntoPop is to democratize ontologies by making them easier to understand and more accessible to a broader range of users beyond information and data architects. By democratizing ontologies, OntoPop enables organisations and individuals to easily visualise, explore and semantically query their knowledge assets, data and relationships.

The OntoPop backend, which includes its event-driven data pipelines and APIs, is an open-source software project that was sponsored by National Highways between 2020 to 2022, and is currently sponsored by HyperLearning AI. The OntoPop frontend app is a closed-source software project that is sponsored by HyperLearning AI.

The OntoPop backend is primarily written in Java 11 and utilizes the following core open-source software frameworks and services:

• OpenJDK 11
• Spring Framework, including the following Spring projects:
  • Spring Boot
  • Spring Data
  • Spring Cloud
    • Spring Cloud Function
    • Spring Cloud Stream
    • Spring Cloud Vault
    • Spring Cloud AWS
    • Spring Cloud Azure
    • Spring Cloud GCP
  • Spring Security
  • Spring Vault
• Apache Maven
• Apache Commons
• Apache Jena
• Apache TinkerPop (including Gremlin)
• JanusGraph
• Elasticsearch
• RabbitMQ
• HashiCorp Vault
• Java OWL API
• Google Guava
• Git
• Node.js

The OntoPop backend is a collection of event-driven services that enable the visualisation, search, exploration and management of version-controlled ontologies. The following diagram illustrates the event-driven microservices available in OntoPop.

OntoPop's event-driven microservices

Below is a brief description of the functional purpose of each of these microservices, with links to their respective Maven module.

The ontology ingestion service ingests a W3C Web Ontology Language (OWL) ontology from a Git-based version control repository and copies it to persistent object storage.

The ontology validation service ingests a W3C Web Ontology Language (OWL) ontology and validates it using semantic reasoners, returning true or false.

The ontology triplestore loading service ingests a W3C Web Ontology Language (OWL) ontology and loads it into a physical RDF triplestore.

The ontology triplestore query service is used to query the RDF triplestore using SPARQL queries and returns the relevant triples.

The ontology parsing service ingests a W3C Web Ontology Language (OWL) ontology and parses it into its constituent objects, including annotation properties, object properties, classes and class relationships.

The property graph modelling service ingests the parsed objects from a W3C Web Ontology Language (OWL) ontology and models them as directed property graph objects, specifically vertices, edges, vertex properties and edge properties.

The property graph loading service ingests the directed property graph objects and loads them into a physical graph database.

The property graph indexing service ingests the directed property graph objects and indexes them into a physical search index.

The property graph query service is used to query the graph database and search index using Gremlin graph and free-text search queries, and returns the relevant result sets.

Every OntoPop service is developed using open standards and open frameworks and, as such, is agnostic of the target deployment environment meaning that OntoPop supports deployment to on-premise, private cloud, public cloud (including Amazon Web Services, Microsoft Azure and Google Cloud Platform), multi-cloud and/or hybrid environments, and supports integration with industry-standard middleware and other open software services.

Given that OntoPop is a collection of event-driven services, each service may be reused beyond the specific purposes of ontology visualisation, search, exploration and management. For example, the OntoPop property graph services may be reused to load, manage and query general graph databases for a wide range of alternative use cases including fraud detection, digital twins, disease modelling, behavioural analysis, recommendation systems and natural language processing.

The following diagram describes the high-level logical system architecture of OntoPop when deployed as a single logical service.

OntoPop logical system architecture

For further information regarding the technical, integration and security architecture of the OntoPop project, please visit the OntoPop documentation at https://docs.ontopop.com.

The following instructions describe how to clone the OntoPop source code repository into your development or deployment environment, and then how to compile, build and package the respective OntoPop software services and applications ready for deployment.

Please ensure that the following prerequisite build tools are installed in your development or deployment environment.

• OpenJDK 11 - open source reference implementation of Java 11.
• Apache Maven - open source build automation tool for Java.
• Git - open source distributed version control system.
• Node.js - open source backend JavaScript runtime environment.
• npm - Package manager for the Node.js JavaScript runtime environment.

The open-source source code for the OntoPop project may be found on GitHub at https://github.com/hyperlearningai/ontopop. To clone the OntoPop source code repository into your development or deployment environment, please run the following Git command via your command line (or via your preferred Git GUI tool). The location of the cloned OntoPop source code project folder will hereafter be referred to as $ONTOPOP_BASE.

# Clone the OntoPop GitHub public repository
$ git clone https://github.com/hyperlearningai/ontopop.git

# Navigate into the OntoPop project folder
# This location will hereafter be referred to as $ONTOPOP_BASE
$ cd ontopop

The following table describes the Maven profiles defined in $ONTOPOP_BASE/pom.xml.

The following table describes the Maven profiles defined in $ONTOPOP_BASE/ontopop-apps/pom.xml.

Please ensure that you have entered the correct configuration into the Spring bootstrap and application property files respectively, appropriate to your target deployment environment, prior to packaging OntoPop. For further information regarding configuring OntoPop, please refer to the OntoPop Spring bootstrap context and application context documentation pages respectively. To compile and build OntoPop backend services and applications from source, please run the following commands via your command line.

# Navigate to $ONTOPOP_BASE
$ cd $ONTOPOP_BASE

# Clean the project working directory.
$ mvn clean

# Compile, build and package OntoPop.
# By default this will build all services and applications.
$ mvn package

# If you wish to build OntoPop's core services only, 
# then disable the "apps" Maven profile as follows.
$ mvn package -P \!apps

# If you wish to build OntoPop's core services but only the
# apps intended for deployment to a self-managed environment
# or to Spring Cloud, then enable the "apps-spring" Maven 
# profile as follows.
$ mvn package -P apps-spring

# If you wish to build OntoPop's core services but only the
# apps intended for deployment to AWS, then enable the 
# "apps-aws" Maven profile as follows.
$ mvn package -P apps-aws

# If you wish to build OntoPop's core services but only the
# apps intended for deployment to Azure, then enable the 
# "apps-azure" Maven profile as follows.
$ mvn package -P apps-azure

If you are running mvn package for the first time, it will take approximately 5 - 10 minutes to complete the build (dependent on the speed of your internet connection) as Maven will download all the required Java dependencies for the first time. Subsequent executions of mvn package should take between 2 - 3 minutes to complete.

Assuming that mvn package completes successfully, you are now ready to deploy OntoPop. Please follow the links below for deployment instructions specific to your target deployment environment.

• Self Managed - deploy OntoPop to a self-managed on-premise, public/private cloud or hybrid environment, integrated with entirely open-source self-managed software services such as HashiCorp Vault, RabbitMQ, MySQL, Apache Jena Fuseki, JanusGraph and Elasticsearch.
  
  
• Amazon Web Services - deploy OntoPop to the Amazon Web Services (AWS) cloud computing platform, integrated with AWS managed services including AWS Secrets Manager, Amazon S3, Amazon MQ, Amazon RDS, AWS Lambda and AWS Elastic Beanstalk.
  
  
• Microsoft Azure - deploy OntoPop to the Microsoft Azure cloud computing platform, integrated with Azure managed services including Azure Key Vault, Azure Blob Storage, Azure Service Bus, Azure Functions and Azure Web Apps.

The OntoPop backend, which includes its event-driven data pipelines and APIs, is an open-source software project available under the GNU General Public License v3.0 (GNU GPLv3) software license (the source code of which is available in this repository). The OntoPop frontend app is a proprietary closed-source software project.

Provided below is the list of organisations and individuals who sponsor, and/or contribute towards, the development of the OntoPop project.

• HyperLearning AI - Project sponsor, management and maintenance
• National Highways - Proof of Value (PoV) stage project sponsor

• Jillur Quddus - Project Lead, Lead Technical Architect & Lead Software Engineer

For further information, please visit the OntoPop documentation website at https://docs.ontopop.com or contact HyperLearning AI using the details below.

• Jillur Quddus
  Chief Data Scientist and Principal Polyglot Software Engineer
  [email protected]