The data catalog backend is based on NoSQL database MongoDB, the Node based web application framework Express and the API framework Loopback technology stack (the "MEL" part of the "MELANIE" technology stack). It provides a REST based API which allows to store information about datasets and to allow to answer queries about the stored meta data for these datasets

You need to setup a MongoDB server. E.g. on a Redhat Linux System the following command will suffice

yum install mongodb-org-server

The needed database will be created automatically when the API server starts. Follow this description to enable authenticated access to the Mongo DB.

git clone https://gitlab.psi.ch/MELANIE/catamel.git
cd catamel
npm install

Add a datasources.local.json file inside server directory with connection data to your mongodb instance

server/datasources.local.json e.g.
{
  "mongo": {
    "host": "mongodbprod.my.site",
    "user": "dacatDBAdmin",
    "password": "myverysecretPW"
  }
}

Inside your config.local.js, you can add the following blocks to enable email notification on submission of a Job. In the future, other notification types will be supported.

    smtpSettings: {
      host: 'mail.ethz.ch',
      port: 587,
      secure: false,
      auth: {user: 'psich\\USER', pass: 'PWD'}
    },
    smtpMessage: {
      from: '[email protected]',
      to: undefined,
      subject: '[SciCat]',
      text: undefined // can also set html key and this will override this
    }

The Job model checks for the existence of these blocks and sends an email from the User specified. You can override the object with the html key and send much more prettified content.

node .

Insert in the following curl commands the respective usernames like admin , archiveManager and ingestor and set their passwords. These are local accounts defined within the loopback application database

curl -X POST --header 'Content-Type: application/json' --header 'Accept:application/json' -d'{"realm":"my.site","username":"...","password":"...!","email":"[email protected]","emailVerified":true}' 'http://localhost:3000/api/v2/Users'

Jobs in Catamel are published to a queue when they are received. It is your responsibility to configure that queue and there is NO default in place.

Without setting this up, none of your job submissions will NOT go anywhere.

The two supported options are:

1. RabbitMQ (loopback-component-mq)
2. Apache Kafka (loopback-connector-kafka)

Both packages are installed at time of install and you can select a queue in config.local.js, like this:

queue: "rabbitmq" // also accepts "kafka" or null

1. Set config.local.js queue value to "rabbitmq".
2. Provide configuration for rabbitmq as a new block in component-config.json. NOTE: Make sure this is empty

1. Set config.local.js queue value to "kafka".
2. Instructions for configuring Kafka can be found here

• Creating a datasource programatically in the model you want to attach will often be the easiest solution.

NOTE: An example of kafka has been set in Job.js

The data model is defined inside the common/models directory according to the rules defined by the Loopback API framework The data model is visualized in form of an

Model UML diagram or
Model Visualizer

The REST API can be tested via the Explorer webpage The OpenAPI definition can be fetched from the Swagger Definition link

The API is automatically created from the Swagger/openAPI (http://swagger.io/) specificication created from the data model using the Swagger Codegen code

java -jar ./swagger-codegen/modules/swagger-codegen-cli/target/swagger-codegen-cli.jar generate -i swagger.json -l python -o dacat-api/client/python

See e.g. the description of Dataset model

The library is generated by the following command

./node_modules/.bin/lb-sdk server/server.js  client/angular2/sdk

Users are authenticated by loopback built in user accounts and by accounts, which are connected via any of the passport supported strategies, in particular OpenID connected and a direct AD connection