The data catalog backend is based on NoSQL database MongoDB, the Node based web application framework Express and the API framework Loopback technology stack (the "MEL" part of the "MELANIE" technology stack). It provides a REST based API which allows to store information about datasets and to allow to answer queries about the stored meta data for these datasets
You need to setup a MongoDB server. E.g. on a Redhat Linux System the following command will suffice
yum install mongodb-org-server
The needed database will be created automatically when the API server starts. Follow this description to enable authenticated access to the Mongo DB.
git clone https://gitlab.psi.ch/MELANIE/catamel.git
cd catamel
npm install
Add a datasources.local.json file inside server directory with connection data to your mongodb instance
server/datasources.local.json e.g.
{
"mongo": {
"host": "mongodbprod.my.site",
"user": "dacatDBAdmin",
"password": "myverysecretPW"
}
}
Inside your config.local.js
, you can add the following blocks to enable email notification on submission of a Job. In the future, other notification types will be supported.
smtpSettings: {
host: 'mail.ethz.ch',
port: 587,
secure: false,
auth: {user: 'psich\\USER', pass: 'PWD'}
},
smtpMessage: {
from: '[email protected]',
to: undefined,
subject: '[SciCat]',
text: undefined // can also set html key and this will override this
}
The Job model checks for the existence of these blocks and sends an email from the User specified. You can override the object with the html
key and send much more prettified content.
node .
Insert in the following curl commands the respective usernames like admin , archiveManager and ingestor and set their passwords. These are local accounts defined within the loopback application database
curl -X POST --header 'Content-Type: application/json' --header 'Accept:application/json' -d'{"realm":"my.site","username":"...","password":"...!","email":"[email protected]","emailVerified":true}' 'http://localhost:3000/api/v2/Users'
Jobs in Catamel are published to a queue when they are received. It is your responsibility to configure that queue and there is NO default in place.
Without setting this up, none of your job submissions will NOT go anywhere.
The two supported options are:
- RabbitMQ (loopback-component-mq)
- Apache Kafka (loopback-connector-kafka)
Both packages are installed at time of install and you can select a queue in config.local.js
, like this:
queue: "rabbitmq" // also accepts "kafka" or null
- Set
config.local.js
queue value to "rabbitmq". - Provide configuration for rabbitmq as a new block in
component-config.json
. NOTE: Make sure this is empty
- Set
config.local.js
queue value to "kafka". - Instructions for configuring Kafka can be found here
- Creating a datasource programatically in the model you want to attach will often be the easiest solution.
NOTE: An example of kafka has been set in Job.js
The data model is defined inside the common/models directory according to the rules defined by the Loopback API framework The data model is visualized in form of an
Model UML diagram or
Model Visualizer
The REST API can be tested via the Explorer webpage The OpenAPI definition can be fetched from the Swagger Definition link
The API is automatically created from the Swagger/openAPI (http://swagger.io/) specificication created from the data model using the Swagger Codegen code
java -jar ./swagger-codegen/modules/swagger-codegen-cli/target/swagger-codegen-cli.jar generate -i swagger.json -l python -o dacat-api/client/python
See e.g. the description of Dataset model
The library is generated by the following command
./node_modules/.bin/lb-sdk server/server.js client/angular2/sdk
Users are authenticated by loopback built in user accounts and by accounts, which are connected via any of the passport supported strategies, in particular OpenID connected and a direct AD connection