kit-data-manager / collection-api Goto Github PK

Collection API implementation following the recommendations of the RDA Research Data Collections WG

License: Apache License 2.0

Java 58.38% JavaScript 34.11% HTML 5.32% Dockerfile 0.38% Shell 1.23% CSS 0.59%

research-data-alliance research-data-repository rda-recommendation

collection-api's Introduction

Collection API

This project provides an implementation of the Collection API as it was proposed by the RDA Recommendation on Research Data Collections doi: 10.15497/RDA00022. The Collection API is implemented as Spring Boot-based Microservice and can be used for building collections of digital objects in a generic way independent from any repository managing the digital objects. The implementation is complete with regard to the recommendations supporting pagination, collection sorting, and listing of sub-collections.

Differences and Improvements

As some aspects of the Collection API are not clearly defined by the recommendation, this implementation contains some fixes [FIX], additions [ADD] and restrictions [RES] marked with the according tag. In all other cases the implementation is expected to behave as recommended.

[FIX] Return type inconsistencies have been fixed, e.g. in /collections/{id}/members/{mid}
[FIX] Delete operations return status 204 (NO_CONTENT) according to the HTTP specification
[FIX] Delete operations are realized idempotent following the HTTP specification. This means, that DELETE can be issued multiple times to a resource and returns HTTP 204 in all cases.
[FIX] Collection operations allow navigation the same way all other operations do, e.g. via prev and next links.
[RES] Listing a collection recursively does not consider the sorting of child elements.
[RES] A recursive listing of a collection will also contain member items of expanded collections. The recommendation document was not clear on this.
[RES] There is currently no build-in PID support. If no PID are provided with a collection or member, a UUID is assigned.
[ADD] Integrated ETag support in order to avoid concurrent modifications.
[ADD] Navigation through a result set is realized using default Spring pagination, e.g. supporting page and size query parameters. The cursors (next and prev) of a result set are pointing to the next/prev page link.

How to build

In order to build the Collection API you'll need:

Java SE Development Kit 11 or higher

After obtaining the sources change to the folder where the sources are located perform the following steps:

user@localhost:/home/user/collection-api$ ./gradlew -Dprofile=minimal build
Running gradle version: 7.4.2
Building collection-api version: 1.2.1-SNAPSHOT
JDK version: 11
Using minimal profile for building collection-api
<-------------> 0% EXECUTING [0s]
[...]
user@localhost:/home/user/collection-api$

The Gradle wrapper will now take care of downloading the configured version of Gradle, checking out all required libraries, build these libraries and finally build the collection-api microservice itself. As a result, a fat jar containing the entire service is created at 'build/jars/collection-api.jar'.

How to start

Prerequisites

PostgreSQL 9.1 or higher

Setup

Before you are able to start the microservice, you have to provide a configuration file according to your local setup. Therefor, copy the file 'config/application-default.properties' to your project folder, rename it to 'application.properties' and customize it. For the Collection API you just may adapt the properties of spring.datasource and change the server.port property. All other properties can be ignored for the time being.

As soon as you finished modifying 'application.properties', you may start the collection-api microservice by executing the following command inside the project folder, e.g. where the service has been built before:

user@localhost:/home/user/collection-api$ ./build/libs/collection-api.jar

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.2.2.RELEASE)
[...]
1970-01-01 00:00:00.000  INFO 56918 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8070 (http) with context path ''

As soon as the microservice is started, you can browse to

http://localhost:8070/swagger-ui.html

in order to see available RESTful endpoints and their documentation. You may have to adapt the port according to your local settings. Furthermore, you can use this Web interface to test single API calls in order to get familiar with the service.

You may create collections via any REST client or using the Web UI available at

http://localhost:8070/collections

The Collection API offers a graphical web frontend to visualize managed collections, collection items and relationships between them as well as associated metadata. In addition it allows a simple search for elements in order to visualize linked nodes. To access the web frontend, open the following link in your browser:

http://localhost:8070/static/overview.html

More Information

License

The KIT Data Manager is licensed under the Apache License, Version 2.0.

collection-api's People

Contributors

Stargazers

Watchers

Forkers

thomasjejkal tegx

collection-api's Issues

Add CodeQL to workflow actions.

Is your feature request related to a problem? Please describe.
Use semantic code analysis engine to find security vulnerabilities.

Describe the solution you'd like
Add this file to your workflow:
.github/workflows/codeql-analysis.yml

# For most projects, this workflow file will not need changing; you simply need
# to commit it to your repository.
#
# You may wish to alter this file to override the set of languages analyzed,
# or to provide custom queries or build logic.
#
# ******** NOTE ********
# We have attempted to detect the languages in your repository. Please check
# the `language` matrix defined below to confirm you have the correct set of
# supported CodeQL languages.
#
name: "CodeQL"

on:
  push:
    branches: [ main ]
  pull_request:
    # The branches below must be a subset of the branches above
    branches: [ main ]
  schedule:
    - cron: '47 3 * * 1'

jobs:
  analyze:
    name: Analyze
    runs-on: ubuntu-latest
    permissions:
      actions: read
      contents: read
      security-events: write

    strategy:
      fail-fast: false
      matrix:
        language: [ 'java', 'javascript' ]
        # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
        # Learn more about CodeQL language support at https://git.io/codeql-language-support

    steps:
    - name: Checkout repository
      uses: actions/checkout@v3

    # Initializes the CodeQL tools for scanning.
    - name: Initialize CodeQL
      uses: github/codeql-action/init@v2
      with:
        languages: ${{ matrix.language }}
        # If you wish to specify custom queries, you can do so here or in a config file.
        # By default, queries listed here will override any specified in a config file.
        # Prefix the list here with "+" to use these queries and those in the config file.
        # queries: ./path/to/local/query, your-org/your-repo/queries@main

    # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
    # If this step fails, then you should remove it and run the build manually (see below)
    - name: Autobuild
      uses: github/codeql-action/autobuild@v2

    # ℹ️ Command-line programs to run using the OS shell.
    # 📚 https://git.io/JvXDl

    # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
    #    and modify them (or add more) to build your code if your project
    #    uses a compiled language

    #- run: |
    #   make bootstrap
    #   make release

    - name: Perform CodeQL Analysis
      uses: github/codeql-action/analyze@v2

Enable auto update of github actions.

Is your feature request related to a problem? Please describe.
It is tedious to keep the githubActions versions up to date.
This can also be done by dependabot.

Describe the solution you'd like
Add the following lines to 'dependabot.yml':

  - package-ecosystem: "github-actions"
    directory: "/"
    target-branch: "development"
    schedule:
      interval: "weekly"

Too many information visible

Some fields returned by the controller are not expected to be visible, e.g. internal identifier of memberItem or list of all members in collection. They should be hidden and, if possible, obained in the proper way, e.g. by /collections/{id}/members

Error getting a collection or collection member by id

I have created a collection and added members. The members themselves are "FDOs" and have their own PID with prefix (e.g. my-pid/77eb7c2b-faf6-4d53-bc08-a754fb41f425). Now when I want to get a member I get a 404 or 400.

404: if I forget to urlencode the slash in the member ID (my-pid/77eb7c2b-faf6-4d53-bc08-a754fb41f425)
400: if I urlencode the slash in the member ID (my-pid%2F77eb7c2b-faf6-4d53-bc08-a754fb41f425)

edit: same applies if the collection's ID contains slashes!

{
    "capabilities": {
        "appendsToEnd": true,
        "id": 3,
        "isOrdered": false,
        "maxLength": -1,
        "membershipIsMutable": true,
        "propertiesAreMutable": false,
        "restrictedToType": "my-pidtype/f4d39f6018e1b8a9db0c",
        "supportsRoles": false
    },
    "description": "This collection holds all sensors used in the pv system",
    "id": "171836da-950c-4e7c-b40d-e75577f085b7",
    "members": [
        {
            "id": 10,
            "mappings": {
                "dateAdded": "2021-08-11T09:20:26.516Z",
                "dateUpdated": null,
                "index": 0,
                "role": null
            },
            "member": {
                "datatype": "my-pidtype/f4d39f6018e1b8a9db0c",
                "description": "The sensor of the second battery power inverted.",
                "id": "my-pid/aae9fc7b-66af-427c-b5e1-b9615cec47d3",
                "location": "https://example.org/sensor.json",
                "ontology": null
            }
        }
    ],
    "properties": {
        "dateCreated": "2021-08-10T10:36:48.801Z",
        "descriptionOntology": null,
        "hasAccessRestrictions": false,
        "id": 3,
        "license": "CC-BY-SA",
        "memberOf": [],
        "modelType": null,
        "ownership": "KIT IAI"
    }
}

Inconsistent handling of required attributes

According to the Swagger API documentation, e.g. the 'id' attribute is required by the model. This is acually not the case for the POST operation, as internal identifiers are assigned if not present. For technical reasons, POST operations work even without providing the 'id' attribute.
However, for PUT operations this requirement is enforced and uploading a document without id will lead to HTTP Bad Request, which is weird as the id is provided in the URL of the request and is used in the code from there.

By removing @NotNull constraints from Collection and CollectionMember entities, this issue should be solved.

Implementation of rule-based collections

Is your feature request related to a problem? Please describe.
Currently, collections and their members are created explicitly. In the RDA recommendation this implementation is based on, something like rule-based collections which are implicitly created, e.g., based on data type or other criteria, were mentioned and it would increase the benefit of a collection registry tremendously.

Describe the solution you'd like
There would be the need of a possibility to specify one or more rule(s) on a collection level. As a first draft idea this could look like:

A user creates collection A
The user defines a rule like "place all members of type to sub-collection "collection-of-t"
The user adds a member M of type to collection A, which:
- A first for existing rules matching the properties of member M
- A finds a rule based on M's type
- A applies the rule while adding M, which results in the creation of "collection-of-t" and adding M to this collection