hale studio is a proven ETL tool optimised for working with complex strucutred data, such as XML, relational databases, or a wide range of tabular formats. It supports all required procedures for semantic and structural transformation. It can also handle reprojection. While hale studio exists as a multi-platform interactive application, its capabilities can be provided through a webservice with an OpenAPI.

Hale Studio to be customized to be used for SWR purpose. Therefore, framework conditions such as transformation models must be determined and applied.

tbd

in federated infrastructures downloads take place at the remote locations, so there are typically no registration of that in the soilwise logs. A common approach is to redirect the user to the remote resource via a local proxy, so you can log the event anyway. As part of this mechanism the server can also validate if the resource is (still) available.

soilwise.eu/proxy?url=http://example.com/mydataset.zip

Develop/deploy export and download functions to and from SWR.

This includes backend and frontend work.

User and organization management, authorization and authentication to be set up.

Definition of done:

• Integrate GitHub as identity provider for Keycloak
• Limit to soilwise organization
• Look into to map roles from GitHub information

To enable a seamless development and deployment process, a server infrastructure to be set up at wetransform and made accessible to all SoilWise development teams.

Consider Containerization

Definition of done:

• Server infrastructure is made available to Soilwise
• Kubernetes cluster has been set up including:
• Keycloak
• hale-connect
• Argo workflows
• MinIO as S3 store
• Longhorn for ReadWriteMany volumes
• Integrated with keycloak for authentication

System Health & Status Monitoring to be set

Details can be found here

DoD:

• https://github.com/soilwise-he/Soilwise-Project-Backlog/issues/9

The SoilWise Repository is expected to fulfill the following functions:

Storage of artifacts
Storage of metadata
Storage of data
Storage of knowledge
Backup and versioning

Spec can be found here

These components make sure that data is interoperable, i.e. provided to agreed upon formats, structures and semantics. They are used to ingest data and transform it to common standard data, e.g. in the central SWR format for soil health.

Detailed descriptions can be found here