Code Monkey home page Code Monkey logo

metalnx-msi's Introduction

![alt text] [1] [1]: docs/IMAGES/emc_metalnx_logo.png

Metalnx MSI is a set of microservices designed to work alongside the iRODS (integrated Rule-Oriented Data System). It provides tools to enable automatic metadata extraction from a specific file types, list of all microservices installed in the grid as well as the MSI package version installed in the system.

Supported file types

  • Image files (JPG, JPEG)
  • SAM files (SAM, BAM, CRAM)
  • Illumina projects (entire Illumina project compressed file)
  • Variant call files (VCF)

Documentation

Metalnx has documentation to help with building and using the tool. Please, check the following links for further information.

Installing Metalnx MSI

The full documentation on how to install Metalnx MSI using .rpm and .deb packages is available in the INSTALL document.

How to build Metalnx MSI packages

Documentation on how to build Metalnx MSI is available in the BUILD file.

License

Copyright © 2015-2017 Dell EMC.

This software is provided under the Software license provided in the LICENSE file.

metalnx-msi's People

Contributors

ngmenyen avatar sgworth avatar trel avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

metalnx-msi's Issues

Impossible to install metalnx-msi

Hello,
I have tried for a few days to install metalnx-msi on my icat-server, but when i try with RPM i have many conflicts :

rpm -ivh --nodeps --conflicts metalnx-msi-plugins-1.3.0-3.rpm
Preparing...                          ################################# [100%]
        file /usr/lib/irods from install of metalnx-msi-plugins-1.3.0-1.x86_64 conflicts with file from package irods-runtime-4.2.1-1.x86_64
        file /usr/lib/irods/plugins from install of metalnx-msi-plugins-1.3.0-1.x86_64 conflicts with file from package irods-runtime-4.2.1-1.x86_64
        file /usr/lib/irods from install of metalnx-msi-plugins-1.3.0-1.x86_64 conflicts with file from package irods-server-4.2.1-1.x86_64
        file /usr/lib/irods/plugins from install of metalnx-msi-plugins-1.3.0-1.x86_64 conflicts with file from package irods-server-4.2.1-1.x86_64
        file /usr/lib/irods/plugins/microservices from install of metalnx-msi-plugins-1.3.0-1.x86_64 conflicts with file from package irods-server-4.2.1-1.x86_64
        file /usr/lib/irods from install of metalnx-msi-plugins-1.3.0-1.x86_64 conflicts with file from package irods-database-plugin-postgres-4.2.1-1.x86_64
        file /usr/lib/irods/plugins from install of metalnx-msi-plugins-1.3.0-1.x86_64 conflicts with file from package irods-database-plugin-postgres-4.2.1-1.x86_64
        file /usr/lib/irods from install of metalnx-msi-plugins-1.3.0-1.x86_64 conflicts with file from package irods-rule-engine-plugin-python-4.2.1-1.x86_64
        file /usr/lib/irods/plugins from install of metalnx-msi-plugins-1.3.0-1.x86_64 conflicts with file from package irods-rule-engine-plugin-python-4.2.1-1.x86_64
        file /usr/lib/irods from install of metalnx-msi-plugins-1.3.0-1.x86_64 conflicts with file from package irods-devel-4.2.1-1.x86_64
        file /usr/lib/irods from install of metalnx-msi-plugins-1.3.0-1.x86_64 conflicts with file from package irods-microservice-plugins-curl-2.1.0-1.x86_64
        file /usr/lib/irods/plugins from install of metalnx-msi-plugins-1.3.0-1.x86_64 conflicts with file from package irods-microservice-plugins-curl-2.1.0-1.x86_64
        file /usr/lib/irods/plugins/microservices from install of metalnx-msi-plugins-1.3.0-1.x86_64 conflicts with file from package irods-microservice-plugins-curl-2.1.0-1.x86_64

I have installed all dependencies on my server, but i have still this issue. Do you have any idea how i can fix it ?

Best Regards,
Joseph Kervellec

metalnx-msi does not work with iRODS 4.2

Following setup of metalnx-msi on a Centos 7 system with iRODS 4.2 (provider mode) the data extraction micro services do not fire. I tried moving them into the new micro service location at /usr/lib/irods/plugins/microservices, but got the same result.

is metalnx-msi still being maintained?

Is metalnx-msi still being maintained and recommended for use with iRODS 4.2.10? It looks like it hasn't been active for a while, and there are open issues from 2017 indicating installation problems. Have the microservices from here been replaced by something else?

Documentation issues

Originally issue associated with metalnx itself

irods-contrib/metalnx-web#26

snap of log..

I ran into a problem now while installing Metalnx-msi. The installation guide says, in order to verify the microservice installation, one should check "/var/lib/irods/plugins/microservices" to be sure that the .so files ( e.g libmsiget_illumina_meta.so..) are there. In my case, I found them in "/usr/lib/irods/plugins/microservices". Do you have idea how I can correct this? How do I make tomcat find the path? I copied them to the "/var/lib/irods/plugins/microservices" directory, but it didn't show on the metalnx-web.
The problem happen while installing on different systems with tomcat7 and tomcat8.

ps: it will be great if you include the download link on the installation guide page too.

RPM build problems

Hello,
I've checkout to the latest tag 1.3.0 to try to build a working RPM following the documentation.

I was not able to get a satisfying RPM because I had to make several (unwanted) modifications: (not blaming anyone here ;)

# git diff
diff --git a/build/README.md b/build/README.md
deleted file mode 100644
index 403b0d0..0000000
--- a/build/README.md
+++ /dev/null
@@ -1,7 +0,0 @@
-# Metalnx Microservices
-
-# Compilation
-
-cd build/
-cmake ..
-make
diff --git a/packaging/build.sh b/packaging/build.sh
index 80d5e97..704eb19 100755
--- a/packaging/build.sh
+++ b/packaging/build.sh
@@ -40,7 +40,7 @@ fi
 # =-=-=-=-=-=-=-=-=-=-
 # Getting build number
 SUFFIX_BUILD_NUMBER=$PLUGINVERSIONINT
-if [ -v BUILD_NUMBER ]; then
+if [ -n BUILD_NUMBER ]; then
     SUFFIX_BUILD_NUMBER=$BUILD_NUMBER
 fi

@@ -158,11 +158,11 @@ fi

 # =-=-=-=-=-=-=-
 # require irods-dev package
-if [ ! -f /usr/lib/libirods_client.a ] ; then
-    echo ""
-    echo "ERROR :: \"irods-dev\" package required to build this plugin" 1>&2
-    exit 1
-fi
+#if [ ! -f /usr/lib/libirods_client.a ] ; then
+#    echo ""
+#    echo "ERROR :: \"irods-dev\" package required to build this plugin" 1>&2
+#    exit 1
+#fi

 # =-=-=-=-=-=-=-
 # detect number of cpus
@@ -171,7 +171,7 @@ if [ "$DETECTEDOS" == "MacOSX" ] ; then
 elif [ "$DETECTEDOS" == "Solaris" ] ; then
     DETECTEDCPUCOUNT=`/usr/sbin/psrinfo -p`
 else
-    DETECTEDCPUCOUNT=`cat /proc/cpuinfo | grep processor | wc -l | tr -d ' '`
+    DETECTEDCPUCOUNT=`cat /proc/cpuinfo | grep -E "processor\s+\:" | wc -l | tr -d ' '`
 fi
 if [ $DETECTEDCPUCOUNT -lt 2 ] ; then
     DETECTEDCPUCOUNT=1
@@ -185,7 +185,7 @@ echo ""
 # =-=-=-=-=-=-=-
 # build the plugin itself
 echo "${text_green}${text_bold}Building...${text_reset}"
-$MAKEJCMD
+#$MAKEJCMD

 # =-=-=-=-=-=-=-
 # generate EPM list file from the template
@@ -221,6 +221,7 @@ fi
 # build package
 cd $TOPLEVEL
 EPMCMD=/usr/bin/epm
+set -x
 if [ "$DETECTEDOS" == "RedHatCompatible" ] ; then # CentOS and RHEL and Fedora
     echo "${text_green}${text_bold}Running EPM :: Generating $DETECTEDOS RPMs${text_reset}"
     EXTENSION="rpm"
diff --git a/packaging/metalnx_msi_plugins.list.template b/packaging/metalnx_msi_plugins.list.template
index a0513a8..ae8973f 100644
--- a/packaging/metalnx_msi_plugins.list.template
+++ b/packaging/metalnx_msi_plugins.list.template
@@ -17,7 +17,7 @@
 %endif

 %if RPM
-%requires irods-dev
+%requires irods-devel
 %requires libexif-devel
 %requires libxml2-devel
 %endif
@@ -110,11 +110,11 @@ END_POSTREMOVE
 # Full File Listing
 # =-=-=-=-=-=-=-
 f 644 root root /usr/include/metalnx.h ./libmetalnx/metalnx.h
-f 755 $OS_IRODS_ACCT $OS_IRODS_ACCT ${IRODS_HOME_DIR}/plugins/microservices/libmsiget_illumina_meta.so ./msiget_illumina_meta/libmsiget_illumina_meta.so
-f 755 $OS_IRODS_ACCT $OS_IRODS_ACCT ${IRODS_HOME_DIR}/plugins/microservices/libmsiobjput_populate.so ./msiobjput_populate/libmsiobjput_populate.so
-f 755 $OS_IRODS_ACCT $OS_IRODS_ACCT ${IRODS_HOME_DIR}/plugins/microservices/libmsiobjjpeg_extract.so ./msiobjjpeg_extract/libmsiobjjpeg_extract.so
-f 755 $OS_IRODS_ACCT $OS_IRODS_ACCT ${IRODS_HOME_DIR}/plugins/microservices/libmsiobjput_mdbam.so ./msiobjput_mdbam/libmsiobjput_mdbam.so
-f 755 $OS_IRODS_ACCT $OS_IRODS_ACCT ${IRODS_HOME_DIR}/plugins/microservices/libmsiobjput_mdvcf.so ./msiobjput_mdvcf/libmsiobjput_mdvcf.so
-f 755 $OS_IRODS_ACCT $OS_IRODS_ACCT ${IRODS_HOME_DIR}/plugins/microservices/libmsiobjput_mdmanifest.so ./msiobjput_mdmanifest/libmsiobjput_mdmanifest.so
-f 755 $OS_IRODS_ACCT $OS_IRODS_ACCT ${IRODS_HOME_DIR}/plugins/microservices/libmsiobjget_version.so ./msiobjget_version/libmsiobjget_version.so
-f 755 $OS_IRODS_ACCT $OS_IRODS_ACCT ${IRODS_HOME_DIR}/plugins/microservices/libmsiobjget_microservices.so ./msiobjget_microservices/libmsiobjget_microservices.so
+f 755 $OS_IRODS_ACCT $OS_IRODS_ACCT ${IRODS_HOME_DIR}/plugins/microservices/libmsiget_illumina_meta.so ./libmsiget_illumina_meta.so
+f 755 $OS_IRODS_ACCT $OS_IRODS_ACCT ${IRODS_HOME_DIR}/plugins/microservices/libmsiobjput_populate.so ./libmsiobjput_populate.so
+f 755 $OS_IRODS_ACCT $OS_IRODS_ACCT ${IRODS_HOME_DIR}/plugins/microservices/libmsiobjjpeg_extract.so ./libmsiobjjpeg_extract.so
+f 755 $OS_IRODS_ACCT $OS_IRODS_ACCT ${IRODS_HOME_DIR}/plugins/microservices/libmsiobjput_mdbam.so ./libmsiobjput_mdbam.so
+f 755 $OS_IRODS_ACCT $OS_IRODS_ACCT ${IRODS_HOME_DIR}/plugins/microservices/libmsiobjput_mdvcf.so ./libmsiobjput_mdvcf.so
+f 755 $OS_IRODS_ACCT $OS_IRODS_ACCT ${IRODS_HOME_DIR}/plugins/microservices/libmsiobjput_mdmanifest.so ./libmsiobjput_mdmanifest.so
+f 755 $OS_IRODS_ACCT $OS_IRODS_ACCT ${IRODS_HOME_DIR}/plugins/microservices/libmsiobjget_version.so ./libmsiobjget_version.so
+#f 755 $OS_IRODS_ACCT $OS_IRODS_ACCT ${IRODS_HOME_DIR}/plugins/microservices/libmsiobjget_microservices.so ./libmsiobjget_microservices.so
  • The README.md deletion is because of a build.sh clean launch
  • I replace the -v test by a -n because my shell complained about it. I haven't been able to find what such test could be used for.
  • The /usr/lib/libirods_client.a stuff is because I don't have such a file, but I DO have irods-devel package installed.
  • The CPU count was bad on my virtualised environment, the pattern processor was present twice: on the expected processor :, but also on the CPU's name (ie. common KVM processor), the replacement fixes the count error.
  • You can ignore the $MAKEJCMD comment, it was only to test the script further, as same as the set -x
  • On CentOS6, the RPM package is not irods-dev, but irods-devel
  • The list template changes are here only because I think I missed smthg during the build process, all .so files have been generated in the current directory instead of a dedicated one.

Few other information... I run onto:

# cat /etc/redhat-release
CentOS release 6.9 (Final)

The irods development package on CentOS 6 is named irods-devel.

I also had to install several packages:

irods-externals-clang3.8-0
openssl-devel
irods-externals*
boost
boost-devel
rpm-build
gcc-c++
libexif-devel
libxml2-devel
cmake3

I had to clone https://github.com/Tencent/rapidjson.git git repo and copy include/rapidjson dir into /usr/include/

I got some error at compilation:

# IRODSVERSION=4.2.2 ./packaging/build.sh > log 2>&1

Log below:

�[36m�[1m
+------------------------------------+
| iRODS Plugin Build Script          |
+------------------------------------+
Tue Apr 17 20:03:42 CEST 2018
�[m�

�[32m�[1mBuild Environment...�[m�
Detected OS                         [RedHatCompatible]
Detected OS Version                 [6.9]
Detected Plugin Name                [metalnx_msi_plugins]
Detected EPM Package Name           [metalnx-msi-plugins-4.2.2]
Detected Plugin Version to Build    [1.3.0]
Detected Plugin Version Integer     [DEV]
Detected Project Directory          [/root/metalnx-msi]
Detected Packaging Directory        [/root/metalnx-msi/packaging]
Detected Target Build Directory     [/root/metalnx-msi/build]
Detected EPM List File              [/root/metalnx-msi/packaging/metalnx_msi_plugins.list]
Detected CPUs                       [4]
Compile Command                     [make -j 7]

�[32m�[1mBuilding...�[m�
[  5%] Linking CXX shared module libmsiobjget_microservices.so
[ 16%] Built target msiobjjpeg_extract
[ 27%] Built target msiget_illumina_meta
[ 33%] Linking CXX shared module libmsirule_deployment.so
[ 44%] Built target msiobjput_mdvcf
[ 55%] Built target msiobjput_mdmanifest
[ 66%] Built target msiobjput_populate
[ 77%] Built target msiobjput_mdbam
[ 88%] Built target msiobjget_version
CMakeFiles/msiobjget_microservices.dir/microservices/msiobjget_microservices/libmsiobjget_microservices.cpp.o: In function `msiobjget_microservices':
/root/metalnx-msi/microservices/msiobjget_microservices/libmsiobjget_microservices.cpp:(.text+0x560): undefined reference to `boost::filesystem::path::filename() const'
CMakeFiles/msiobjget_microservices.dir/microservices/msiobjget_microservices/libmsiobjget_microservices.cpp.o: In function `boost::filesystem::exists(boost::filesystem::path const&)':
/root/metalnx-msi/microservices/msiobjget_microservices/libmsiobjget_microservices.cpp:(.text._ZN5boost10filesystem6existsERKNS0_4pathE[_ZN5boost10filesystem6existsERKNS0_4pathE]+0x1c): undefined reference to `boost::filesystem::detail::status(boost::filesystem::path const&, boost::system::error_code*)'
CMakeFiles/msiobjget_microservices.dir/microservices/msiobjget_microservices/libmsiobjget_microservices.cpp.o: In function `boost::filesystem::directory_iterator::directory_iterator(boost::filesystem::path const&)':
/root/metalnx-msi/microservices/msiobjget_microservices/libmsiobjget_microservices.cpp:(.text._ZN5boost10filesystem18directory_iteratorC2ERKNS0_4pathE[_ZN5boost10filesystem18directory_iteratorC2ERKNS0_4pathE]+0x63): undefined reference to `boost::filesystem::detail::directory_iterator_construct(boost::filesystem::directory_iterator&, boost::filesystem::path const&, boost::system::error_code*)'
CMakeFiles/msiobjget_microservices.dir/microservices/msiobjget_microservices/libmsiobjget_microservices.cpp.o: In function `boost::filesystem::directory_entry::status() const':
/root/metalnx-msi/microservices/msiobjget_microservices/libmsiobjget_microservices.cpp:(.text._ZNK5boost10filesystem15directory_entry6statusEv[_ZNK5boost10filesystem15directory_entry6statusEv]+0x1c): undefined reference to `boost::filesystem::directory_entry::m_get_status(boost::system::error_code*) const'
CMakeFiles/msiobjget_microservices.dir/microservices/msiobjget_microservices/libmsiobjget_microservices.cpp.o: In function `boost::filesystem::directory_iterator::increment()':
/root/metalnx-msi/microservices/msiobjget_microservices/libmsiobjget_microservices.cpp:(.text._ZN5boost10filesystem18directory_iterator9incrementEv[_ZN5boost10filesystem18directory_iterator9incrementEv]+0x15): undefined reference to `boost::filesystem::detail::directory_iterator_increment(boost::filesystem::directory_iterator&, boost::system::error_code*)'
clang-3.8: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [libmsiobjget_microservices.so] Error 1
make[1]: *** [CMakeFiles/msiobjget_microservices.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
CMakeFiles/msirule_deployment.dir/microservices/msirule_deployment/libmsirule_deployment.cpp.o: In function `boost::filesystem::is_regular_file(boost::filesystem::path const&)':
/root/metalnx-msi/microservices/msirule_deployment/libmsirule_deployment.cpp:(.text._ZN5boost10filesystem15is_regular_fileERKNS0_4pathE[_ZN5boost10filesystem15is_regular_fileERKNS0_4pathE]+0x1c): undefined reference to `boost::filesystem::detail::status(boost::filesystem::path const&, boost::system::error_code*)'
clang-3.8: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [libmsirule_deployment.so] Error 1
make[1]: *** [CMakeFiles/msirule_deployment.dir/all] Error 2
make: *** [all] Error 2

I don't know how to fix that, I'm not at ease with C++ world.

I may have miss some details, but I've tried to log all so that the compilation be simpler in the future, or for other folks.

Anyway, thanks for the work already done!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.