Code Monkey home page Code Monkey logo

lemur's Introduction

GoDoc

HPDD HSM Agent and Data Movers for Lustre

Lemur is new, open source implementation of HSM tools for the Lustre Filesystem based on this design. This project is developed by the Intel High Performance Data Division, one of the main contributors to the Lustre Filesystem. We welcome community involvement and contributions. If you would like to raise an issue or file a bug, please use the LMR project on our community Jira server instead of Github issues.

Build

We use Docker for building packages, so as long as you have Docker running, make rpm will take care of everything needed to create a clean build environment and build a set of Lemur RPMs. The results will be stored in ./output/RPMS.

For development and testing, it is generally more convenient to use a Linux host (or virtual machine) with Lustre installed. We recommend RHEL or CentOS 7.x, Lustre 2.7 or above, and at least Go 1.6.

Testing Quickstart

The unit tests are run with make test and do not require a Lustre environment to be configured.

The user acceptance test (UAT) automates basic testing with the agent and datamovers. The harness does not manage Lustre filesystems -- you will need to create one and mount a client somewhere, and ensue the Coordinator is enabled on the MDT. The harness must run as root, because the agent must also run as root in order to fiddle with secure xattrs and do other root-y stuff.

  1. make rpm
  2. Copy built RPMs to Lustre client host and install them
  3. sudo /usr/libexec/lemur-testing/lemur-uat-runner

Set uat/README.md for more details on running and confgiuring the user acceptance tests.

lemur's People

Contributors

johnsonw avatar mhugues avatar mjmac avatar rread avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

lemur's Issues

make local-rpm fails on CentOS 7.6 with lfs 2.12.3 installed

Complete terminal output attached to the ticket.

[root@lustre-client-4 lemur]# lfs --version
lfs 2.12.3
[root@lustre-client-4 lemur]# uname -a
Linux lustre-client-4 3.10.0-1062.7.1.el7.x86_64 #1 SMP Mon Dec 2 17:33:29 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@lustre-client-4 lemur]# lsb_release -r
Release: 7.6.1810
[root@lustre-client-4 lemur]#

[root@lustre-client-4 lemur]# make local-rpm
make -C packaging/rpm NAME=lemur VERSION=0.6.0_4_g4655df8 RELEASE=1 URL="https://github.com/intel-hpdd/lemur"
make[1]: Entering directory `/root/lemur/packaging/rpm'
cd ../../ &&
.........
........
........
github.com/intel-hpdd/lemur/vendor/github.com/aws/aws-sdk-go/service/s3/s3manager
github.com/intel-hpdd/lemur/cmd/lhsm-plugin-s3
install -d $(dirname /root/rpmbuild/BUILDROOT/lemur-hsm-agent-0.6.0_4_g4655df8-1.x86_64//usr/bin/lhsm-plugin-s3)
install -m 755 lhsm-plugin-s3 /root/rpmbuild/BUILDROOT/lemur-hsm-agent-0.6.0_4_g4655df8-1.x86_64//usr/bin/lhsm-plugin-s3
go build -v -i -ldflags "-X 'main.version=0.6.0_4_g4655df8'" -o lhsm ./cmd/lhsm
github.com/intel-hpdd/lemur/vendor/github.com/intel-hpdd/go-lustre/pkg/pool
github.com/intel-hpdd/lemur/cmd/lhsmd/agent/fileid
github.com/intel-hpdd/lemur/vendor/github.com/intel-hpdd/go-lustre/llapi
github.com/intel-hpdd/lemur/vendor/gopkg.in/yaml.v2
github.com/intel-hpdd/lemur/vendor/gopkg.in/urfave/cli.v1

github.com/intel-hpdd/lemur/vendor/github.com/intel-hpdd/go-lustre/llapi

cgo-gcc-prolog: In function '_cgo_c110903d49cd_C2func_llapi_get_version':
cgo-gcc-prolog:58:2: warning: 'llapi_get_version' is deprecated (declared at /usr/include/lustre/lustreapi.h:398) [-Wdeprecated-declarations]
cgo-gcc-prolog: In function '_cgo_c110903d49cd_Cfunc_llapi_get_version':
cgo-gcc-prolog:107:2: warning: 'llapi_get_version' is deprecated (declared at /usr/include/lustre/lustreapi.h:398) [-Wdeprecated-declarations]

github.com/intel-hpdd/lemur/vendor/github.com/intel-hpdd/go-lustre/llapi

vendor/github.com/intel-hpdd/go-lustre/llapi/changelog.go:273:39: cannot use _Ctype_int(r.flags) (type _Ctype_int) as type int32 in argument to _Cfunc_hsm_get_cl_flags
make[2]: *** [lhsm] Error 2
make[2]: Leaving directory `/root/rpmbuild/BUILD/lemur-0.6.0_4_g4655df8/src/github.com/intel-hpdd/lemur'
error: Bad exit status from /var/tmp/rpm-tmp.cPPeEL (%install)

RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.cPPeEL (%install)
make[1]: *** [rpm] Error 1
make[1]: Leaving directory `/root/lemur/packaging/rpm'
make: *** [local-rpm] Error 2
[root@lustre-client-4 lemur]#

make_local-rpm_error.txt

make rpm fails in a docker container

Was told on Lustre discussion to open a ticket for it.

I am trying to install Lemur on CentOS 7.6 (7.6.1810) to integrate with Object storage but the install fails. I used the instructions on below page to install. I already had Lustre client (2.12.3) installed on the machine, so I started with steps for Lemur.

https://wiki.whamcloud.com/display/PUB/HPDD+HSM+Agent+and+Data+Movers+%28Lemur%29+Getting+Started+Guide

Steps followed:

git clone https://github.com/whamcloud/lemur.git
cd lemur
git checkout master
service docker start
make rpm

[root@lustre-client-4 lemur]# make rpm
make -C packaging/docker
make[1]: Entering directory /root/lemur/packaging/docker' make[2]: Entering directory /root/lemur/packaging/docker/go-el7'
Building go-el7/1.13.5-1.fc32 for 1.13.5-1.fc32
docker build -t go-el7:1.13.5-1.fc32 -t go-el7:latest --build-arg=go_version=1.13.5-1.fc32 --build-arg=go_macros_version=3.0.8-4.fc31 .
Sending build context to Docker daemon 4.608 kB
Step 1/9 : FROM centos:7
---> 5e35e350aded
Step 2/9 : MAINTAINER Robert Read [email protected]
---> Using cache
---> 4be0d7fa27a2
Step 3/9 : RUN yum install -y @development golang pcre-devel glibc-static which
---> Using cache
---> ac83254f37f7
Step 4/9 : RUN mkdir -p /go/src /go/bin && chmod -R 777 /go
---> Using cache
---> fdbb4d031716
Step 5/9 : ENV GOPATH /go PATH $GOPATH/bin:$PATH
---> Using cache
---> 216c5484727e
Step 6/9 : RUN go get github.com/tools/godep && cp /go/bin/godep /usr/local/bin
---> Running in aed86ac3eb87

/bin/sh: go: command not found
The command '/bin/sh -c go get github.com/tools/godep && cp /go/bin/godep /usr/local/bin' returned a non-zero code: 127
make[2]: *** [go-el7/1.13.5-1.fc32] Error 127
make[2]: Leaving directory /root/lemur/packaging/docker/go-el7' make[1]: *** [go-el7] Error 2 make[1]: Leaving directory /root/lemur/packaging/docker'
make: *** [docker] Error 2
[root@lustre-client-4 lemur]#

Is this repo for Lemur the most updated version?

[root@lustre-client-4 lemur]# lfs --version
lfs 2.12.3
[root@lustre-client-4 lemur]#

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.