Code Monkey home page Code Monkey logo

xrootd-hdfs's Introduction

A HDFS filesystem plugin for XRootD

This plugin allows an XRootD host to talk directly to a HDFS filesystem via the OSS plugin layer.

To enable, add the following line to your Xrootd build:

ofs.osslib /usr/lib64/libXrdHdfs.so
ofs.ckslib * /usr/lib64/libXrdHdfs.so

Once enabled, it will use the default locations to bootstrap the HDFS configuration (currently /etc/hadoop/conf); if the output of hdfs dfs -ls shows your Hadoop instance, not the local POSIX filesystem, you should be set.

One complex part of launching an embedded JVM is determining the CLASSPATH variable. To do this, the plugin will internally perform the following:

source /etc/sysconfig/xrootd-hdfs && /usr/libexec/xrootd-hdfs/xrootd_hdfs_envcheck

and look in the output for updated values of LD_LIBRARY_PATH, CLASSPATH, and LIBHDFS_OPTS.

For the OSG packaging of HDFS, this should all work smoothly; you may need to re-implement the xrootd_hdfs_envcheck script if you want to port this plugin to a non-RHEL platform.

xrootd-hdfs's People

Contributors

bbockelm avatar brianhlin avatar cvuosalo avatar cwseys avatar ddavila0 avatar djw8605 avatar edquist avatar efajardo avatar jthiltges avatar matyasselmeci avatar osschar avatar zvada avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

xrootd-hdfs's Issues

gfal-rename fails with davs

gfal-rename fails with davs arguments. This bug may possibly occur with any kind of protocol. The error may be related to this message:

hdfsBuilderConnect(forceNewInstance=1, nn=default, port=0, kerbTicketCachePath=(NULL), userName=) error:
IllegalArgumentException: Null userjava.lang.IllegalArgumentException: Null user

Question: What is the status and future of this plugin?

Hi all,

Seeing as this plugin is no longer part of the OSG documentation, I assume it is being phased out.

Are there any plans regarding

  • future developments
  • archiving this repo
  • talks with the xrootd team to change ownership?

If there are no plans: Could you please add a LICENSE to make clear if future modification/forks are allowed?

Checksum failure

Hi,
Checksum fails when it is enabled on the server and client also marks -K ADLER32 flag. (https is know yet not supporte)

$ gfal-copy listing root://transfer-2.ultralight.org:1095//store/temp/user/jjj.112/listing201 -K ADLER32
Copying file:///afs/cern.ch/user/j/jbalcas/listing   [FAILED]  after 0s
gfal-copy error: 21 (Is a directory) - Error on XrdCl::CopyProcess::Run(): [ERROR] Server responded with an error: [3016] Unable to fchmod /store/temp/user/jjj.112/listing201; is a directory

From the server:

180714 03:07:06 20533 ofs_open: jbalcas.27779:[email protected] Unable to fchmod /store/temp/user/jjj.112/listing201; is a directory
Resulting PFN: /store/temp/user/jjj.112/listing201
180714 03:07:06 20533 jbalcas.27779:[email protected] ofs_close: use=0 fn=dummy

Let me know if more info is needed or should I increase verbose mode.

Thanks

Utilize `hdfsRead` when possible

hdfsRead has a few pipelining- and readahead-related improvements over hdfsPread. However, it can only be used when the remote client is streaming the file (no skips).

However, that should be easy to detect - use hdfsRead until a non-contiguous read is detected, then switch to hdfsPread.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.