Code Monkey home page Code Monkey logo

libpyhdfs's Introduction

senseb

senseb的后花园

libpyhdfs's People

Watchers

 avatar

libpyhdfs's Issues

listdir of non-existent paths segfaults (seems to be a hadoop/jvm bug)

The following will crash the jvm, as long the as the path exists:

    fs = pyhdfs.connect(host, port)
    l = pyhdfs.listdir(fs, '/doesnotexist')

This really does not seem to be a bug in libpyhdfs, but I figured I'll file 
this ticket so anyone else looking for this problem sees this.

I have no idea how hadoop can trigger a jvm segfault.

This is the output:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007ff0fadb0ee8, pid=29972, tid=140673005336304
#
# JRE version: 6.0_16-b01
# Java VM: Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed mode linux-
amd64 )
# Problematic frame:
# V  [libjvm.so+0x3d4ee8]
#
# An error report file with more information is saved as:
# /home/tv/src/libpyhdfs.git/test/hs_err_pid29972.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

A sample logfile is attached.

Original issue reported on code.google.com by [email protected] on 3 Nov 2009 at 11:35

Attachments:

Please don't use data_files to ship jars and prebuilt shared lib

Including jars, hadoop config and a prebuilt shared lib is extremely 
unhelpful:

1. it actually prevents using this package with an actual Hadoop setup, 
e.g. looking up the real config file

2. putting the files at a hardcoded /usr/local location prevents installing 
the project with a --prefix= (distutils aborts with a sandbox error).

3. using the same .so isn't going to work across architectures

4. using a .so or jars you just download over the net is a security risk

Attaching a suggested patch dont-bundle.diff that just recommends using the 
Cloudera debs.

Setting CLASSPATH inside the C source is also a bad idea, as it embeds the 
version numbers of the jars in it, and they just won't ever match what the 
person running the thing will have. Attaching patch no-classpath.diff that 
removes that. Sadly, as running Java just never really got nicely 
integrated with the operating system, and Hadoop provides no easy mechanism 
for finding the right paths, the caller will have to set the CLASSPATH. But 
at least that can happen outside of C, in Python or shell, and it can be 
done so that it matches the runtime environment.

Original issue reported on code.google.com by [email protected] on 23 Oct 2009 at 2:47

Attachments:

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.