Code Monkey home page Code Monkey logo

data-pipeline's People

Watchers

 avatar  avatar

data-pipeline's Issues

hdfs safe off

hdfs가 비정상적으로 종료되거나, 네임노드가 start 되는 과정에서 safe-mode가 발생한다.

그러다 보니, hbase의 데이터를 hdfs에 써야 하는 상황에서, safe mode로 인해 data를 write하지 못함.

safe-mode에서는 hdfs에 파일을 write 하지 못한다.

safe mode를 off하기 위해 아래와 같이 명령어를 사용한다.

hdfs dfsadmin -safemode leave 또는 hdfs dfsadmin -safemode forceExit

happyhbase 이슈

hbase 구축후 데이터를 python으로 넣기 위해 happyhbase 라이브러리를 사용하고자 했음.

허나 아래와 같은 이슈가 발생함

Traceback (most recent call last):
  File "./insert_hbase.py", line 5, in <module>
    print(connection.tables())
  File "/usr/local/lib/python3.8/site-packages/happybase/connection.py", line 242, in tables
    names = self.client.getTableNames()
  File "/usr/local/lib/python3.8/site-packages/thriftpy2/thrift.py", line 219, in _req
    return self._recv(_api)
  File "/usr/local/lib/python3.8/site-packages/thriftpy2/thrift.py", line 231, in _recv
    fname, mtype, rseqid = self._iprot.read_message_begin()
  File "thriftpy2/protocol/cybin/cybin.pyx", line 439, in cybin.TCyBinaryProtocol.read_message_begin
cybin.ProtocolError: No protocol version header

phoenix - hbase 연동 이슈

환경

  • hbase 2.3.1
  • phoenix 5.0.0

phoenix에서 queryserver를 실행하고 sqlline-thin.py 로 접속했을 때 아래와 같이 나타난다.

> ./bin/queryserver.py start
> ./sqlline-thin.py http://localhost:8765
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.phoenix.shaded.com.google.protobuf.UnsafeUtil (file:/mnt/c/Users/jgb71/git/data-pipeline/apache-phoenix-5.0.0-HBase-2.0-bin/phoenix-5.0.0-HBase-2.0-thin-client.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.phoenix.shaded.com.google.protobuf.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

hdfs missing block

There are 1 missing blocks. The following files may be corrupted:


blk_1073741827 /logs/simpletable.csv
Please check the logs or run fsck in order to identify the missing blocks. See the Hadoop FAQ for common causes and potential solutions.

HDFS web에 접속했을때 위와 같은 Missing block 현상이 발생

hbase docker 셋팅 이슈

현재 실행

  • zookeeper
  • HMaster
  • RegionServer
  • Rest
  • Thrift

zookeeper 가 제대로 실행되지 않아 아래 서비스들도 실행 안되는것으로 보임

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.