jaegeunbang / data-pipeline Goto Github PK
View Code? Open in Web Editor NEWdata pipeline을 docker 로 셋팅해보기 위함
data pipeline을 docker 로 셋팅해보기 위함
hdfs가 비정상적으로 종료되거나, 네임노드가 start 되는 과정에서 safe-mode가 발생한다.
그러다 보니, hbase의 데이터를 hdfs에 써야 하는 상황에서, safe mode로 인해 data를 write하지 못함.
safe-mode
에서는 hdfs에 파일을 write 하지 못한다.
safe mode를 off하기 위해 아래와 같이 명령어를 사용한다.
hdfs dfsadmin -safemode leave
또는 hdfs dfsadmin -safemode forceExit
hbase 구축후 데이터를 python으로 넣기 위해 happyhbase 라이브러리를 사용하고자 했음.
허나 아래와 같은 이슈가 발생함
Traceback (most recent call last):
File "./insert_hbase.py", line 5, in <module>
print(connection.tables())
File "/usr/local/lib/python3.8/site-packages/happybase/connection.py", line 242, in tables
names = self.client.getTableNames()
File "/usr/local/lib/python3.8/site-packages/thriftpy2/thrift.py", line 219, in _req
return self._recv(_api)
File "/usr/local/lib/python3.8/site-packages/thriftpy2/thrift.py", line 231, in _recv
fname, mtype, rseqid = self._iprot.read_message_begin()
File "thriftpy2/protocol/cybin/cybin.pyx", line 439, in cybin.TCyBinaryProtocol.read_message_begin
cybin.ProtocolError: No protocol version header
환경
phoenix에서 queryserver를 실행하고 sqlline-thin.py 로 접속했을 때 아래와 같이 나타난다.
> ./bin/queryserver.py start
> ./sqlline-thin.py http://localhost:8765
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.phoenix.shaded.com.google.protobuf.UnsafeUtil (file:/mnt/c/Users/jgb71/git/data-pipeline/apache-phoenix-5.0.0-HBase-2.0-bin/phoenix-5.0.0-HBase-2.0-thin-client.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.phoenix.shaded.com.google.protobuf.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
There are 1 missing blocks. The following files may be corrupted:
blk_1073741827 /logs/simpletable.csv
Please check the logs or run fsck in order to identify the missing blocks. See the Hadoop FAQ for common causes and potential solutions.
HDFS web에 접속했을때 위와 같은 Missing block 현상이 발생
docker container 자동 삭제 및 재생성을 위한 스크립트 생성 및 관련 자료 찾기
현재 실행
zookeeper 가 제대로 실행되지 않아 아래 서비스들도 실행 안되는것으로 보임
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.