Code Monkey home page Code Monkey logo

byconity-docker's Introduction

Welcome to ByConity

ByConity Arch 2023

Byconity, an advanced database management system, is a derivative of ClickHouse DBMS, building upon the robust codebase from ClickHouse v21.8. However, Byconity's development path has since diverged, thanks in part to insights gained from Snowflake's architecture.

Our key innovations include the introduction of a compute-storage separation architecture, a state-of-the-art query optimizer, multiple stateless workers, and a shared-storage framework. These enhancements, inspired by both ClickHouse's strength and Snowflake's innovative approach, offer substantial performance and scalability improvements.

We deeply appreciate the profound contributions from the ClickHouse team, with whom we had an early discussion to share our open-source vision and technical implementations. However, given the substantial architectural differences that emerged in our modifications, the ClickHouse team assessed that integrating these changes directly into the original ClickHouse project was not feasible. As a result, we decided to launch Byconity as an independent downstream open-source project. This approach preserves the integrity of both projects while offering distinct solutions for diverse database management needs.

Query Large Scale Data with Speed and Precision When dealing with large-scale data, performance is crucial. Byconity shines in this aspect by providing powerful querying capabilities that excel in large-scale environments. With Byconity, you can extract valuable insights from vast amounts of data quickly and accurately.

Break Down Data Silos with Byconity Data silos pose significant challenges in data management. With different systems and processes often resulting in isolated islands of data, it hampers data analysis and insights. Byconity addresses this issue by seamlessly ingesting both batch-loaded data and streaming data, thus enabling your systems to break down silos for smoother data flow.

Designed for the Cloud, Flexible for Your Needs Byconity is designed with a cloud-native approach, optimized to take full advantage of the cloud's scalability, resilience, and ease of deployment. It can work seamlessly on both Kubernetes clusters and physical clusters, offering you the flexibility to deploy in the environment that best meets your requirements. This broad compatibility ensures that you can leverage Byconity's benefits, irrespective of your infrastructure.

Benefits

  • Unified Data Management: Byconity eliminates the need to maintain separate processes for batch and streaming data, making your systems more efficient.
  • High-Performance Data Querying : Byconity's robust querying capabilities allow for quick and accurate data retrieval from large-scale datasets.
  • Avoid Data Silos : By handling both batch and streaming data, Byconity ensures all your data can be integrated, promoting better insights.
  • Cloud-Native Design : Byconity is built with a cloud-native approach, allowing it to efficiently leverage the advantages of the cloud and work seamlessly on both Kubernetes and physical clusters.
  • Open Source: Being an open-source project, Byconity encourages community collaboration. You can contribute, improve, and tailor the platform according to your needs.

Build and Run ByConity

The easiest way to build ByConity is built in docker dev-env. If you build on your local machine, the ByConity executable file depends on the Foundation DB library libfdb_c.so. So to run it, we need to install the FoundationDB client package. This link tells how to install. We can download the client package from FoundationDB GitHub release pages, for example here.

In case you want to build ByConity in the metal machine, follow this guide

Using Docker Compose would be convenient for running a ByConity cluster.

Useful Links

byconity-docker's People

Contributors

canhld94 avatar clark0 avatar dmthuc avatar hustnn avatar kevinthfang avatar kunth avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

byconity-docker's Issues

docker-compose up failed

image

" worker-write Error 34.0s0 ⠿ daemon-manager Error"

Error response from daemon: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: "

408 Request Time-out

\nYour browser didn't send a complete request in time.\n\n"

hi , I try to use "docker-compose up" to bring a cluster up ,but I got the error above
Would you please help me?

Failing to bring up ByConity Cluster

I've been trying to use Byconity with docker and I am running into some errors.

After using the 'docker-compose up' command I get this

Lark20230426-203110

After waiting some time for the containers to be ready I then use the test commands given in the documentation.

When I use 'curl '127.0.0.1:8123/?query=select%20count()%20from%20system.one' I get '1' as the response.

However when I use the other test commands to indicate the read and write workers are working

(curl '127.0.0.1:8123/?query=select%20count()%20from%20cnch(vw_default,system,one)'
curl '127.0.0.1:8123/?query=select%20count()%20from%20cnch(vw_write,system,one)'
)

I get this error:

Code: 46. DB::Exception: Unknown table function cnch: or incorrect parameterized view. (UNKNOWN_FUNCTION) (version 23.3.1.1561 (official build))

occurred

2023.01.11 08:59:07.300901 [ 43 ] {} TxnGCThread: Error occurs during daemon TxnGCThread execution: Code: 439, e.displayText() = DB::Exception: Cannot schedule a task: no free thread (timeout=0) (threads=16, jobs=16) SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

  1. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int) @ 0x18e345d2 in /opt/byconity/usr/bin/clickhouse
  2. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0xb230820 in /opt/byconity/usr/bin/clickhouse
  3. DB::Exception::Exception<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, unsigned long, unsigned long&>(int, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, unsigned long&&, unsigned long&) @ 0xb26bc72 in /opt/byconity/usr/bin/clickhouse
  4. void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda'(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&)::operator()(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) const @ 0xb26c6de in /opt/byconity/usr/bin/clickhouse
  5. void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional) @ 0xb26820b in /opt/byconity/usr/bin/clickhouse
  6. ThreadPoolImpl::scheduleOrThrow(std::__1::function<void ()>, int, unsigned long) @ 0xb269926 in /opt/byconity/usr/bin/clickhouse
  7. DB::DaemonManager::DaemonJobTxnGC::cleanTxnRecords(std::__1::vector<DB::TransactionRecord, std::__1::allocatorDB::TransactionRecord > const&) @ 0x154af26c in /opt/byconity/usr/bin/clickhouse
  8. DB::

docker-compose up fail

[lighthouse@VM-8-5-centos byconity]$ docker-compose up
ERROR: build path /home/lighthouse/byconity/docker/client either does not exist, is not accessible, or is not a valid URL.

do i need to create a dir client manully, before i start to run "docker-compose up"

a problem about hdfs port

Hello there,

After building the system using the command docker-compose up, I found that port 9000 is exposed on the ip 0.0.0.0
Pasted Graphic 10

As far as I can see, port 9000 is designed for the inner communication of hdfs process, it should be listened on the local ip(e.g. 127.0.0.1) usually.

So do you think is it a problem here?

After truncate, the hdfs file still exists

server application log
mmexport1673423324319

Linux 93861685cbd4 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 GNU/Linux

Debian GNU/Linux 9

truncate table helloworld.my_first_table;

kafka数据丢失

应用服务器临时断电重启应用后,发现kafka存储表中数据总数和kafka客户端工具显示的数据总数不一致,两者的offset是一致的。

resource-manager-0 error

The cluster is started using docker-compose.yml.multiworkers. The following error log appears after the resource-manager-0 container is started.

2023.01.29 08:10:56.358771 [ 1 ] {} ElectionController: Code: 49, e.displayText() = DB::Exception: Worker group wg_default already exists in VW vw_default SQLSTATE: HY000, Stack trace (when copying this message, always include the lines below):

  1. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int) @ 0x18e345d2 in /opt/byconity/usr/bin/clickhouse
  2. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0xb230820 in /opt/byconity/usr/bin/clickhouse
  3. DB::ResourceManagement::VirtualWarehouse::addWorkerGroup(std::__1::shared_ptrDB::ResourceManagement::IWorkerGroup const&, bool) @ 0x135dc005 in /opt/byconity/usr/bin/clickhouse
  4. DB::ResourceManagement::ResourceManagerController::initialize() @ 0x135c17da in /opt/byconity/usr/bin/clickhouse
  5. DB::ResourceManagement::ElectionController::pullState() @ 0x135c6e80 in /opt/byconity/usr/bin/clickhouse
  6. DB::ResourceManagement::ElectionController::onLeader() @ 0x135c6866 in /opt/byconity/usr/bin/clickhouse
  7. DB::ResourceManagement::ElectionController::ElectionController(DB::ResourceManagement::ResourceManagerController&) @ 0x135c613f in /opt/byconity/usr/bin/clickhouse
  8. DB::ResourceManagement::ResourceManagerController::ResourceManagerController(std::__1::shared_ptr<DB::Context const>) @ 0x135ba938 in /opt/byconity/usr/bin/clickhouse
  9. DB::ResourceManager::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > > > const&) @ 0x135a8896 in /opt/byconity/usr/bin/clickhouse
  10. Poco::Util::Application::run() @ 0x18dbfcaa in /opt/byconity/usr/bin/clickhouse
  11. DB::ResourceManager::run() @ 0x135a7fb5 in /opt/byconity/usr/bin/clickhouse
  12. Poco::Util::ServerApplication::run(int, char**) @ 0x18dd1b1b in /opt/byconity/usr/bin/clickhouse
  13. mainEntryClickHouseResourceManager(int, char**) @ 0x135aa0eb in /opt/byconity/usr/bin/clickhouse
  14. main @ 0xb22aae8 in /opt/byconity/usr/bin/clickhouse
  15. __libc_start_main @ 0x2409b in /lib/x86_64-linux-gnu/libc-2.28.so
  16. _start @ 0xb22a2ea in /opt/byconity/usr/bin/clickhouse
    (version 21.8.7.1)

Permission denied: user=clickhouse, access=WRITE

I encountered a permission denied error when trying to import csv file using clickhouse-client 18.16.1. The error message indicates that the user ‘clickhouse’ does not have write access to the file.

 clickhouse-client  --port=9000 --query='INSERT INTO t FORMAT CSV' < ~/t.csv

Received exception from server (version 21.8.7):
Code: 1000. DB::Exception: Received from localhost:9000, 127.0.0.1. DB::Exception: Access to file denied: Permission denied: user=clickhouse, access=WRITE, inode="/":hadoop:supergroup:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:506)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:346)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermissionWithContext(FSPermissionChecker.java:370)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:240)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1943)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1927)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1886)
at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:60)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3405)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1159)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:740)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:604)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:572)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976)
SQLSTATE: HY000

Unable to start the docker in local deploy

Hi!

I am getting the below error after running docker-compose up -

0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0Warning: Transient problem: timeout Will retry in 5 seconds. 10 retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 9
worker-default-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 9
worker-write-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 9
daemon-manager-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: timeout Will retry in 5 seconds. 9 retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 8
worker-default-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 8
worker-write-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 8
daemon-manager-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: timeout Will retry in 5 seconds. 8 retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 7
worker-default-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 7
daemon-manager-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 7
worker-write-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: timeout Will retry in 5 seconds. 7 retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 6
worker-default-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 6
worker-write-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 6
daemon-manager-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 5
worker-default-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 5
worker-write-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 5
daemon-manager-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0Warning: Transient problem: timeout Will retry in 5 seconds. 6 retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 4
worker-default-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 4
worker-write-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 4
daemon-manager-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: timeout Will retry in 5 seconds. 5 retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 3
worker-default-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 3
worker-write-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 3
daemon-manager-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: timeout Will retry in 5 seconds. 4 retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 2
worker-default-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 2
daemon-manager-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 2
worker-write-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: timeout Will retry in 5 seconds. 3 retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 1
worker-default-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 1
daemon-manager-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: connection refused Will retry in 5 seconds. 1
worker-write-0 | Warning: retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: timeout Will retry in 5 seconds. 2 retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to server port 21557: Connection refused
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to server port 21557: Connection refused
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to server port 21557: Connection refused
worker-default-0 exited with code 7
daemon-manager-0 exited with code 7
worker-write-0 exited with code 7
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: timeout Will retry in 5 seconds. 1 retries left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: tso-0
server-0 exited with code 6

I am attaching full log as well.
full-log.txt

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.