Code Monkey home page Code Monkey logo

bbc1's Introduction

Core system of BBc-1 (Beyond Blockchain One)

License CircleCI

This project is a Python-based reference implementation of BBc-1, a trustable system of record keeping beyond blockchains.

The design paper (white paper) and the analysis paper are available here and here. BBc-1 is inspired from blockchain technologies like Bitcoin, Ethereum, Hyperledger projects, and so on. BBc-1 is a simple but reliable distributed ledger system in contrast with huge and complicated existing blockchain platforms. The heart of BBc-1 is the transaction data structure and the relationship among transactions, which forms a graph topology. A transaction should be signed by the players who are the stake holders of the deal. BBc-1 achieves data integrity and data transparency by the topology of transaction relationship and signatures on transactions. Simply put, BBc-1 does not have blocks, and therefore, requires neither mining nor native cryptocurrency. BBc-1 can be applied to both private/enterprise use and public use. BBc-1 has a concept of domain for determining a region of data management. Any networking implementation (like Kademlia for P2P topology management) can be applied for each domain. Although there are many TODOs in BBc-1, this reference implementation includes most of the concept of BBc-1 and would work in private/enterprise systems. When sophisticated P2P algorithms are ready, BBc-1 will be able to support public use cases.

The source codes in this repository is a platform of BBc-1 and bbc_core.py is the main process of a core node. The APIs of BBc-1 is defined in bbc_app.py and bbclib.py. So application developers should import them in your apps. For building a BBc-1 system, bbc1 package needs to be installed in the hosts and you need to run bbc_core.py on each host. In order to configure the BBc-1 network, the utilities in utils/ directory are available. They are a kind of BBc-1 application, so that you can develop your own management tools.

For the details, please read documents in docs/ directory. Not only documents but slide decks (PDF) explain the design of the BBc-1 and its implementation.

Trouble shooting

Installing bbc1 through pip sometimes fails owing to pip cache trouble. It might occur in the case that you terminate the install process during libbbcsig building process. This leads to a defect in the pip cache of libbbcsig module, and resulting in fail installing forever.

To solve the problem, you need to remove pip cache or pip install without using cache. How to solve it is explained below.

Solution 1

Removing pip cache directory is a fundamental solution to this problem. The cache directories in various OS platform are as follows:

  • Linux and Unix
    • ~/.cache/pip
  • macOS
    • ~/Library/Caches/pip
  • Windows
    • %LocalAppData%\pip\Cache

After removing the cache directory, install bbc1 module again.

python3 -mvenv venv
. venv/bin/activate
pip install bbc1

Solution 2

Disabling cache and re-installing the module is another solution, which is easier way.

python3 -mvenv venv
. venv/bin/activate
pip --no-cache-dir install -I bbc1 

Recent changes regarding DB meta table

In the update to v1.3, a meta table of DB is updated to support timestamp-based search. The main table for transaction data itself remains unchanged, so that just updating meta table is need for migration. Migration tool is provided by utils/db_migration_tool.py. How to migrate is described below.

Migration step 1

Install new module by pip command if you use pip bbc1 module. Then, stop the old bbc_core.py process and start new one. The meta table of DBs are automatically upgraded when the new bbc_core.py boots up.

Migration step 2

Run db_migration_tool.py by specifying target working directory. If using pip module, the tool can be invoked directly as follows:

db_migration_tool.py -w 'working_dir' 

You will see records are upgrading by the tool. In the case of high transaction rate, some records might remain unchanged. In that case, re-run the tool.

Note that you have to perform step 1 and 2 for each working directory because the process reads config file in the working directory and upgrades the DBs specified in the config.

Documents

Some documents are available in docs/.

Environment

  • Python

    • Python 3.5.0 or later
    • virtualenv is recommended
      • python -mvenv venv
    • In some environment, pipenv does not work well.
      • Some bugs seems to be in the installation scripts. So, please do not use pipenv now.
  • tools for macOS by Homebrew

    brew install libtool automake python3
    pip3 install virtualenv
    
  • tools for Linux (Ubuntu 16.04 LTS, 18.04 LTS)

    sudo apt-get update
    sudo apt-get install -y git tzdata openssh-server python3 python3-dev python3-pip python3-venv libffi-dev net-tools autoconf automake libtool libssl-dev make
    

Quick start

From source code in github

  1. Install development tools (libtool, automake)

  2. Install python and pip

  3. Clone this project

  4. Prepare OpenSSL-based library in the root directory

    sh prepare.sh
    
  5. Install dependencies by the following command (in the case of python 3.6)

    python -mvenv venv
    source venv/bin/activate
    pip install -r requirements.txt
    
  6. Start bbc_core.py on a terminal

    cd core
    python bbc_core.py
    
  7. Start a sample app in another terminal (should be initially at bbc1/ top directory)

    pipenv shell
    cd examples
    python file_proof.py arg1 arg2..
    

Use pip

  1. Install development tools (libtool, automake)
  2. Install python and pip
  3. Install BBc1 by pip
    python -mvenv venv
    source venv/bin/activate
    pip install bbc1
    

Use docker (See README.md in docker/)

  1. Install docker on your host
  2. Clone this project
  3. Build docker image If you want source codes in your container,
    cd docker
    ./docker-bbc1.sh gitbuild
    
    or, if you just want to use BBc-1,
    cd docker
    ./docker-bbc1.sh pipbuild
    
  4. Run a docker container
    ./docker-bbc1.sh start
    
  5. Log in to the container
    ./docker-bbc1.sh shell
    
    or
    ssh -p 10022 root@localhost
    
    The initial password is "bbc1".

working directory

The working directory of BBc-1 on the docker container is mounted on docker/data/.bbc1/. You will find a config file, ledger DB and file storage directory in the working directory.

Files/Directories

  • bbc1/core/
    • core functions of BBc-1
  • utils/
    • BBc-1 system configuration utilities
  • examples/
    • sample applications on BBc-1
  • docker/
    • docker environments
  • tests/
    • test codes for pytest
  • docs/
    • docs about BBc-1 and its reference implementation
  • somewhere/.bbc1/
    • default working directory name of bbc_core.py
  • requirements.txt
    • python modules to be required
  • setup.py
  • MANIFEST.in
  • prepare.py
    • for creatign python modules
  • prepare.sh
    • setup script

bbc1's People

Contributors

araya-avanoa avatar arkth-araya avatar beyond-blockchain avatar chike0905 avatar ebiiim avatar imony avatar junkurihara avatar kichinosukey avatar ks91 avatar oopth avatar quvox avatar renovate[bot] avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

bbc1's Issues

Pipenv install fails

Error with "pyopenssl 18.0.0 has requirement cryptography>=2.2.1, but you'll have cryptography 2.1.4 which is incompatible." results in failure of pip installation.

Domain is unsupported in ledger subsystem

Domain is currently not supported in the ledger subsystem.
Also, the ledger subsystem code contains redundant db functions copied from bbc_ledger.
(bbc_ledger may have a problem when tables with arbitrary indices are created in the auxiliary db.)

[bbclib] Unreasonable structure of transactions

I just checked the current structure of bbclib, and found that the transaction could be composed in bson and bson+zlib in addition to the original dedicated binary format. The format of a transaction (bson, bson+zlib or dedicated one) is described OUTSIDE data structure. In fact, it is specified by the first 2 bytes of the output of bbclib, i.e., a transaction data, as a flag. I believe that from the following reason, either one of bson or dedicated format should be eliminated from the core structure of bbclib:

  • Signatures do not protect the flag in a transaction currently. This implies that attackers can modify such flags of transactions and then users cannot correctly decode such modified transactions. Hence the system could be compromised.
  • The definition bbclib is currently ambiguous. IMHO, it should be a small 'core' module to sign/verify/compose/decompose transactions, and should not be a composer of wire formats.

From the above observations, I would like to propose to define (de)composers of wire format, i.e., bson and bson+zlib, as output plugins of bbc1, and fix the transaction format only with the dedicated one.

[pip] `libbbcsig.dylib` not found

I have installed the latest version of bbc1 using pip.
But from bbc1.core import bbclib does not work and returns an OSError indicating that libbbcsig.dylib does not exist. (I confirmed both Pipenv and venv)

Python 3.7.2 (default, Feb 12 2019, 08:15:36) 
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from bbc1.core import bbclib
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/{PATH_TO_PIPENV}/lib/python3.7/site-packages/bbc1/core/bbclib.py", line 33, in <module>
    import bbc1.core.libs.bbclib_utils as bbclib_utils
  File "/{PATH_TO_PIPENV}/lib/python3.7/site-packages/bbc1/core/libs/bbclib_utils.py", line 29, in <module>
    from bbc1.core.libs.bbclib_config import DEFAULT_ID_LEN
  File "/{PATH_TO_PIPENV}/lib/python3.7/site-packages/bbc1/core/libs/bbclib_config.py", line 22, in <module>
    from bbc1.core.libs.bbclib_keypair import KeyType
  File "/{PATH_TO_PIPENV}/lib/python3.7/site-packages/bbc1/core/libs/bbclib_keypair.py", line 31, in <module>
    libbbcsig = cdll.LoadLibrary(os.path.join(directory, "libbbcsig.dylib"))
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 434, in LoadLibrary
    return self._dlltype(name)
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 356, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/{PATH_TO_PIPENV}/lib/python3.7/site-packages/bbc1/core/libs/libbbcsig.dylib, 6): image not found

How can I install it successfully?

Sophisticated transaction search methods

Currently, the bbc_core offers a very simple search method to retrieve the transaction with the specified transaction_id. Probably, we need to consider more convenient functions.

Proof of existence by using Cross Reference

In BBC-1, inter-domain collaboration through cross reference mechanism is one of the important features. At this point, BBC-1 does not have cross reference searching function in a inter-domain environment.

System statistics

Currently, BBc-1 does not have any function to gather system statistics.

Conflict with source code and pip module

requirements.txt includes "bbc1". This causes the conflict between the source codes and the installed pip module in the case that we develop a new function in the source codes. Even though a new function is implemented in the source codes, the installed pip module "bbc1" is used.

"bbc1" in requirements.txt is for Dockerfile_pip. One of the possible solutions would be that Dockerfile_pip installs bbc1 by using "pip install bbc1" instead of pip install -r requirements.txt.

When importing bbc_app RuntimeError has occurred in flask

macOS
Python 3.7.3
Flask 1.0.1
bbc1 1.3

For example, run below code.

from flask import Flask

from bbc1.core import bbc_app

app = Flask(__name__)

@app.route('/')
def test():
    return 'test'

if __name__ == '__main__':

    app.run()

The debugger display below.
monkey.patch_all() seems to cause it.
Comment out this line from bbc_app.py, warning does not appear but I couldn't understand its comment out is good or bad.

~/bbc1/venv/lib/python3.7/site-packages/bbc1/core/bbc_app.py:19: MonkeyPatchWarning: Monkey-patching outside the main native thread. Some APIs will not be available. Expect a KeyError to be printed at shutdown.
  monkey.patch_all()
~/bbc1/venv/lib/python3.7/site-packages/bbc1/core/bbc_app.py:19: MonkeyPatchWarning: Monkey-patching not on the main thread; threading.main_thread().join() will hang from a greenlet
  monkey.patch_all()

My flask app would be affected this.
I couldn't research sufficiently to solve this, but I thought it could be better to make this issue than not to do.
Thank you,

Documents are needed

The current document sets are very poor. Documents should be added and updated as they are ready. So, a new branch "feature/doc-update" was created. The update in the branch will be merged even without version update.

Library (or app) for decoupling identifier and public key

In bitcoin system, an address (which corresponds to user_id in bbc1) is created from its public key. In such a case, if a user forgets (or loses) his/her key, the user can never use his asset.
We plan to develop a library to decouple an identifier and a public key and allow user to update key bindings.

Need update for py-bbclib v1.5

New classes, BBcAssetRaw and BBcAssetHash, are introduced in py-bbclib v1.5.
The header version should be 3 to support these classes.

data_handler needs to be understand BBcAssetRaw and BBcAssetHash to create appropriate meta tables.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

  • Update dependency pytest to v6.2.5
  • Update circleci/python Docker tag to v3.10
  • Update dependency attrs to v21.4.0
  • Update dependency cffi to v1.17.0
  • Update dependency py to v1.11.0
  • Update dependency pycparser to v2.22
  • Update dependency python-dateutil to v2.9.0.post0
  • Update dependency zope.event to v4.6
  • Update dependency zope.interface to v5.5.2
  • Update dependency attrs to v24
  • Update dependency chardet to v5
  • Update dependency greenlet to v3
  • Update dependency iniconfig to v2
  • Update dependency mysql-connector-python to v9
  • Update dependency packaging to v24
  • Update dependency pluggy to v1
  • Update dependency pyOpenSSL to v24
  • Update dependency pyparsing to v3
  • Update dependency pytest to v8
  • Update dependency zope.event to v5
  • Update dependency zope.interface to v7
  • Update ubuntu Docker tag to v24
  • ๐Ÿ” Create all rate-limited PRs at once ๐Ÿ”

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

circleci
.circleci/config.yml
  • circleci/python 3.9
  • circleci/python 3.9
dockerfile
docker/Dockerfile_git
  • ubuntu bionic
docker/Dockerfile_pip
  • ubuntu bionic
pip_requirements
requirements.txt
  • attrs ==21.2.0
  • bson ==0.5.10
  • certifi ==2020.12.5
  • cffi ==1.14.5
  • chardet ==4.0.0
  • click ==7.1.2
  • cryptography ==3.4.7
  • Flask ==1.1.2
  • gevent ==21.1.2
  • greenlet ==1.1.0
  • idna ==2.10
  • iniconfig ==1.1.1
  • itsdangerous ==1.1.0
  • Jinja2 ==2.11.3
  • MarkupSafe ==1.1.1
  • msgpack-python ==0.5.6
  • mysql-connector-python ==8.0.24
  • packaging ==20.9
  • pluggy ==0.13.1
  • protobuf ==3.16.0
  • py ==1.10.0
  • py-bbclib ==1.6.1
  • pycparser ==2.20
  • pyOpenSSL ==20.0.1
  • pyparsing ==2.4.7
  • pytest ==6.2.4
  • python-dateutil ==2.8.1
  • requests ==2.25.1
  • six ==1.16.0
  • toml ==0.10.2
  • urllib3 ==1.26.4
  • Werkzeug ==1.0.1
  • zope.event ==4.5.0
  • zope.interface ==5.4.0
pip_setup
setup.py
  • pyOpenSSL >=16.2.0
  • jinja2 >=2.8.1
  • requests >=2.12.4
  • gevent >=1.2.1
  • cryptography >=2.1.4
  • pytest >=5.3.0
  • msgpack-python >=0.4.8
  • mysql-connector-python >=8.0.5
  • py-bbclib >=1.6
  • Flask >=0.10.1,<=1.0.1

  • Check this box to trigger a request for Renovate to run again on this repository

Prepare.sh script fails in some cases

In version 1.1, libbbcsig is updated, so it needs to be rebuild by running prepare.sh.

However, if libs/ directory exists in the top directory, prepare.sh script will fail.
Before running prepare.sh, remove libs directory.

Cannot add signature in an unpacked transaction object

In unpacking transaction data into transaction object, the mapping info between user_id and the index of BBcSignature list in a transaction is not correctly recovered if the transaction contains BBcReference and BBcWitness objects concurrently.

inconsistent data format in BBcAsset (asset_file_digest)

In the packing method of BBcAsset (bbclib_asset.py), the "asset_file_digest" is packed with length value for data packing. However, for calculating "asset_id", asset_file_digest is packed without length value. This is confusing and inconsistent in terms of format design (it may be just a kind of bug). Even though it is not fatal issue, this should be fixed.

"asset_id" is just an identifier. So, adding length field in packing for calculation of asset_id would be OK.

In addition, asset_file_digest is truncated by id_length in some cases in bbclib_asset.py. It is also inappropriate and should be fixed.

Backward compatible code fails upon serialization

When testing some old code written for the past core <v1.2 (bbc1-lib-std and bbc1-lib-tokens)
by modifying to import bbclib as follows:
from bbc1.core.compat import bbclib

results in:
E AttributeError: 'BBcTransaction' object has no attribute 'pack'
during serialization.
(bbc1/core/bbclib.py:90: AttributeError)

File digest in BBcAsset must be full length

In bbclib_asset.py, asset_file_digest is cut off based on id_length as follows:

self.asset_file_digest = hashlib.sha256(bytes(asset_file)).digest()[:self.id_length]

The digest is not ID, so it must not be cut off.

'archive-all' is not a git command

Environment

  • macOS Sierra
  • Python 3.6.3 (with pyenv)
% ./docker-bbc1.sh gitbuild
git: 'archive-all' is not a git command. See 'git --help'.
mv: rename ../bbc1.tar.gz to ./bbc1.tar.gz: No such file or directory
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /Users/yuki-sh/Projects/bbc1/Dockerfile_git: no such file or directory
rm: bbc1.tar.gz: No such file or directory

https://github.com/beyond-blockchain/bbc1/blob/develop/docker/docker-bbc1.sh#L35

core/ethereum set up problem

core/ethereum cannot be set up when "How to use" in README is followed.
It fails at "5. Deploy BBcAnchor smart contract".

This is because "genesis.json" and "populus.json" are wrongfully included in the release.
They need to be removed from the repository.

Also, in README, there need to be a new line before "5. Deploy BBcAnchor smart contract".

v0.10 file_proof sample fails to get a file

$ python file_proof.py get test.txt

fails with the following message:

Traceback (most recent call last):
  File "file_proof.py", line 487, in <module>
    get_file(file=parsed_args.target_file)
  File "file_proof.py", line 263, in get_file
    ret = bbc_app_client.search_asset(asset_group_id, fileinfo["asset_id"])
AttributeError: 'BBcAppClient' object has no attribute 'search_asset'

v0.10 will fix basic features of BBc-1

The next minor version up, v0.10, will have the basic features enough to operate various systems.
The next version after v0.10 will be v1.0, with bug fixing, refactoring, package rearrangement and document update over v0.10.

Cannot create readthedoc.io page

API documents in readthedoc.io is something wrong (no detailed information is output). The documentation tool "sphinx" works fine in a local environment, but not in readthedoc.io environment.

A transaction with multiple asset_group_ids is not properly registered to the ledger

In the case that a transaction to be inserted includes multiple BBcEvents and multiple asset_group_ids, the transaction should be registered to the ledgers of these asset_group_ids. However, the method insert_transaction() in bbc_core.py requires a parameter of asset_group_id, leading to registering just once for the asset_group_id.

introspection (active check of falsification)

Currently, bbc_core checks whether the accessing transaction has been falsified or not. This means that the check is conducted passively. We probably need a kind of active check functionality.

Object type asset body sometimes leads to inconsistent digests in different programing language platforms

MessagePack is used for object type asset body. Decoding a packed data by golang bbclib in python is OK and vice versa. However, the transaction digest calculated in python is sometimes different from that in golang because encoding rule is little bit different.

A value (char, short, int, long,,,) seems to introduce such an inter-platform gap. So, the object type asset body should be in {"string key": "string value"} type for inter-platform compatibility.

Test always fails

The tag in the top README.md always shows failing.
Tests in TravisCI seems to fail in setup.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.