The cfxdb from crossbario

Complete CI coverage for all DB tables

Every DB table defined in Flatbuffers schemata should have test coverage like in https://github.com/crossbario/cfxdb/blob/master/cfxdb/tests/test_xbr.py, with at least 2 test functions per table:

test_<table>_roundtrip
test_<table>_roundtrip_perf

Here is the current complete list:

Important: we should focus on the xbr and xbrnetwork tables first, as for the other classes this issue here depends on #9

arealm::AnonymousPrincipal (schema)
arealm::ApplicationRealm (schema)
arealm::ApplicationRealmRole (schema)
arealm::Authenticator (schema)
arealm::AuthenticatorPrincipal (schema)
arealm::CryptosignPrincipal (schema)
arealm::Role (schema)
arealm::ScramPrincipal (schema)
arealm::TicketPrincipal (schema)
arealm::TlsPrincipal (schema)
arealm::WampCraPrincipal (schema)
common::KeyValue (schema)
eventstore::Event (schema)
eventstore::Publication (schema)
eventstore::Session (schema)
log::MasterNodeUsage (schema)
log::MasterRun (schema)
log::MNodeLog (schema)
log::MNodeSession (schema)
log::MRealmRun (schema)
log::MWorkerLog (schema)
meta::Attribute (schema)
meta::Comment (schema)
meta::Doc (schema)
meta::Reaction (schema)
meta::Tag (schema)
meta::TagDefinition (schema)
mrealm::ManagementRealm (schema)
mrealm::Node (schema)
mrealm::WebCluster (schema)
mrealm::WebClusterNodeMembership (schema)
user::ActivationToken (schema)
user::Organization (schema)
user::User (schema)
user::UserMrealmRoles (schema)
user::UserOrganizationRoles (schema)
xbr::Actor (schema)
xbr::Api (schema)
xbr::Block (schema)
xbr::Catalog (schema)
xbr::Market (schema)
xbr::Member (schema)
xbr::Offer (schema)
xbr::PayingChannelRequest (schema)
xbr::PaymentChannel (schema)
xbr::PaymentChannelBalance (schema)
xbr::TokenApproval (schema)
xbr::TokenBalance (schema)
xbr::TokenTransfer (schema)
xbr::Transaction (schema)
xbrnetwork::Account (schema)
xbrnetwork::UserKey (schema)
xbrnetwork::VerifiedAction (schema)

tid + signature for all on-chain object tables

Every DB table that represent an on-chain object should have these 2 fields defined in the flatbuffers table schema already:

    /// Transaction hash of the transaction this change was committed to the blockchain under.
    tid: [uint8] (ethhash);

    /// When signed off-chain and submitted via ``XBRCatalog.createCatalogFor``.
    signature: [uint8] (ethsig);

Note: the transaction id (tid) allows to associate the DB record insert/update with the on-chain ethereum transaction that originates the insert/update. The signature allows to store the signature of pre-signed transactions where the transaction submitter is different from the signer.

However, these 2 fields are not yet added to the Python DB table classes. The current list of DB tables representing XBR related on-chain objects which need to be expanded:

xbr::PaymentChannel (schema)

DB schema: fix actor types

there is 1 new value PROVIDER_CONSUMER == 3, which allows joining a market as both a consumer-actor and a provider-actor in one transaction

https://github.com/crossbario/xbr-protocol/blob/f2811532894e9a34228228e280cd67fc5b1fc6c6/contracts/XBRTypes.sol#L33

Expand DB scheme for catalogs and consent

there is some work to do after wamp-proto/wamp-xbr#90 has landed in this file
https://github.com/crossbario/cfxdb/blob/master/cfxdb/xbrnetwork.fbs

Catalogs
APIs
Consent

Expand DB classes for catalogs and consent

Update and create new DB classes here :

Catalog and Catalogs (MapUuidFlatBuffers) classes for XBRTypes.Catalog schema definition
Api and Apis (MapUuidFlatBuffers) classes for XBRTypes.Api schema definition
Consent and Consents classes for tbd

This depends on #3

CI: datetime roundtrip tests 1h off

________________________________________________________________ summary ________________________________________________________________
  flake8: commands succeeded
  mypy: commands succeeded
  yapf: commands succeeded
ERROR:   pytest: commands failed
  sphinx: commands succeeded


cfxdb/tests/user/test_organization.py::test_org_fbs_roundtrip FAILED
cfxdb/tests/user/test_organization.py::test_org_cbor_roundtrip FAILED
cfxdb/tests/user/test_user.py::test_user_fbs_roundtrip FAILED
cfxdb/tests/user/test_user.py::test_user_cbor_roundtrip FAILED

=============================================================== FAILURES ================================================================
_____________________________________________________ test_user_fbs_roundtrip_perf ______________________________________________________

user_fbs = <cfxdb.user.user_fbs.UserFbs object at 0x7fda1544ad30>, builder = <flatbuffers.builder.Builder object at 0x7fda154770b0>

    def test_user_fbs_roundtrip_perf(user_fbs, builder):
        # serialize to bytes (flatbuffers) from python object
        obj = user_fbs.build(builder)
        builder.Finish(obj)
        data = builder.Output()
    
        # create python object from bytes (flatbuffes)
        def loop():
            _user = UserFbs.cast(data)
            if True:
                assert _user.oid == user_fbs.oid
                assert len(str(_user.oid)) == 36
                assert _user.label == user_fbs.label
                assert _user.description == user_fbs.description
                assert _user.tags == user_fbs.tags
    
                assert _user.email == user_fbs.email
                assert _user.registered == user_fbs.registered
                assert _user.pubkey == user_fbs.pubkey
                assert len(_user.pubkey) == 64
    
        N = 5
        M = 50000
        samples = []
        print('measuring:')
        for i in range(N):
>           secs = timeit.timeit(loop, number=M)

cfxdb/tests/test_perf_serializer.py:193: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../cpy39/lib/python3.9/timeit.py:233: in timeit
    return Timer(stmt, setup, timer, globals).timeit(number)
../../../cpy39/lib/python3.9/timeit.py:177: in timeit
    timing = self.inner(it, self.timer)
<timeit-src>:6: in inner
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def loop():
        _user = UserFbs.cast(data)
        if True:
            assert _user.oid == user_fbs.oid
            assert len(str(_user.oid)) == 36
            assert _user.label == user_fbs.label
            assert _user.description == user_fbs.description
            assert _user.tags == user_fbs.tags
    
            assert _user.email == user_fbs.email
>           assert _user.registered == user_fbs.registered
E           assert datetime.datetime(2022, 3, 27, 3, 48, 57, 955976) == datetime.datetime(2022, 3, 27, 2, 48, 57, 955976)
E            +  where datetime.datetime(2022, 3, 27, 3, 48, 57, 955976) = <cfxdb.user.user_fbs.UserFbs object at 0x7fda149dbf40>.registered
E            +  and   datetime.datetime(2022, 3, 27, 2, 48, 57, 955976) = <cfxdb.user.user_fbs.UserFbs object at 0x7fda1544ad30>.registered

cfxdb/tests/test_perf_serializer.py:184: AssertionError
_____________________________________________________ test_user_cbor_roundtrip_perf _____________________________________________________

user_cbor = <cfxdb.user.user.User object at 0x7fda15409280>

    def test_user_cbor_roundtrip_perf(user_cbor):
        # serialize to bytes (cbor) from python object
        obj = user_cbor.marshal()
        data = cbor2.dumps(obj)
    
        # create python object from bytes (cbor)
        def loop():
            _obj = cbor2.loads(data)
            _user = User.parse(_obj)
            if True:
                assert _user.oid == user_cbor.oid
                assert _user.label == user_cbor.label
                assert _user.description == user_cbor.description
                assert _user.tags == user_cbor.tags
                assert _user.email == user_cbor.email
                assert _user.registered == user_cbor.registered
                assert _user.pubkey == user_cbor.pubkey
    
        N = 5
        M = 50000
        samples = []
        print('measuring:')
        for i in range(N):
>           secs = timeit.timeit(loop, number=M)

cfxdb/tests/test_perf_serializer.py:228: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../cpy39/lib/python3.9/timeit.py:233: in timeit
    return Timer(stmt, setup, timer, globals).timeit(number)
../../../cpy39/lib/python3.9/timeit.py:177: in timeit
    timing = self.inner(it, self.timer)
<timeit-src>:6: in inner
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def loop():
        _obj = cbor2.loads(data)
        _user = User.parse(_obj)
        if True:
            assert _user.oid == user_cbor.oid
            assert _user.label == user_cbor.label
            assert _user.description == user_cbor.description
            assert _user.tags == user_cbor.tags
            assert _user.email == user_cbor.email
>           assert _user.registered == user_cbor.registered
E           assert datetime.datetime(2022, 3, 27, 3, 48, 58, 138489) == datetime.datetime(2022, 3, 27, 2, 48, 58, 138489)
E            +  where datetime.datetime(2022, 3, 27, 3, 48, 58, 138489) = <cfxdb.user.user.User object at 0x7fda154095e0>.registered
E            +  and   datetime.datetime(2022, 3, 27, 2, 48, 58, 138489) = <cfxdb.user.user.User object at 0x7fda15409280>.registered

cfxdb/tests/test_perf_serializer.py:220: AssertionError
________________________________________________________ test_org_fbs_roundtrip _________________________________________________________

org_fbs = <cfxdb.user.organization_fbs.OrganizationFbs object at 0x7fda149db6a0>
builder = <flatbuffers.builder.Builder object at 0x7fda1543ed60>

    def test_org_fbs_roundtrip(org_fbs, builder):
        # serialize to bytes (flatbuffers) from python object
        obj = org_fbs.build(builder)
        builder.Finish(obj)
        data = builder.Output()
        assert len(data) == 192
    
        # create python object from bytes (flatbuffes)
        _org = OrganizationFbs.cast(data)
    
        # assert _org == org_fbs
    
        assert _org.oid == org_fbs.oid
        assert _org.label == org_fbs.label
        assert _org.description == org_fbs.description
        assert _org.tags == org_fbs.tags
        assert _org.name == org_fbs.name
        assert _org.otype == org_fbs.otype
>       assert _org.registered == org_fbs.registered
E       assert datetime.datetime(2022, 3, 27, 3, 51, 33, 288125) == datetime.datetime(2022, 3, 27, 2, 51, 33, 288125)
E        +  where datetime.datetime(2022, 3, 27, 3, 51, 33, 288125) = <cfxdb.user.organization_fbs.OrganizationFbs object at 0x7fda149dbe80>.registered
E        +  and   datetime.datetime(2022, 3, 27, 2, 51, 33, 288125) = <cfxdb.user.organization_fbs.OrganizationFbs object at 0x7fda149db6a0>.registered

cfxdb/tests/user/test_organization.py:69: AssertionError
________________________________________________________ test_org_cbor_roundtrip ________________________________________________________

org_cbor = <cfxdb.user.organization.Organization object at 0x7fda153a80a0>

    def test_org_cbor_roundtrip(org_cbor):
        # serialize to bytes (cbor) from python object
        obj = org_cbor.marshal()
        data = cbor2.dumps(obj)
        assert len(data) == 177
    
        # create python object from bytes (cbor)
        _obj = cbor2.loads(data)
        _org = Organization.parse(_obj)
    
        # assert _org == org_cbor
    
        assert _org.oid == org_cbor.oid
        assert _org.label == org_cbor.label
        assert _org.description == org_cbor.description
        assert _org.tags == org_cbor.tags
        assert _org.name == org_cbor.name
        assert _org.otype == org_cbor.otype
>       assert _org.registered == org_cbor.registered
E       assert datetime.datetime(2022, 3, 27, 3, 51, 33, 295186) == datetime.datetime(2022, 3, 27, 2, 51, 33, 295186)
E        +  where datetime.datetime(2022, 3, 27, 3, 51, 33, 295186) = <cfxdb.user.organization.Organization object at 0x7fda153a8430>.registered
E        +  and   datetime.datetime(2022, 3, 27, 2, 51, 33, 295186) = <cfxdb.user.organization.Organization object at 0x7fda153a80a0>.registered

cfxdb/tests/user/test_organization.py:105: AssertionError
________________________________________________________ test_user_fbs_roundtrip ________________________________________________________

user_fbs = <cfxdb.user.user_fbs.UserFbs object at 0x7fda153cc700>, builder = <flatbuffers.builder.Builder object at 0x7fda15379190>

    def test_user_fbs_roundtrip(user_fbs, builder):
        # serialize to bytes (flatbuffers) from python object
        obj = user_fbs.build(builder)
        builder.Finish(obj)
        data = builder.Output()
    
        # create python object from bytes (flatbuffes)
        _user = UserFbs.cast(data)
    
        # assert _user == user_fbs
    
        assert _user.oid == user_fbs.oid
        assert _user.label == user_fbs.label
        assert _user.description == user_fbs.description
        assert _user.tags == user_fbs.tags
        assert _user.email == user_fbs.email
>       assert _user.registered == user_fbs.registered
E       assert datetime.datetime(2022, 3, 27, 3, 51, 33, 301906) == datetime.datetime(2022, 3, 27, 2, 51, 33, 301906)
E        +  where datetime.datetime(2022, 3, 27, 3, 51, 33, 301906) = <cfxdb.user.user_fbs.UserFbs object at 0x7fda153cc190>.registered
E        +  and   datetime.datetime(2022, 3, 27, 2, 51, 33, 301906) = <cfxdb.user.user_fbs.UserFbs object at 0x7fda153cc700>.registered

cfxdb/tests/user/test_user.py:72: AssertionError
_______________________________________________________ test_user_cbor_roundtrip ________________________________________________________

user_cbor = <cfxdb.user.user.User object at 0x7fda153a8b50>

    def test_user_cbor_roundtrip(user_cbor):
        # serialize to bytes (cbor) from python object
        obj = user_cbor.marshal()
        data = cbor2.dumps(obj)
    
        # create python object from bytes (cbor)
        _obj = cbor2.loads(data)
        _user = User.parse(_obj)
    
        # assert _user == user_cbor
    
        assert _user.oid == user_cbor.oid
        assert _user.label == user_cbor.label
        assert _user.description == user_cbor.description
        assert _user.tags == user_cbor.tags
        assert _user.email == user_cbor.email
>       assert _user.registered == user_cbor.registered
E       assert datetime.datetime(2022, 3, 27, 3, 51, 33, 307615) == datetime.datetime(2022, 3, 27, 2, 51, 33, 307615)
E        +  where datetime.datetime(2022, 3, 27, 3, 51, 33, 307615) = <cfxdb.user.user.User object at 0x7fda153a8940>.registered
E        +  and   datetime.datetime(2022, 3, 27, 2, 51, 33, 307615) = <cfxdb.user.user.User object at 0x7fda153a8b50>.registered

cfxdb/tests/user/test_user.py:92: AssertionError
=========================================================== warnings summary ============================================================
.tox/pytest/lib/python3.9/site-packages/argon2.py:4
  /home/oberstet/scm/crossbario/cfxdb/.tox/pytest/lib/python3.9/site-packages/argon2.py:4: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================== short test summary info ========================================================
FAILED cfxdb/tests/test_perf_serializer.py::test_user_fbs_roundtrip_perf - assert datetime.datetime(2022, 3, 27, 3, 48, 57, 955976) ==...
FAILED cfxdb/tests/test_perf_serializer.py::test_user_cbor_roundtrip_perf - assert datetime.datetime(2022, 3, 27, 3, 48, 58, 138489) =...
FAILED cfxdb/tests/user/test_organization.py::test_org_fbs_roundtrip - assert datetime.datetime(2022, 3, 27, 3, 51, 33, 288125) == dat...
FAILED cfxdb/tests/user/test_organization.py::test_org_cbor_roundtrip - assert datetime.datetime(2022, 3, 27, 3, 51, 33, 295186) == da...
FAILED cfxdb/tests/user/test_user.py::test_user_fbs_roundtrip - assert datetime.datetime(2022, 3, 27, 3, 51, 33, 301906) == datetime.d...
FAILED cfxdb/tests/user/test_user.py::test_user_cbor_roundtrip - assert datetime.datetime(2022, 3, 27, 3, 51, 33, 307615) == datetime....
========================================== 6 failed, 54 passed, 1 warning in 347.20s (0:05:47) ==========================================
ERROR: InvocationError for command /home/oberstet/scm/crossbario/cfxdb/.tox/pytest/bin/pytest -sv cfxdb (exited with code 1)
________________________________________________________________ summary ________________________________________________________________
ERROR:   pytest: commands failed
(cpy39_9) (base) oberstet@intel-nuci7:~/scm/crossbario/cfxdb$

Complete XBR API schema

Complete https://github.com/crossbario/cfxdb/blob/master/work/xbr2.fbs

Add schema tables parse function

we currently lack helpers like

cfxdb/cfxdb/xbrnetwork/account.py

Line 457 in c9717cd

def parse(data):

for other classes. eg:

(cpy382_1) oberstet@intel-nuci7:~/scm/crossbario/xbr-www/cloud/planet_xbr_crossbarfx$ make import_db 
crossbarfx shell import database \
	/tmp/.xbrnetwork \
	/tmp/db.dat \
	--include-schemata xbr,xbrnetwork --exclude-tables xbr.blocks


Database table statistics [dbpath="/tmp/.xbrnetwork"]:

xbrnetwork.accounts....................: 20 records
xbrnetwork.idx_accounts_by_username....: 20 records
xbrnetwork.idx_accounts_by_email.......: 11 records
xbrnetwork.idx_accounts_by_wallet......: 14 records
xbrnetwork.verified_actions............: 63 records
xbrnetwork.user_keys...................: 42 records
xbrnetwork.idx_user_key_by_account.....: 42 records

Importing database [dbpath="/tmp/.xbrnetwork", filename="/tmp/db.dat", filesize=52426]:

Traceback (most recent call last):
  File "/home/oberstet/cpy382_1/bin/crossbarfx", line 11, in <module>
    load_entry_point('crossbarfx', 'console_scripts', 'crossbarfx')()
  File "/home/oberstet/scm/crossbario/crossbarfx/crossbarfx/__init__.py", line 221, in run
    sys.exit(main.run())
  File "/home/oberstet/scm/crossbario/crossbarfx/crossbarfx/shell/main.py", line 1412, in run
    cli()  # pylint: disable=E1120
  File "/home/oberstet/cpy382_1/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/oberstet/cpy382_1/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/oberstet/cpy382_1/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/oberstet/cpy382_1/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/oberstet/cpy382_1/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/oberstet/cpy382_1/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/oberstet/cpy382_1/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/oberstet/scm/crossbario/crossbarfx/crossbarfx/shell/main.py", line 426, in cmd_import_database
    exporter.import_database(filename,
  File "/home/oberstet/scm/crossbario/cfxdb/cfxdb/exporter.py", line 257, in import_database
    val = table.parse(val)
AttributeError: 'TokenApprovals' object has no attribute 'parse'
Makefile:146: recipe for target 'import_db' failed
make: *** [import_db] Error 1

Depend on argon2-cffi, not argon2

currently, cfxdb depends on argon2

cfxdb/requirements.txt

Line 4 in a0be695

argon2>=0.1.10

however, it should depend on argon2-cffi, same as in autobahn[scram]

https://github.com/crossbario/autobahn-python/blob/3da0c268681d599c622ab87d14c604fcdf4df9d4/setup.py#L101

Database schema classes

Every database namespace should have a database class Schema defined:

arealm
mrealm
eventstore
user
log https://github.com/crossbario/cfxdb/blob/master/cfxdb/log/schema.py
meta https://github.com/crossbario/cfxdb/blob/master/cfxdb/meta/schema.py
xbr https://github.com/crossbario/cfxdb/blob/master/cfxdb/xbr/schema.py
xbrnetwork https://github.com/crossbario/cfxdb/blob/master/cfxdb/xbrnetwork/schema.py

For managing a set of CrossbarFX core nodes with proxy and router workers interconnected as a uniform group with respect to application realms and (listening transports, we need a couple of DB tables (for use within master nodes):

here are some notes rgd a potential design

management realms
    |
    +-- management realm
            |
            +-- nodes
                    |
                    +-- (managed) node
            |
            +-- cluster
                    |
                    +-- application realms
                            |
                            +-- application realm
                    |
                    +-- client authenticators
                            |
                            +-- client authenticator
                    |
                    +-- proxy (listening) transports
                            |
                            +-- (listening) transport
                    |
                    +-- router (listening) transports
                            |
                            +-- (listening) transport
                    |
                    +-- (cluster) nodes
                            |
                            +-- (cluster) node:
                                    - number of proxy workers
                                    - number of router workers
                                    - list of application realms
                                    
                                    |
                                    +-- router workers
                                            |
                                            +-- router worker
                            |
                            +-- proxy worker group
                                    |
                                    +-- proxy workers
                                            |
                                            +-- proxy worker

cluster
=======

cluster -> node
    with #proxy/#router workers, and list of application realms

cluster -> application realm

cluster -> client authenticator

cluster -> proxy (listening) transport

cluster -> router (listening) transport

Migrate master tables to flatbuffers

Currently, even though we do have a properly defined schema

https://github.com/crossbario/cfxdb/blob/master/cfxdb/mrealm.fbs

we only have CBR-based zLMDB tables, which follow those schemata by convention, but use CBOR as value-type

We should migrate those tables to proper Flatbuffers value-typed tables. This should be (almost) without any impact on the existing code in cfx master! at least it is a good test of this promise;) as in, "how it should work as designed" ..

Refactor tests to individual file

From slack

oberstet  2:18 AM
^ splitting up the test files, eg https://github.com/crossbario/cfxdb/blob/master/cfxdb/tests/test_xbr.py into 1 file per tested table (that is, currently 2 tests per file: 1x roundtrip, 1x performance): yes, agreed!
those test have classically grown from 1 to more .. without any refactoring. but I totally agree now: those files are simply too big .. confusing. 1 table == 1 test file (with consistent naming test_<namespace>_<table>) solves this

Add UserRole.SUPERUSER

cfxdb/cfxdb/user.fbs

Line 55 in ce72820

/// CFC user roles (built into code, unconfigurable)

Document standard DB types

Document the "standard" DB types and their corresponding Python types we are using in XBR (and generally in CFXDB):

Python <=> Flatbuffers
np.datetime64 <=> uint64 (timestamp)
uuid.UUID <=> [uint8] (uuid)
...

Refactor DB classes into modules

all DB classes for a flatbuffers schema / namespace, like eg

https://github.com/crossbario/cfxdb/blob/master/cfxdb/xbr.fbs

should reside in a separate module of corresponding name

https://github.com/crossbario/cfxdb/tree/master/cfxdb/xbr

Split up XBR DB

currently, every market maker running in a xbrmm worker runs its own embedded database with all tables (see log at the end).

the xbr schema should be split up into 2 Flatbuffer namespace and correspondig DB schemata:

xbr db: xbr
xbr market maker db: xbrmm

this includes the following list of tables:

so in other words, all tables from the following should be moved toa new schema xbrmm:

cfxdb.xbr.transaction
cfxdb.xbr.offer
cfxdb.xbr.channel
cfxdb.xbr.consent

2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 001 for database table <9f87bad9-695e-439d-86ad-9e23695d3f67>: cfxdb.xbr.api.Apis
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 002 for database table <a4a0553e-24fa-4280-9959-5805f034d861>: cfxdb.xbr.block.Blocks
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 003 for database table <60ba3189-d127-4522-bfbc-ed416bf7233c>: cfxdb.xbr.catalog.Catalogs
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 004 for database table <f3d1f052-7160-48a7-b36a-221035fc9373>: cfxdb.xbr.token.TokenApprovals
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 005 for database table <7dcfffd2-4859-45d6-b9cd-ba7930a7a107>: cfxdb.xbr.token.TokenTransfers
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 006 for database table <d1808139-5a3b-4a4e-abad-152dd4cd1131>: cfxdb.xbr.member.Members
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 007 for database table <861b0942-0c3f-4d41-bc35-d8c86af0b2c9>: cfxdb.xbr.market.Markets
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 008 for database table <7c3d67b4-35a3-449f-85a6-2695636fc63e>: cfxdb.xbr.market.IndexMarketsByOwner
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 009 for database table <1863eb64-322a-42dd-9fce-a59c99d5b40e>: cfxdb.xbr.actor.Actors
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 010 for database table <4f50a97a-4531-4eab-a91b-45cc42b3dd21>: cfxdb.xbr.market.IndexMarketsByActor
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 011 for database table <b3d01946-85ae-49f3-ad96-b78194eb82fe>: cfxdb.xbr.channel.PaymentChannels
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 012 for database table <cffd5253-72f8-41a9-8b76-5e6ff3654e67>: cfxdb.xbr.channel.IndexPaymentChannelByDelegate
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 013 for database table <878ac002-a830-488b-bfe9-f06371b8eecb>: cfxdb.xbr.channel.PaymentChannelBalances
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 014 for database table <4e7e7c8d-db0d-4dea-8409-ac8f21ce1e10>: cfxdb.xbr.channel.PayingChannels
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 015 for database table <cee954be-fdb2-43cc-8891-529d6c7a0c3b>: cfxdb.xbr.channel.IndexPayingChannelByDelegate
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 016 for database table <c0931d5d-6d5d-4f9c-b2a3-29664a0f4c07>: cfxdb.xbr.channel.PayingChannelBalances
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 017 for database table <dc6d175b-3dd0-4b1f-a6e8-2aec7f0e3fe5>: cfxdb.xbr.offer.Offers
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 018 for database table <ef5f1cdc-4871-4a03-ac1c-c60e80875b8b>: cfxdb.xbr.offer.IndexOfferByKey
2020-03-26T14:51:35+0100 [Marketplace  24205] Allocated new slot 019 for database table <87cf8eac-10d8-470d-a645-23e11343e065>: cfxdb.xbr.transaction.Transactions
2020-03-26T14:51:35+0100 [Marketplace  24205] Attached XBR Market Maker database [dbpath=".xbr1-mkr1-db", maxsize=1073741824]
2020-03-26T14:51:35+0100 [Marketplace  24205] MarketMaker.start() ..
2020-03-26T14:51:35+0100 [Marketplace  24205] Ok, XBR market maker is associated on-chain and will be working for market=8a2853d1-e210-40c5-83a8-27d13b0aeaf3!

Expand master DB schema

WebCluster: group of 1 or more nodes (running proxy workers) - see here
RouterCluster: group of 1 or more nodes (running data planes)
DataPlane: group of 1 or more router workers
ApplicationRealm: are application routing realms created in data planes (running in router clusters with router workers kept in sync) - see here

autoapi linking broken for RTD

https://cfxdb.readthedocs.io/en/latest/management-realm/index.html

it works for https://zlmdb.readthedocs.io/en/latest/reference.html

close_balance in a channel should return uint256

Currently the close_balance https://github.com/crossbario/cfxdb/blob/master/cfxdb/xbrmm/channel.py#L512 tries to return an integer value. msgpack is not able to serialize that value. It wouldn't be an issue if we were using CBOR everywhere, but due to crossbario/autobahn-js#466 we are currently using MsgPackSerializer in Planet and Markets UI.

>>> import msgpack
>>> msgpack.dumps(100000000000000000000)  # 100XBR
Traceback (most recent call last):
  File "msgpack/_packer.pyx", line 171, in msgpack._cmsgpack.Packer._pack
OverflowError: Python int too large to convert to C unsigned long

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/om26er/scm/crossbario/xbr-www/venv/lib/python3.8/site-packages/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 182, in msgpack._cmsgpack.Packer._pack
OverflowError: Integer value out of range

The other option would be: I fix CBOR in ABJS once and for all ;-)

Travis CI reports “Expected - Waiting for status to be reported”

https://travis-ci.community/t/known-issue-travis-ci-reports-expected-waiting-for-status-to-be-reported-on-the-github-status-api-but-the-status-never-arrives/1154

Database classes: cover roundtrip fbs+cbor and copy fbs+cbor

actually, to fully cover the complete interface the database table classes expose, the 2 test functions we currently have per database table are not enough, but we additionally need

roundtrip cbor
roundtrip fbs
copy cbor to fbs
copy fbs to cbor

like for example here https://github.com/crossbario/cfxdb/blob/master/cfxdb/tests/test_user.py

Add slots to DB table classes

The DB table classes currently lack slots (as in eg https://github.com/crossbario/autobahn-python/blob/5b3f28549f48fc0f3df54512dd6a2eb1826fbcd2/autobahn/wamp/message.py#L540).

We should use slots - part of the reasons is that it avoids accidently setting a non-existing attribute

crossbario / cfxdb Goto Github PK

cfxdb's People

Contributors

Watchers

Forkers

cfxdb's Issues

Recommend Projects

Recommend Topics

Recommend Org