ambiretech / adex-validator Goto Github PK

what happens if you receive the first ApproveState but don't know about the channel yet? it will fail, and not receive the message; perhaps go back to Init or just do a Heartbeat

we should be able to handle a couple of thousand requests per second

Implement validator fees that will be calculated based on # of processed events.

For the publisher side, this adds more incentive to forge events, but the same security guarantees apply (they're not the state leader)

For the advertiser side, incentives are OK cause they lose out more from forging events (impression events pay to the publisher).

So if the payment spec defines fees per IMPRESSION event, pay them out - that'd be pretty simple.

Implement reading the campaignSpec (IPFS?) and then respecting the minAvgPerImpression

because of the way the isValidTransition function works, you can transition to states like {a: 20, b: -2}; cause we assume if a key is not there before, and appeared, it will be positive

there may be more bugs related to negative BNs like this

perhaps quit if there's a recent Heartbeat (and no Exit? this would be a new special event type to signal graceful stopping)

rather than using the sum(approved), we use sum(min(ours[k], approved[k]) for k in intersection(keys(ours), keys(approved)))

this basically means "take the revenues we both agreed on, and ensure their sum is within 95% of my accounting" (assuming 950 promilles as health threshold)

Periodically sign a Heartbeat event and send it to the other validators; this event might contain some data to distinguish the time, e.g. timestamp or sequence

this would solve #13, help with #6 and help the market determine the health of the channel

more accurately, it allows the market to distinguish between more detailed off-chain state: a waiting state where we are waiting for all validators to post a heartbeat, and then offline (a type of unhealthy state) which happens if some validator hasn't sent a heartbeat recently;

once this is done, tests must filter validator messages by type (otherwise the last messages will be full of Heartbeat)

see also: http://github.com/adexnetwork/adex-market/issues/1 and https://github.com/AdExNetwork/adex-protocol/blob/master/components/market.md

extension of #43

this would add an ability to query specific time periods via the API,

and furthermore merge many aggregate objects into aggregates for certain time units: e.g. "week", "day", "hour", "minute"

currently the AGGR_THROTTLE is set to 5 seconds, meaning that a "minute" unit would mean merging 120 aggregates into one

this should be done at a mongo level, otherwise it would be way too heavy

In relation to the external tests:
https://github.com/adexnetwork/adex-validator-stack-tests

This includes:

• adapt the third party tests to work with the latest iteration of the validator stack (in integration-tests branch); the biggest thing here would be the change to the validator-messages API (.validatorMessages instead of .messages)
• add missing test cases: submitting validator messages without authentication
• add npm run test-integration-external, which will use (parts of) our bash script to set a new mongodb db name(s) and run the 4 components (follower sentry/stack, leader sentry/stack) and run adex-validator-stack-tests; adex-validator-stack-tests can be included as an npm dependency to this repo
• add npm run test-integration-external to .travis.yml

we should test with various minPerImpression values in the spec

we should do this after we refactor the tests into multiple files

Create an integration test suite for the AdEx validator stack

Protocol spec: https://github.com/AdExNetwork/adex-protocol

Validator stack implementation: https://github.com/AdExNetwork/adex-validator-stack-js

OUTPACE: https://github.com/AdExNetwork/adex-protocol/blob/master/OUTPACE.md

a bunch of integration tests need to be written, including a group that tests inter-validator relationships and possible attacks: e.g. leader signs an invalid state, and followers should detect and not sign

other ones:

• follower should mark unhealthy, but not if the difference is too little
• try denial of service attack vector

Testing the regular code paths should work in the following way:

• create two environments (dockerized?) where each connects to it's own MongoDB database (could be connected to the same server, but at a different DB)
• both DBs will be seeded: example in test/prep-db
• each environment will run a validator stack: a sentry and a validator worker, initially with the dummy adapter
• in the test data (in the DB), there should be a channel that appoints the first validator worker as a leader, and the other as a follower
• start sending events to both sentries (example in test/send-events) and see how the system reacts
• run integration tests for the sentry methods (e.g. getting info for a channel, detailed info for a channel, etc)

later, we'd like to have larger coverage, including the ethereum adapter and watcher

furthermore, integration tests of the adapters, individually, are important as well

Important note regarding Gitcoin: there is a gitcoin bounty for this issue: https://gitcoin.co/issue/AdExNetwork/adex-validator-stack-js/1/1996

This test suite is supposed to test one of the most critical components in AdEx (the validator stack), so we believe in creating our own in-house implementation as well as posting a bounty for a third-party implementation. Ideally, we will end up with two independent test suites for this critical component.

the protocol defines what's being signed as a keccak256(channelId, balanceRoot) but currenly we only sign balanceRoot

see https://github.com/AdExNetwork/adex-protocol/blob/master/OUTPACE.md#specification
and https://github.com/AdExNetwork/adex-protocol-eth/blob/master/contracts/AdExCore.sol#L73

in the follower tick, we should evaluate the signature of NewState for extra security

(even though we already check authentication on POST /validator-messages)

This actually goes deeper: you can enter arbitrary messages of any kind (NewState too) if you compromise the auth, which implies more attack options; for example, submitting an ApproveState{ health: 'HEALTHY' ...} from the leader to the follower, to trick the market into thinking the channel is healthy

the follower tick is checking if the balance tree in the newState msg represents a proper state transition, but it's not checking if the stateRoot is the correct hash for that tree

two solutions

1. check the stateRoot against our own computed stateRoot via adapter.getSignableStateRoot(channel, tree.getRoot())
2. compute that stateRoot at the end when signing; that way, even if the leader tricks us, we won't sign their stateRoot

But before doing anything, we should first write an integration test that proves this

the version of the node should be considered in the design of the #44, cause newer versions might not be compatible with older versions of the node

it's also possible to upgrade without affecting compatibility with older nodes

a good way to do this would be

• major version difference means hard update
• minor version differens means soft update

Hard updates cannot be performed on nodes in between campaigns. If you upgrade your node to a new major version, all campaigns you participate in from the previous major version will consider your node offline

what we need to do is

• document this behavior
• include major version in validator messages, require the major version of everything we receive to be the same as ours
• include this in the #44
• include major version in the campaignSpec: https://github.com/AdExNetwork/adex-protocol/blob/master/campaignSpec.md (perhaps in the wrapper)

create an API to get a filtered set of event aggregates - that'd be nice for analytics

we have to follow the principles set in https://github.com/adexnetwork/adex-protocol#privacy-of-publishersadvertisers

current candidates are:

• PostgreSQL
• CockroachDB: https://www.cockroachlabs.com/
• Noria: https://github.com/mit-pdos/noria

we first need to go through a release cycle with release-1 to see what the load/requirements are and if any additional requirements will arise

also this will be useful for updating the event aggregates from multiple threads: https://github.com/jonhoo/rust-evmap

another important factor for this is the existance of a mature Rust driver with async IO

The get status api ( /channel/:id/status ) throws an error if the channelStateTrees returns null.
https://github.com/AdExNetwork/adex-validator-stack-js/blob/master/routes/channel.js#L36
There should be a check to see if the returned result is null

the style right now is not very consistent, and it should be

the CI needs to run:

• npm test
• npm run test-integration
• npm run test-integration-external (see #41)

and multiple if possible

e.g. get the last ApproveState and the last NewState in one call

Write tests for:

• services/sentry/eventAggregator
• services/sentry/eventAggregator/lib/eventReducer
• services/validatorWorker/producer: this one contains pure functions that can be moved into a lib/reducer as the previous one

e.g.

• the wrong sig test should ensure the validator detects this as wrong sig
• same for deceptive stateRoot and an invalid state transition

in order to do that, consider a new, non-propagated validatorMessage, specifically for this

related to #65

in the best case we can do everything through an NGINX config

initial limit ideas:

• event submissions per (channel, publisher) per IP
• event subimssions per (channel, publisher) per solved memory-bound PoW (Cuckoo Cycle?)

do this for the first release instead of #4

implement a docker stack for everything - sentry, validator, watcher

after reaching the deposit limit, mark the channel as exhausted and don't accept any more events for it

for example, check if balances is valid on submission of a NewState

furthermore, if we have the #31 and #23, we can get the last ApproveState that we generated, and see if the NewState is a valid state transition right away, before even going to the validator worker (which will ease unit testing)

we should accept events only if the channel is in an active state

advertisers should be able to dynamically change the price per impression; this could be done by having them submit a validator message to both validators, which will update their eventRecorders to address the change

the original payment spec should contain a thresholds: see pricingBounds here

this would require changing eventAggregator, eventReducer and the channel routes so that existing recorders can be updated dynamically

for details on the schema, see AIP#6

@samparsky start by going off the comment in the AIP that outlines the spec and PR-ing it to campaignSpec.md in adex-protocol

then, implement the date structures in Rust primitives and open a PR so we can review

Then proceed to implementing in the JS and Rust implementations, with comprehensive tests - unit tests for rule matching and integration test with a few compound rules matched together

The issue is as follows:

Let's say that:

• A is the last ApproveState message by a follower node N, approving some state S(n)
• B is >2/3 ApproveState messages for some state S(m)
• C is a NewState message for some state S(m+1) generated by validator Q
• N has fallen behind, and it's m is significantly higher than n: m>n
• now, sending a NewState message C to node N, will make it compare the transition from S(n) to S(m+1); the correct thing to do is compare S(m) to S(m+1); that way, we allow Q to trick N into signing a malicious state transition where the balance has been reduced compared to S(m) but is still higher compared to S(n), making it a valid state transition from the point of view of N

in other words:

// we compare only OUR last approved state to the NewState
// instead, we must compare from the latest state approved by any >=2/3 validators
// otherwise, the leader may perform an invalid state transition from latest approved to a NewState, and trick us into signing it cause we'd be comparing with our own (presumably old)

rather than storing eventType=>publisher=>count, store the earnings instead, and calculate the per impression earnings at a sentry level

this has the following benefits

• more precise control over per-impression earnings
• easier aggregation for the worker
• ability to do custom events easily by aggregating them all into CUSTOM, rather than having to store each one individually; this is super important for making things future-proof

drawbacks:

• losing critical data like count of impressions; we can mitigate that by keeping separate eventCounts and eventPayouts
• calculating validator fees becomes more complicated; also mathematically ensuring all funds will be distributed at the end, but not exceeded; this can be mitigated by just incrementing eventPayouts with a full value (e.g. 150 per event), and then subtracting the validator fees and clamping the total when the tree is built; this meens having to store two full trees: beforeFees and afterFees; but we might need to do that anyway to make the math work properly; fortunately, this only applies to channelStateTrees: everywhere else (NewState, health comparisons) we just use the final tree

this one is for testing, very similar to what we have in adapter/ as of today

this would be a special event that would trigger a state mutation that pays out the remainder of the funds back to the the channel creator (channel.creator)

the event should only be submittable by the channel creator themselves (so authRequired + checking if uid == channel.creator)

also instead of calculating the remainer, you can just increment the balance of channel.creator with the full channel.depositAmount for the given event aggregate; the mergeAggrs function in the valiator will ensure that only the remainder is added and there are no overflows

this is high priority: make authentication, channel watching, etc. work via the ethereum adapter (and watcher)

like the ethereum adapter, but for the substrate parachain

in OUTPACE, the follower signing any valid new state is OK cause the leader is distributing their own money - so a scenario where they steal their own money is a no-op

however, in practice, if an advertiser delegates another validator as an advertiser platform (that validator will be the leader) they can just assign all of the funds to themselves and the follower will sign

one solution is to use the getHealth math and have thresholds - less than 95% means unhealthy, but less than 90% means "don't sign new states"; at first glance, it seems this may cause a liveness failure in case one validator is temporarily offline, but then again, this is also an issue with the health in general (in light of this, see #48)

this applies to fees as well - the leader can cheat the follower out of a fee by using the same technique

a sentry subset that only does:

• /channel/list
• POST /channel/:id/events

the validator stack is architected in such a way that allows easy implementation of an event-only sentry even in ecosystems that don't have good DB drivers/asynchronous IO; this is because we only save 1 eventAggregate every few seconds, and everything else is kept in memory

the purpose of this task would be to see how far we can go performance wise

see #2

will be solved via #17

• getLatestMsg - SHOULD RELY ON PROPER TIMESTAMP/SEQUENCE; maybe a primitive DB abstraction?
• #61
• those tests should be more robust - and rely on some feedback messages from the validator stack #61 - perhaps InvalidNewState message
• split tests in multiple files
• decrease repetitiveness in tests (DRY)
• try to decrease waitTime/propagation logs
• tests: go through each one, try to break them by touching the code

there should be a mode in which no new channels are accepted (the watcher stops entering channels into the DB) and once all are exhausted, we quit

allow anyone who runs the validator stack to configure the policy of what channels they accept

• requirement: they should have 2 validators (cause of #4)
• optional: what validator fees are paid out
• optional: creator whitelist: only accept channels created by certain addresses
• optional: validator whitelist (if you want to allow only certain addresses to be the second validator)
• optional: reward token whitelist

this will help in many ways:

• easy way to get "the latest event of a type"; currently this is used in the follower worker, but it would be useful for #23 too
• allow us to migrate to Postres in v2.0, where we can have static, simple and relational data types (even trees can be stored in a table)
• allow us to abstract away watching for new event aggregates/validator messages, therefore removing wait/snooze periods; under the hood it can be implemented with redis pubsub or via postgres

since the ethereum channels can't be entirely collected from the blockchain, we need to be able to submit info about the channel to the sentry

we also need to be able to negotiate whether the validator will take the channel beforehand: for example, the validator should have a configurable threshold of funds/whitelisted tokens (see #30)

in practice, this should probably work like this:

• ask the validator whether they'd accept the channel; you'd send all info about the channel except the validators
• submit the channel, with full info; at this stage, we will know the channel hash

if the second operation fails for some reason, we repeat the whole thing

after the second operation, we should create some sort of a temporary record that will not be considered by the ticks and routes (https://github.com/AdExNetwork/adex-validator-stack-js/blob/master/bin/validatorWorker.js#L32 and https://github.com/AdExNetwork/adex-validator-stack-js/blob/master/routes/channel.js#L55) until the watcher approves it; otherwise, this record should also auto-expire;
the watcher approval should happen when it sees the channel on the blockchain, possibly after having some # of blocks after as a safety threshold against block reorgs

watch the network and add channels to the DB

also ensure the data is absolutely validated and as static as possible before inserting into the DB

Challenge description:

• Create CLI tools to manage AdEx channels
• Open channel: create it on-chain and submit it to the validators
• close channel by submitting the CHANNEL_CLOSE event
• withdraw from channel
• withdraw from channel after it's expired (return remaining funds to creator)
• to understand how https://github.com/adexnetwork/adex-protocol-eth and https://github.com/AdExNetwork/adex-validator-stack-js interact, read https://github.com/adexnetwork/adex-protocol

Submission requirements:

• Usable CLI for adex-protocol-eth + adex-validator-stack-js

Submission deadline:

• April 30

Judging criteria:

• Best submission
• Usability of the CLI; the more user-friendly, the better
• Correctness and well structured source code will be appreciated
• contributions to other open-source packages will be appreciated

Judging date:

• April 30

Bounty:

• 1000 DAI

since the follower tick is triggered by the shared producer, and that is triggered when there are new eventAggrs, this means that we end up with a situation where we've received new NewState messages to approve, but our tick is never triggered

this is a fundamental timing issue since the handling of eventAggrs will always be sooner than receival of NewState msg