brave-intl / bat-ledger Goto Github PK

1. Deprecate these calls in the ledger:

   GET /v2/publisher/identity
   GET /v2/publisher/identity/verified

2. Move these calls from the ledger to the eyeshade server:

   POST /v2/publisher/ruleset
   PATCH /v2/publisher/rulesets
   PUT /v2/publisher/ruleset/{publisher}
   DELETE /v2/publisher/ruleset/{publisher}
   GET /v2/reports/publisher/rulesets

3. Add messages from eyeshade to ledger that provide the functionality in the five calls above.

To simplify Eyeshade and dedupe data, we decided to move the Site channel verification flow so it's entirely within Publishers (https://github.com/brave-intl/creators-private-issues/issues/940). The Publishers task and this one need to completed together.

As a result, Eyeshade would no longer have verification tokens or unverified channels.
(TODO: Check if unverified channels exist in Eyeshade).

currently we have a number of ledger clients stuck using old surveyors instead of the current one. it seems most likely that this is caused by brave-intl/bat-client#35 in combination with the fact that some clients reconcile date was not pushed back on grant claim and grants were made unredeemable before their maturity date.

we will fix the client side - to go out in a browser-laptop release but as a quick fix we should unstick clients by upgrading old surveyors servers side on demand.

currently we expose the list of currencies that the user has a card for, we should add possible currencies such that the response includes both lists

To implement https://github.com/brave-intl/creators-private-issues/issues/960 publishers needs to know the total amount an owner earned during the last settlement.

It appears as though a settlement document only has the information for a single channel, and we are only returning one settlement per owner in this request.

cc @evq

Many publishers have requested that their accounts be deleted. We do not have an easy way to do this. Right now we must manually delete the records from both eyeshade and publishers. This is error prone and not scalable.

We should create some endpoint on eyeshade that publishers can hit that deletes all records associated with an owner (owner, their channels, token, etc). Something like this would work:

DELETE /v2/owners/{owner}

Related to https://github.com/brave-intl/creators-private-issues/issues/969

will require the cards:write scope

In the Brave browser, the payments tab has publisher verified green checkmarks. This green checkmark is queried from Ledger.

Instead of Ledger fetching this information from Eyeshade, we're going to fetch the verified_status for an owner from the publishers api.

We'll be disabling the data propagation in a separate task.

during the test for settling a grant, the tester should not have to manage the db before running the tests. manual management can cause the db to be in poor state and for tests to fail. instead, automate by making sure that during a test, a single grant is added to the db, it is used, and subsequently flushed from the redis db by the end of the test.

• eyeshade-server @mrose17
• ledger-server @evq

we should retire github auth in eyeshade and other services where possible, we're moving toward a publishers based admin panel and our github auth has known issues, e.g. it does not work across dynos:

if the first hit to v1/login (redirects the user to github) and the second (user is redirected back from github) happen on different dynos, the login does not succeed

apps we could remove it for currently:

• balance
• helper
• extractor

@evq @mrose17 we should revisit and come up with a game plan regarding when this should happen to reduce exposure

To avoid unexpected interactions and allow use of fixtures, tests should clean some or all database data during setup or teardown.

cf., http://swaggerstats.io/apidoc.html#operation/%2Fswagger-stats%2Fmetrics

use -- to pass flags to the following scripts
lint
docker-up

The root cause appears to be fixed in bitgojs
BitGo/BitGoJS@6b2296f
we should update when a new release is available

while it is implicit currently, the promotion creation under the npm script create-promotion should be a test that can be isolated and integrated with the rest of the testing flow

e.g., DASH, BCH, BTG.

this requires adding the appropriate network address. we probably want to rework the algorithm so that an address generation request is sent to uphold only if the user selects a particular alt-currency (at present, a wallet creation gets addresses for BTC, ETH, and LTC).

there is an invalid error check that occurs on ledger/controllers/registrar.js:164

currently only non-website channels are verified (all are sent by publishers), but with brave-intl/creators-private-issues#940 we will need this change to keep things in sync until we can remove eyeshade's copy of the channel/owner relationships

there are a variety of improvements that could be made to make the code more legible and testable

legible

• using imports instead of requires and filename patterns
• breaking apart monolithic files
• destructuring
• using default values

testable

• breaking apart monolithic files
• more (and exporting existing) pure functions (where available)

these items can be addressed in any order, as well as broken up by server / worker

for batching voting surveyor

cc: @evq @davidtemkin

blacklisting publishers currently happens manually on voting document basis
mark publisher's excluded boolean if they are registered in our blacklist
if excluded is true, do not include in payout information
take the publisher, check the blacklist, flip the bool if

create mongodb indeces for blacklist and query them
  eyeshade/workers/wallet.js:voting-report
    mark excluded
  eyeshade/workers/reports.js:report-publishers-contributions
    sanity check to make sure no publishers in report have

@evq

might be hard to pull off due to latencies, but I think there's a potential side channel attack
@diracdeltas fyi

setup scripts have been abstracted to tests. remove setup folder, update readme

see https://github.com/brave-intl/bat-ledger/blob/master/ledger/controllers/wallet.js#L244, we use the client provided surveyor. depending on client behavior perhaps we can reject payments where the surveyorId does not match that of the current surveyor

when starting up the extractor worker, the relation download errs in

npm run start-extractor-worker

saying that it does not exist.

when hitting /v1/rates immediately after the server starts up, a subset of rate conversions are sent back, instead of the full 20, which is denoted in the helper tests.

To reproduce, shut everything down, start up the servers, and run the helper integration test and see it fail the first time because only 8 values are sent back. Sometimes 9 are sent back, depending on if other tests are running in parallel.

Publishers are reporting that their statements are not generating.

See: https://twitter.com/robek_world/status/981563504686784517

and have bat-ledger and bat-eyeshade talk to it instead of the exchange rate services directly

isolate why this is happening and allow tests to turn off / reduce time during testing

during the script

npm run test-integration

or, if using the docker container

docker-compose run --rm -v $(pwd)/test:/usr/src/app/test ledger-web npm run test-integration

the following error is received. if the endpoint is hit too often

status: 429
http://ledger-web:3001/v1/grants/d6a9c116-5779-4442-a0dc-6237e295ae29
{ statusCode: 429,
  error: 'Too Many Requests',
  message: 'try again in 86400 seconds' }

bring up all relevant interoperating services
ledger services, grant server, referral server and publishers

the /v*/publisher/* endpoints are read-only for the browser, so it's a good thing to move over.

we were sending unix time (as is incorporated into grants), but the browser assumes javascript style ms since epoch. changed in the server promotion payload since that doesn't require a new release.

cf. brave/browser-laptop#13354

If you've registered potato.com and become a Brave publisher; and potato.com expires and Cate Camper registers it right away, contributions to potato.com should no longer flow to you.

eyeshade, as the canonical registration checker, should be the service which checks periodically.
it also needs communicate this info with publishers: https://github.com/brave-intl/creators-private-issues/issues/1211