shapeshift / unchained Goto Github PK
View Code? Open in Web Editor NEWUnchained is a multi-blockchain backend interface
License: MIT License
Unchained is a multi-blockchain backend interface
License: MIT License
Add thorchain client to unchained using swagger definition if we can find one
or use one from npm if cant find a good swagger
or write custom client class is none of those exist
We need to deploy a Bitcoin Ingester to the open-source/public production cluster. Consult with @kaladinlight prior to starting ticket.
CI flow should support multiple deployments as a precursor to implementing a blue/green deployment strategy.
Scope the work needed and document an implementation plan in the comments on this issue
Identify challenges or other considerations that are unique to this chain
Task breakdown & create Github issues to track the work
Estimate how long it will take to complete
Thorchain specific:
Daemon
Indexer
The Indexer maintains an index of transaction history by address.
API
The API is an adapter layer that provides a consistent set of endpoints for the ShapeShift client. Implement these endpoints using Bitcoin & Ethereum as a reference
/account/{pubkey}
/account/{pubkey}/txs
(deferred for Thorchain, get tx history from WebSocket during account sync)/send
Ingester
The Ingester filters new transactions and sends only the relevant ones to each client
**Tests **
Ingester:
Currently the only notion of network is in the pulumi config under the daemon
resource. This needs to be a more overarching idea for a coinstack that can be used in the pulumi iac as well as reference and returned from the app as an env var.
AC:
getNetwork()
endpoint to the BaseAPI
and support in current coinstacksChange the favicon and page title so the browser tab shows as a ShapeShift page, rather than "Swagger UI".
See screenshot and arrow for more clarity about what needs to change.
Ideally the favicon and css would live in the coinstacks/common/api so that it can be referenced by each coinstack moving forward. We are using https://github.com/scottie1984/swagger-ui-express to generate the swagger doc ui which allows for custom styling options to be passed in.
Ideally the favicon and css would live in the coinstacks/common/api so that it can be referenced by each coinstack moving forward. We are using https://github.com/scottie1984/swagger-ui-express to generate the swagger doc ui which allows for custom styling options to be passed in.
Each exchange is just a string and the returned message is just a blob of data. It would be nice to wrap the exchange with the messages to be able to type the interaction with rabbit during the ingestion pipeline.
AC:
Acceptance Criteria:
AC:
AC:
This needs to be discussed and scoped out by @kaladinlight before work starts.
Please update the ticket with acceptance criteria when it's known.
We need a getTransaction
endpoint to grab transaction data, and whatever else that is needed by chain-adapters to send Bitcoin transactions.
Acceptance criteria:
Correctly extract addresses from special txs such as:
From address is an optional argument that can be sent to the estimateGas
call and may have implications on the estimated gas cost returned.
AC:
from
query paramter for estimateGas
endpointAC:
AC:
.circleci/config.yml
to deploy develop and production public coinstacksSteps:
indexer.daemon.chain
(specified in your copy of coinstacks/ethereum/pulumi/Pulumi.sample.yaml
) parameter works correctly to deploy a local kubernetes instance of an ethereum ropsten coinstack
asset
name in coinstack/ethereum/pulumi/index.ts
to append the chain type if it is not mainnet
.coinstacks/ethereum/daemon/init.sh
(should work the same as mainnet node)coinstacks/ethereum/daemon/readiness.sh
(should work the same as mainnet node)daemon-chain
paramterGet new blocks from the node and put them in the txid queue
Examine traffic levels since the launch of the public api.ethereum.shapeshift.com endpoint, determine if it would be prudent to implement ingress throttling, and implement if so.
Instead of lots of flat endpoints like /balance
, /utxos
, etc., use a more resource driven restful design pattern like:
/account/{pubkey}
/account/{pubkey}/txs
/account/{pubkey}/utxos
/gas/estimate
/gas/price
Does what it says on the box.
Please involve community members and have them shadow you when completing this ticket, such that they are able to add more arbitrary chains in future.
The BTC chain adapter currently fetches network fee estimations from bitcoinfees.earn.com. This is effective, but fragile in the case of unforeseen external service interruptions. Unchained should provide BTC fee estimates via API call, with the network fees determined independently, without the use of any external service. For reference, look at how BTC fees are estimated in Watchtower.
Currently all errors are being treated as axios errors, but this is an incorrect assumption and we should ultimately use proper type checking on errors (treating err as unknown) so we handle all error types correctly
AC:
How to add a new chain. Note - this template is only a guide. Each chain is different so make changes to the plan as neccesary
Daemon
Indexer
The Indexer maintains an index of transaction history by address. Most nodes don't do this so a separate database is needed.
API
The API is an adapter layer to the Indexer that provides a consistent set of endpoints for the ShapeShift client. Implement these endpoints using Bitcoin & Ethereum as a reference
/account{pubkey}
/account/{pubkey}/txs
/account/{pubkey}/utxos
/send
Ingester
The Ingester filters new transactions and sends only the relevant ones to each client
Tests - Ingester
There is currently only a single rabbitmq sts deployed for each coinstack which will result in down time whenever kube nodes roll. It would be best if we could update to use a rabbitmq cluster or replica set deployment, similar to mongodb
https://github.com/pusher/k8s-spot-termination-handler
Currently, spot inst termination signals are not caught and handled resulting in corrupted dbs during sync if nodes churn.
AC:
Acceptance Criteria:
The API is an adapter layer that provides a consistent set of endpoints for the ShapeShift client. Implement these endpoints using Bitcoin & Ethereum as a reference
/account/{pubkey}
/send
Instead of manually managing snapshotting and restore, we want this to be an automated process.
AC:
Use the following checklist as a guide:
Daemon
Indexer
The Indexer maintains an index of transaction history by address. Most nodes don't do this so a separate database is needed.
API
The API is an adapter layer that provides a consistent set of endpoints for the ShapeShift client. Implement these endpoints using Bitcoin & Ethereum as a reference
/account{pubkey}
/account/{pubkey}/txs
/account/{pubkey}/utxos
/send
Ingester
The Ingester filters new transactions and sends only the relevant ones to each client
**Tests **
Ingester:
We need a Bitcoin node in production. Use Pulumi to deploy a stack to the open-source/public production cluster.
AC:
We need a Blockbook instance in production. Use Pulumi to deploy to the open-source/public production cluster.
We don't have the ability to view logs of a pod that is no longer running. We need a way to collect and store a rolling window of logs for analysis.
AC:
Veryify working coinstack locally (init.sh, readiness.sh)
Deploy to personal aws account
Update .circleci/config.yml to deploy develop and production public coinstacks
Steps:
Verify indexer.daemon.chain (specified in your copy of coinstacks/ethereum/pulumi/Pulumi.sample.yaml) parameter works correctly to deploy a local kubernetes instance of an ethereum ropsten coinstack
There will be pulumi resource naming conflicts that need to be handled. It would make the most sense to update the asset name in coinstack/ethereum/pulumi/index.ts to append the chain type if it is not mainnet.
Ensure signal catching works correctly still as defined in coinstacks/ethereum/daemon/init.sh (should work the same as mainnet node)
Ensure readiness check works as defined in coinstacks/ethereum/daemon/readiness.sh (should work the same as mainnet node)
Deploy and sync ethereum ropsten coinstack to personal aws account
Update .circleci/config.yml with jobs to deploy ethereum ropsten coinstacks to the develop and production environments, specifying the correct daemon-chain paramter
Acceptance Criteria:
The ingester_meta object is not being sanitized or passed through completely on registration which results in always syncing from block 0.
AC:
Follow the Ethereum spec and deploy an API to production. Add tests with chain-adapters.
Both the Socket and Worker classes were abstracted into the common ingester package but left process.env artifacts.
These were recently moved into the constructor to allow publishing of the package without failing on import without env vars set as a quick fix.
This should really be updated to pass in the values in the constructor to be configurable more correctly.
AC:
Unchained thorchain ingester
We don't need to be checking in all of the generated code, but we do want to ensure the most recent generated code in a few circumstances.
coinstacks/api/src/routes.ts
does not need to be tracked. this is automatically generated on yarn build
and kept up to date during development. We run yarn build
and persist the workspace in CircleCI, so we will have the routes available for deployed image.coinstacks/api/src/swagger.json
should be tracked. There is currently a pre-push
hook that runs generate, but it is easy to miss the update and merge without actually committing the most recent swagger. It would be ideal to pull this out so that on merge, we run generate on the project and commit any swagger changes before merge to ensure they are up to date in the repository.packages/client/src/generated
does not need to be tracked. We should set up a way to handle hot regeneration of the client for development. Right now publishing is manual, we will want to generate and publish this package after deployment of coinstacks.A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.