transmute-industries / transmute Goto Github PK

View Code? Open in Web Editor NEW

58.0 13.0 12.0 51.65 MB

The Transmute Command

Home Page: https://transmute.industries

License: Apache License 2.0

Shell 0.13% JavaScript 1.43% TypeScript 98.44%

transmute cryptography decentralized-identifiers github-actions graph ietf jose jsonld jwe jwk

transmute's Introduction

Transmute Command

Questions? Contact Transmute | Verifiable Data Platform | Our Guide | About Transmute

Usage

As a global binary:

npm i -g @transmute/cli

As a github action:

- uses: transmute-industries/[email protected]
  with:
    neo4j-uri: ${{ secrets.NEO4J_URI }}
    neo4j-user: ${{ secrets.NEO4J_USERNAME }}
    neo4j-password: ${{ secrets.NEO4J_PASSWORD }}
    json: |
      {
        "@context": ["https://www.w3.org/2018/credentials/v1"],
        "type": ["VerifiablePresentation"],
        "verifiableCredential": [
        ...

Commands

Develop

npm i
npm t

Testing Commands

Install a local build of the cli globally using:

npm i -g .

Test a local build with this npm script alias:

npm run transmute -- controller key generate \
--alg ES384 \
--output examples/keys/private.ES384.json

Testing GitHub Actions

You will need to use a remote neo4j instance to test with act locally.

Help wanted resolving related docker network issues.

act -j act-preview-neo4j --secret-file .env

transmute's People

Contributors

Stargazers

Watchers

Forkers

joshuacox ramene ramilexe daniel-yavorovich gjgd cardgameindustries vladimirkobets dysnix vanquishsecurity fruitbox12 thedigitaloctopus

transmute's Issues

Broken ipfs directory permissions

OS: MacOS X High Sierra 0.13.4 (17E199)
Minikube version: v0.24.1
Kubernetes 1.10

DaemonSet has run params:

      securityContext:
        fsGroup: 1000
        runAsUser: 1000

But mounted directory owned by root.

$ kubectl logs decentralized-storage-ipfs-0
ipfs version 0.4.9
Error: /data/ipfs is not writeable by the current user
initializing IPFS node at /data/ipfs

ipfs is broken when bundled by framework...

one wonders why we even bundle ipfs instead of using dependency injection...

here is a good example of why DI would be better:

var fs = require("fs");

var ipfsAPI = require("ipfs-api");
var ipfs = ipfsAPI({ host: "localhost", port: "5001", protocol: "http" });

// const TransmuteFramework = require("transmute-framework").default;
// T = TransmuteFramework.init({
//   providerUrl: "http://localhost:8545",
//   ipfsConfig: {
//     host: "localhost",
//     port: "5001",
//     options: {
//       protocol: "http"
//     }
//   },
//   TRANSMUTE_API_ROOT: "http://localhost:3001"
// });
// let ipfs = T.TransmuteIpfs.ipfs;

let cid = "/ipfs/QmU38PjH6AJGqYdVgRq6yHkPkV4DETrfUX6Cwjfk9PYMx4";

const writeFile = (path, data) => {
  return new Promise((resolve, reject) => {
    fs.writeFile(path, data, err => {
      if (err) {
        reject(err);
      }
      resolve(true);
    });
  });
};

ipfs.files.get(cid, (err, files) => {
  //   console.log(files);
  files.forEach(async file => {
    if (file.content) {
      let localPath = file.path.replace(
        "QmU38PjH6AJGqYdVgRq6yHkPkV4DETrfUX6Cwjfk9PYMx4",
        "./example"
      );
      await writeFile(localPath, file.content.toString("utf8"));
    }

  });
});

truffle:test failed

Hello! I have problem with truffle tests running:

$ lerna run --scope transmute-framework truffle:test
...
Error parsing /Users/daniel/.transmute/git/transmute/packages/transmute-framework/contracts/EventStore.sol: ParsedContract.sol:13:16: ParserError: Expected identifier, got 'LParen'
    constructor() public {
               ^
Compilation failed. See above.
...

$ git branch
* master
$ helm ls ganache
NAME   	REVISION	UPDATED                 	STATUS  	CHART            	NAMESPACE
ganache	1       	Mon May  7 01:28:26 2018	DEPLOYED	ganache-cli-0.1.0	default

This is deploy/configuration issue (maybe need use other software version) or code bug?

Thank you!

Implement proper logger in bootstrap process

https://github.com/TrueBitFoundation/truebit-os/blob/master/os/logger.js is a good place to start here.

DI Everywhere...

the framework should not take json artifacts when its instantiated... instead these should be provided when needed later.

The framework should expose classes which support DI, so that we dont have to bundle things like firebase and ipfs.

Add certbot instructions to setup

https://certbot.eff.org/all-instructions/

Wait on tiller is not implemented correctly

https://travis-ci.org/transmute-industries/transmute/builds/387173103#L1698

transmute k8s init macOS-cluster fails when service account already exists

##### create tiller service account
Error from server (AlreadyExists): serviceaccounts "tiller" already exists
exec: 
real	0m1.957s
user	0m0.561s
sys	0m0.211s
Error from server (AlreadyExists): serviceaccounts "tiller" already exists

/usr/local/lib/node_modules/transmute-cli/src/commands/runner.js:8
    throw new Error('Command FAILED, tell us what happened so we can fix it: https://github.com/transmute-industries/transmute/issues/new');

CLI: Init should explain why sudo is required.

asks for password to modify /etc/hosts but does no explain why.

We basically just need an echo here:

https://github.com/transmute-industries/transmute/blob/master/scripts/initializer#L147

Need to run `lerna bootstrap` twice after fresh clone of repo

Getting this error the first time:

lerna info version 2.8.0
lerna info versioning independent
lerna info Bootstrapping 8 packages
lerna info lifecycle preinstall
lerna info Installing external dependencies
lerna ERR! execute callback with error
lerna ERR! Error: Command failed: npm install
lerna ERR! npm ERR! cb() never called!
lerna ERR! 
lerna ERR! npm ERR! This is an error with npm itself. Please report this error at:
lerna ERR! npm ERR!     <https://github.com/npm/npm/issues>
lerna ERR! 
lerna ERR!     at Promise.all.then.arr (/usr/local/lib/node_modules/lerna/node_modules/execa/index.js:236:11)
lerna ERR!     at <anonymous>
lerna WARN complete Waiting for 3 child processes to exit. CTRL-C to exit immediately.
{ Error: Command failed: npm installdencies: verb installed leaf transmute-dash
npm ERR! cb() never called!

npm ERR! This is an error with npm itself. Please report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

    at Promise.all.then.arr (/usr/local/lib/node_modules/lerna/node_modules/execa/index.js:236:11)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
  code: 1,
  killed: false,
  stdout: '',
  stderr: 'npm ERR! cb() never called!\n\nnpm ERR! This is an error with npm itself. Please report this error at:\nnpm ERR!     <https://github.com/npm/npm/issues>\n',
  failed: true,
  signal: null,
  cmd: 'npm install',
  timedOut: false,
  exitCode: 1 }

CLI: Split up initializer.

The initializer script needs to be split up into components:

Kong
IPFS
Ganache

Instead of waiting for all of these to be installed at once, we want to make the CLI aware of how to install each of them, independently.

For example:

transmute k8s microservice install kong

Should install kong.

For example:

transmute k8s microservice install ganache

Should install ganache and configure kong to support ganache.

The initializer bash script should also be moved into the CLI npm package, and that package should be organized by microservice, for example:

transmute-cli/src/microservices/kong/install_kong.sh
transmute-cli/src/microservices/ganache/install_ganache.sh

The idea being that the CLI is used to install microservices, and is aware of how to do so for minikube, azure and gke.

Wallet Initialization error

  const makeWallet = async (T) => {
    let mnemonic = await T.Toolbox.generateMnemonic(); // normally this should be a secret!
    let address = T.Toolbox.getDefaultAddressFromMnemonic(mnemonic);
    let wallet = T.Toolbox.getWalletFromMnemonic(mnemonic);
    return {
      mnemonic,
      wallet,
      address
    };
  };
  
  let wallet2 = await makeWallet(T);

  let walletConfig = {
    ...T.config,
    wallet: wallet.wallet
  };
  T = T.init(walletConfig);

though web3 can clearly send funds from the wallet, the EventStore seems to not be reiniatlized correctly...

(node:17069) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Error: could not unlock signer account

Demo Ganache RPC Exposed and allows bots to siphon funds from accounts

Helm tuning for local install

By default helm is not response from host machine by http/8080:

Error: Get http://localhost:8080/version: dial tcp 127.0.0.1:8080: connect: connection refused

Please add to script workaround for allow access to helm:

kubectl -n kube-system patch deployment tiller-deploy -p '{"spec": {"template": {"spec": {"automountServiceAccountToken": true}}}}'

Can't create instance of EventStore

Using dev config for transmute-dashboard yields the following

Uncaught (in promise) Error: Cannot create instance of EventStore; no code at address 0x057d4e4cb4568426600fea17ec3e51192c487e17

Transmute Daemon

We'd like a simple rest api service to run inside the cluster, to provide a standard way of asking questions of services running in the cluster, and also to help with maintenance and configuration.

To start, the daemon should implement a healthcheck for the services installed.

provision-minikube fails of macOS after bootstrap

› transmute k8s provision-minikube macOS-cluster
Starting local Kubernetes v1.9.0 cluster...
Starting VM...
E0626 11:40:13.524691   61673 start.go:168] Error starting host: Error getting state for host: machine does not exist.

Dashboard ESigner Demo Fails Migrations

ESigner contract needs to import Factory, or truffle will not generate assets for it, and you will see:

Saving artifacts...
Running migration: 2_deploy_contracts.js

/usr/local/lib/node_modules/truffle/build/cli.bundled.js:62064
  throw new Error("Could not find artifacts for " + import_path + " from any sources");
        ^
/usr/local/lib/node_modules/truffle/build/cli.bundled.js:62064
  throw new Error("Could not find artifacts for " + import_path + " from any sources");
  ^

Error: Could not find artifacts for transmute-framework/contracts/EventStoreFactory from any sources

Here is the change that needs to be made.

pragma solidity ^0.4.19;

import "transmute-framework/contracts/EventStore.sol";
import "transmute-framework/contracts/EventStoreFactory.sol";

contract ESigner is EventStore {

   
}

.ci directory cleanup

folders for
linux, mac, local

deps of bootstrap like: ubuntu-compile-nsenter.sh

don't belong in the ci directory.

.ci directory should be only things needed by ci to perform work.

./setup/3.install_ipfs.sh script syntax error

./setup/0.init.sh: 5: ./setup/3.install_ipfs.sh: hthhps://raw.githubusercontent.com/openebs/openebs/master/k8s/openebs-storageclasses.yaml: not found

API / Dashboard JWT Incompatibility

When trying to leverage the api from the dashboard with the okta-auth react package, the following error is thrown: claim 'cid' value '0oaeszrkwvP4Z4SHL0h7' does not match expected value '0oaeszxzq7o0zBFcy0h7'

This is because the client id in the API and Dashboard do not match.

Remove READMEs from package subdirectories

These pull down from published npm modules which are likely incompatible with the existing changes in the repo.

lack of encryption on ipfs objects

Now transmute encrypt only ipfs key (part of url), but content is open.

This arch can implement men-in-middle vulnerability, because proxy servers using the ssl and CONNECT method allowed to catch links.

Remove LocalStorage

Right now this is causing the demo to persist information between separate sessions where the data in a subsequent session may no longer exist.

Bootstrap fails to check for and install virtualbox on osx

transmute cli is unable to provision minikube because of this.

brew cask install virtualbox

Transmute Contracts

Pull truffle contract stuff into seperate package from framework.

add https://github.com/sc-forks/solidity-coverage

Bootstrap script not installing transmute cli on Mac

After running curl -Ls https://git.io/transmute_bootstrap | bash, I get -bash: transmute: command not found when trying to use the cli.

Fix Dashboard contracts

import 'transmute-framework/contracts/EventStoreFactory.sol'; to any inheriting smart contract.

this resolves the

Running migration: 2_deploy_contracts.js

  throw new Error("Could not find artifacts for " + import_path + " from any sources");

Error: Could not find artifacts for transmute-framework/contracts/EventStoreFactory from any sources

error

truffle hangs with lerna

when running lerna run truffle:test - truffle will hang and not run all test suites
when scoping with lerna run truffle:test --scope transmute-services - truffle will hang and not run any tests at all

you can run truffle test in these directories directly, though, and things will work fine.

Intermittent Build Failure: gpg: Can't check signature: No public key

gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
gpg: requesting key CD9775F2 from hkp server pool.sks-keyservers.net
gpgkeys: key A2CFB51FA275A7286234E7B24D17C995CD9775F2 can't be retrieved
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
gpg: requesting key CD9775F2 from hkp server ipv4.pool.sks-keyservers.net
gpgkeys: key A2CFB51FA275A7286234E7B24D17C995CD9775F2 can't be retrieved
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
gpg: Signature made Wed 13 Jun 2018 01:31:35 AM UTC using RSA key ID CD9775F2
gpg: Can't check signature: No public key

Can't download the key (key server probably feels spammed). can't validate signature...

K8s default version

Before using the cli to provision and initialize minikube you need to run:

export KUBE_VERSION=v1.9.0

on osx.

Update .example.env with empty values

export MINIKUBE_IP=$(minikube ip)
export KONG_NGROK_HOST=[YOUR_KONG_NGROK_HOST]
export KONG_CONSUMER_USERNAME=[YOUR_KONG_CONSUMER_USERNAME]
export KONG_CONSUMER_ID=[YOUR_KONG_CONSUMER_ID]

export OKTA_HOSTNAME=[YOUR_OKTA_HOSTNAME]
export OUTPUT_FILE='./setup/okta.pem'

export OKTA_CLIENT_ID=[YOUR_OKTA_CLIENT_ID]
export OKTA_CLIENT_SECRET=[YOUR_OKTA_CLIENT_SECRET]
export OKTA_USERNAME=[YOUR_OKTA_USERNAME]
export OKTA_PASSWORD=[YOUR_OKTA_PASSWORD]

export SECONDS_FOR_KONG=600

export IPFS_API_HOST=[YOUR_IPFS_API_HOST]
export IPFS_API_PORT='32443'
export IPFS_API_PROTOCOL='https'

Dashboard README does not include contract linking and deployment

When you try to deploy a new EventStore, you get the following:

Error: EventStore contains unresolved libraries. You must deploy and link the following libraries before you can deploy a new version of EventStore: AddressSetLib, Bytes32SetLib, EventStoreLib

Add jwk-to-pem to package.json

https://github.com/Brightspace/node-jwk-to-pem

Add Example .env for Dashboard

init w8 is messy on osx.

transmute k8s init macOS-cluster

yields:

.cut: illegal option -- -
usage: cut -b list [-n] [file ...]
       cut -c list [file ...]
       cut -f list [-s] [-d delim] [file ...]
.cut: illegal option -- -
usage: cut -b list [-n] [file ...]
       cut -c list [file ...]
       cut -f list [-s] [-d delim] [file ...]
.cut: illegal option -- -
usage: cut -b list [-n] [file ...]
       cut -c list [file ...]
       cut -f list [-s] [-d delim] [file ...]

Consider Pender

https://github.com/velopert/redux-pender

Error running bootstrap

### Installing coreutils
Error: coreutils 8.29 is already installed.

mac mini (late 2014)
macOS: 10.13.3

Directory transmute-config not exist

I have some problems with local deploy transmute:

$ npm run setup
...
[yn]y
ok, I will copy the example environment file to transmute-config/.env
cp: ./transmute-config/.example.env: No such file or directory
Would you like to edit the environment file?
[yn]

$ curl -Ls https://raw.githubusercontent.com/transmute-industries/transmute/master/bootstrap | bash
...
bash: line 475: cd: /Users/daniel/.transmute/git/transmute/transmute-config: No such file or directory

Each installer try to find transmute-config dir in root directory of repository, but this dir doest not exist.

Where I can find config example? I have found 2 dirs:

./packages/transmute-framework/src/transmute-config
./packages/transmute-dashboard/src/transmute-config

But I think that this is not what I need.

Thank you.

Bootstrap script is not idempotent

fails when run multiple times.

Work in progress branch:

https://github.com/transmute-industries/transmute/tree/fix/transmute-bootstrap-for-osx

The approach... clone this branch, run bootstrap locally repeatedly... eliminate all errors.

bootstrap errors

While working working with JVL I came across the bootstrap not acting right,

it dies on:

You have $NVM_DIR set to "/home/xmute/.nvm", but that directory does not exist. Check your profile files and environment.

and also

certbot-auto: Permission denied

Cannot write to ‘certbot-auto’ (Permission denied).

Can't build lerna repo

When running lerna run build --ignore transmute-contracts --ignore transmute-framework --ignore transmute-alpha, I get the following:

lerna info version 2.8.0
lerna info versioning independent
lerna info ignore [ 'transmute-contracts',
lerna info ignore   'transmute-framework',
lerna info ignore   'transmute-alpha' ]
lerna ERR! build Errored while running script in 'transmute-crypto'
lerna ERR! execute callback with error
lerna ERR! Error: Command failed: npm run build
lerna ERR! sh: rimraf: command not found
lerna ERR! npm ERR! file sh
lerna ERR! npm ERR! code ELIFECYCLE
lerna ERR! npm ERR! errno ENOENT
lerna ERR! npm ERR! syscall spawn
lerna ERR! npm ERR! [email protected] prebuild: `rimraf dist`
lerna ERR! npm ERR! spawn ENOENT
lerna ERR! npm ERR! 
lerna ERR! npm ERR! Failed at the [email protected] prebuild script.
lerna ERR! npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
lerna ERR! 
lerna ERR! npm ERR! A complete log of this run can be found in:
lerna ERR! npm ERR!     /Users/ericolszewski/.npm/_logs/2018-03-13T19_58_26_904Z-debug.log
lerna ERR! 
lerna ERR! > [email protected] prebuild /Users/ericolszewski/Developer/Transmute-Industries/projects/transmute/packages/transmute-crypto
lerna ERR! > rimraf dist
lerna ERR! 
lerna ERR! 
lerna ERR!     at Promise.all.then.arr (/usr/local/lib/node_modules/lerna/node_modules/execa/index.js:236:11)
lerna ERR!     at <anonymous>
lerna WARN complete Waiting for 1 child process to exit. CTRL-C to exit immediately.
lerna ERR! build Errored while running script in 'transmute-dashboard'
{ Error: Command failed: npm run build
sh: rimraf: command not found
npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn
npm ERR! [email protected] prebuild: `rimraf dist`
npm ERR! spawn ENOENT
npm ERR! 
npm ERR! Failed at the [email protected] prebuild script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/ericolszewski/.npm/_logs/2018-03-13T19_58_26_904Z-debug.log

> [email protected] prebuild /Users/ericolszewski/Developer/Transmute-Industries/projects/transmute/packages/transmute-crypto
> rimraf dist


    at Promise.all.then.arr (/usr/local/lib/node_modules/lerna/node_modules/execa/index.js:236:11)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
  code: 1,
  killed: false,
  stdout: '\n> [email protected] prebuild /Users/ericolszewski/Developer/Transmute-Industries/projects/transmute/packages/transmute-crypto\n> rimraf dist\n\n',
  stderr: 'sh: rimraf: command not found\nnpm ERR! file sh\nnpm ERR! code ELIFECYCLE\nnpm ERR! errno ENOENT\nnpm ERR! syscall spawn\nnpm ERR! [email protected] prebuild: `rimraf dist`\nnpm ERR! spawn ENOENT\nnpm ERR! \nnpm ERR! Failed at the [email protected] prebuild script.\nnpm ERR! This is probably not a problem with npm. There is likely additional logging output above.\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR!     /Users/ericolszewski/.npm/_logs/2018-03-13T19_58_26_904Z-debug.log\n',
  failed: true,
  signal: null,
  cmd: 'npm run build',
  timedOut: false,
  exitCode: 1 }