phptek / silverstripe-verifiable Goto Github PK

When subitting hashes to the backend for the first time, the system will only receive a preliniary JSON proof in return that contains a date, the hash and the UUID (hash_node_id field). We would like to be able to fetch the full Chainpoint Proof JSON-LD document and cache that to the specific versioned object, but this is only available 1-2 hours after submisison due to the time taken for the Bitcoin network's PoW system to confirm the transaction in which our hash (among many others) is included.

Suggest we use silverstripe/crontask module to periodically query backends for a full proof, and update the version accordingly. FYI the module already skips the "Proof" field when in verifiable_fields "mode" so there is no danger of red-herring validation errors down the line.

User Story

As a content author, I would like to manually verify that my data and its hash have been anchored to the backend properly, so that I be assured that my proof is always verifiable.

Acceptance Criteria

1). When I run a "Verify" process on a verified hash, in addition I can also see a full, v3 JSON-LD Chaincode Proof that;

1.1. Highlights or the hash of my versioned data
1.2. Highlights a link to ID of the backend's transaction (e.g. BTC's TXID)
1.3. Highlights a link to ID of the backend's block (e.g. BTC's Block ID)

2). I can see the following information, that is not directly recoverable in the proof:

2.1. I can manually regenerate a version's SHA256 hash, and compare it with its remote equivalent from the JSON-LD proof
2.2. I can see the Merkle Root of which my SHA256 is a mathematical component
2.3. I can manually calculate/verify that the Merkle Root Hash is in part composed of the current version's SHA256 hash, by means of the intermediate hashes available to me in the JSON-LD v3 chainpoint proof.

Technical Notes

For 2.1). This is mostly being done already and just needs to be lifted/adapted out of VerifiableAdminController::getStatus()

For 2.3). The most useful way I have seen so far, of calculating intermediate hashes, is to use a semi pre-baked method, as per the https://github.com/chainpoint/chainpoint-parse lib.

Useful resources:

• https://github.com/chainpoint/chainpoint-parse (JS)
• https://github.com/chainpoint/chainpoint-cli (JS)
• https://github.com/aantonop/chainpoint (Python)
• https://github.com/NebulousLabs/merkletree (Go)

The change doesn't appear in the asset admin CMS fields

PhpTek\Verifiable\Backend\BackendServiceFactory:
  backend: chainpoint

SilverStripe\Assets\File:
  extensions:
    - PhpTek\Verifiable\Model\VerifiableExtension
  verifiable_fields:
    - Title

At present, there is a single service (VerifiableService) that is used as the broker between SilverStripe's versioned DataObject's and the currently active backend (e.g. "chainpoint"). The idea was that the service should be flexible and backend agnostic such that it could adapt to any backend.

Some non-critical, but nevertheless annoying drawbacks to this approach have come to light; the main one being that methods named in the service do not match the functionality required of them by SilverStripe nor do they match the functionality afforded by the current backend.

As an example: In our VerifiableExtension we call a read() method on the service, which then makes a getProof() call on the backend. The service knows it can reasonably expect a getProof() method to exist becuase on the backend, becuase all backends implement BackendProvider. However; it does take a second or two to wrap your head around what you want the application to do, and how the backend will acheive it, based on method-naming alone.

The solution would be something like the classic "Gateway" <--> "Service" mapping, where for each backend a dedicated gateway class exists that would declare its methods identically to the backend in-use, and a service class that mapped the gateway methods to those required within SilverStripe.

Therefore; for each available backend, you'd have a separate service and gateway. A first-pass at this would be to simply rename rename the Chainpoint class to ChainpointGateway and rename its methods to suot those available in Tierion's REST API. Then to rename VerifiableService as ChainpointService and rename its methods to suit the SilverStripe side of things, and then map the latter to the former. All that remains is to ensure the correct service is instantiated by SIlverStripe's Injector which can be done fairly easily using a custom injector factory (See Injector::setObjectCreator() and the creator YML config.

As a developer, I would like to be able to include field-data on related tables, so that more of my data can be covered by a blockchain hash

User Story

As a developer, I would like to craft a call to verify() without triggering a verification procedure, so that I have control over the verification process.

Acceptance Criteria

• I can write a custom verifiy() method that returns false
• Methods that return false do not trigger a verification "write" procedure to the Chainpoint network

User Story

As a developer, I need to ensure that changing the verifiable_fields in YML config, will not affect the verifiability of previous versions.

Acceptance Criteria

• I can change the list of fields in verifiable_fields, and previously verified versions, are still verifiable

Notes

When the verifiable_fields hash-mode is configured, we need to cache those fields locally and ensure the cached versions are only used in subsequent verification rounds.

This is to prevent developers from changing the list of fields (adding, removing, or re-ordering) in YML config which will obviously lead to different hashes and therefore red-herring results when validating previously verified versions against the backend.

User Story

As a content author or web-application owner, I would like to be able to check the integrity of the verifiable module's codebase, so that I know it has not been hacked or modified in an unauthorised manner.

Acceptance Criteria

• I can see a checksum or hash of all the files that comprise the project on GitHub
• I can see a checksum or hash of all the files that comprise the project on the filesystem
• I can visually compare the above
• I am alerted in some way that these two hashes or checksums differ

Notes

• https://stackoverflow.com/questions/460576/hash-code-and-checksum-whats-the-difference#460581

As an applocation user, I would like to check that a chainpoint proof for my verified data, is a valid chainpoint document, so that I can be assured it is authentic.

See: https://github.com/chainpoint/chainpoint-proof-json-schema

Proofs are not sought from the backend or written, on object creation. Therefore it makes no sense to display "Version 1" in the CMS' "Verify" section. Disable or omit it.

At the moment, the UpdateProofController will only submit individual UUID's (node_hash_id JSON field) to the proofs REST endpoint. This endpoint is however capable of having multiple (up to 1000) submitted simultaneously.

In order to reduce the no. network requests the application needs to make to the Chainpoint network, the CLI controller should be able to produce full-proof requests for each version of each applicable class, all at the same time.

Some thought is needed around what should happen viz verifiability" and UX when Versioned::onAfterRollback() or DataObject::onAfterDelete() are called. This is legit behaviour in SilverStripe, and for CMS users in any system (programmatically too), but goes against the grain of verifiability if one can no longer verifiy becuade an item is not immutible.

We could disallow rolling-back to previous versions if it's verifiable - and if indeed it's even a problem (Can we revert back to the version prior to rolling back!!?). Also "Deletion" in a Versioned context means "archived" right, so it's still available?

But so far, the module deals only with verifiability on the pretext that records exist in some way, they've just been changed. What about if a version was hard-out deleted? Then the current model falls over, as a CMS author can not be reasonably expected to remember all available versions of all her versioned pages (for example).

One idea is to create a cron-task that periodically queried Tierion for hashes submitted by our system. But becuase the module doesn't know what it doesn't know viz deleted DB records, this implies that our hashes have a unique signature (or are really encrypted strings, salted with a known nonce - e.g. a nonce that is at least the same across all versions of a given record - nonce-like if you prefer). This means that theoretically, logic could be written that queries the DB for all nonces, and for each hash composed of each nonce, pulls the related proofs out of Tieron (How!!?). A comparison is then made against the database where we diff known, local hashes with those out of Tierion. Should a discrepency occur we might be able to calculate or infer for which SiteTree record, a version is missing.

While the UpdateFullProofController deals with automated proof querying, it should also be possible to fetch them independently for example for admin users, in between crons. Use the above controller via XHR to query and update a version's proof if an updated one exists. The task already accepts GET params, so this is just a JS+UX task

User Story

As an administrative user, I would like to be alerted whenever my content changes its verifiable status from "Verified" to anything else

Acceptance Criteria

• I can be optionally alerted by Email or SMS (for example) should any content be flagged as changing status from "Verified" to anything else.

This is related to #37.

Upon publishing any content model, the SilverStripe "success" message should be optionally augmented with information around Chainpoint and BTC submission status.

Rename the current verify action on VerifiableController to adminverify (or similar) and then add "CMS-only" checks to it, to prevent access to it.

As an author, I would like to see the status of my content, so that I can quickly review that my content is verified.

The "Proof" field should not form part of ordinary day-to-day content editing. A prevention measure should be implemented that prevents it from being shown in the admin UI via *CMSFields().

Acceptable solutions might be:

• Throwing an exception
• Returning a user_error()
• Always rendering such an input field as readonly

If a bad status is returned when selecting "verify" in the admin section, the button's icon should not be a tick, it should be an "x" or something similar.

User Story

As an administrative user, I would like to see all of my application's records' verified status, so that I can be made aware of any content-verified status changes at-a-glance.

Acceptance Criteria

• I can see a list in the SilverStripe admin backend, that shows me the verified status of all of my verifiable content

Technical Notes

• An automated cron-task will need to run, that "simulates" what a user currently does manually to verifiy individual content, and from this a set of internal records can be periodically produced from which a report can be produced

Sometimes in the admin area, selecting the "verify" button will incorrectly (or not at all) pickup the selected version from the dropdown and undefined is used, which obviously fails.

This is most often observed when selecting a page from the site-tree, and then selecting the "Verify" tab and "verify" button as though the correct load/select JS handler isn't being called.

User Story

As a stakeholder, I would like to ensure that my application data is verifiable, so that I know it has not been tampered with.

Acceptance Criteria

1. I can verify that my data hasn't been altered
2. I can optionally provide a digital signature for notarisation of data and documents

Technical Notes

• Verification backends are:
• Trillian
• Tieron
• Only one verification backend is enabled at any time
• Backends are modeled with adaptors that implement BackendProvider
• Each adaptor may comprise >1 class to independently handle connection, querying, writing etc.
• Config allows 2 modes of operation, with only one mode at any time able to be in effect

1. Hashes unsigned
2. Hashes digitally signed

If for any reason the selected object version contains no proof. Disable the "Verify" button using client-side logic. This does of course beg the question "Why is it empty if it isn't the first version?" (See #10). So perhaps it would pay to interrogate the database in some way, to see if any malfeascent behaviour can be acertained from Created and LasteEdited fields.

User Story

As a content author or CMS user, I would like to know how the verification process works, so I can be sure of the verifiability of my content.

Acceptance Criteria

• I can see a help (or FAQ) section in the relevant "Verify" admin-area
• The help section describes what verifiability is, and why it's a good thing
• The help section describes how I can use the data at-hand, to "manually" verify that the my local data has actually been anchored to a 3rd party backend.

Tech Notes

• Relates to work done in #23

User Story

As an administrative user, I would like to be assured that my local content hashes are intact, so I know they form part of the advertised Merkle Root.

Background

Chainpoint, like Bitcoin, makes use of a Merkle Tree, the root of which is stored as part of a Bitcoin transaction approximately every hour. There is no easy, user-friendly means to calculate that any hash is part of a tree with a given root, besides going through a Chainpoint node.

In addition to the unidirectional tree used in version 1.0 of this module, we can use the same underlying library to quickly take a user inputted hash and generate its merkle root in accordance with a chainpoint receipt. Users should be able to determine that a false or modified hash does not hash to the same root value.

Admin URL format seems to have changed such that verification requests are no longer working. The fault lies in the poor URL parsing in client/dist/js/verifiable.js and the model ID not being passed to the rest of the JS logic.

Solutions:

• Remove all non-numeric values from urlArray and take the first nmeric value that comes after the selected ClassName.

As a content migrator, I would like to migrate my verified content to a newer version of SilverStripe so that I can be assured the content hashes will remain unchanged.

When the verifiable_fields mechanism is used, it is desirable not to be hashing anything that might be described as "styling" or "presentation".

Imagine a web-app featuring this module being be ported to a new version of SilverStripe, with a new theme. Its content having been littered with markup and inline styles, all added via a WYSIWYG editor. The latter would need to be stripped out to suit the new design or theme. If this were done, it leaves the stored hashes in the existing database as being invalid themselves, their having been created with the spurious markup included, resulting in red-herring validation warnings.

This "spurious content" should be stripped out, prior to hashing.

User Story

As an administrative user, I would like to see all the possible validation states a versioned record can be in, so that I have context for when something becomes invalid

Acceptance Criteria

• Upon validating a record, or in future, viewing a record's validation status, I can see a bullet-list of all available states that a versioned record can be in
• Upon validating a record, or in future, viewing a record's validation status, the selected record's status is clearly indicated to me via highlighting or by some other means

At the moment, the module will only submit and retrieve hashes to/from the Tierion network to a single, randomly selected, chainpoint node IP. Tierion reccommends submitting to multple nodes to reduce the chances of a node being unavailable for subsequent requests - part way through an initial --> full proof transformation process.

This issue is complete when:

1. A minimum of three nodes are sent requests to the /hashes endpoint
2. The no. nodes submitted to, is configurable in YML config

As a content author, I would like to view the state of my verified content, so I can be assured nothing has been tampered with.

A user friendly display of the selected version's hash, UUID, submitted date, verification date and Bitcoin TXID in the UI after a verification request is completed.

As an author, I would like to be able to notarise my data
So that I can verify that it was me that made a change

This story adds a further layer to data verification, whereby users can not only verify that data hasn't changed, but that the initial data was created by them, and them only. (By them, we mean "person" or "system" in posession of the signing private key)

Rename the x2 DB fields declared in VerifiableExtension and prefix with 'v' or 'verifiable` so developers and DBAs can distinguish them easily.

Fix the tight-coupling that exists between Extension --> Service --> Chainpoint backend

As a user, I would like to push my verification hashes directly to the Bitcoin network, so that I know no intermediary has messed with my hashes.

Implement logic that allows a local Bitcoind or Geth node to be queried and written-to directly. It is not clear at this stage if this would be for "one-off" notarisation events from within the admin area or programmatically, or for bulk hash submission to the blockchain network. If the latter case, an intermediary step of Merkle Tree creation and root-hash submission would need to feature - lest the user need to pay exhorbitant TX fees for anchoring hashed data to the blockchain.

Currently, hash subissions occur onBeforeWrite(), but verification hashes are only relevant to published objects, given that this is exactly when new version records are created.

The possibility exists for tampering to occur in the time elapsed between an object's publication, and the Bitcoin transaction in which that published version's hash is embedded, is confirmed.

A user could publish an article, and parts of it could be subject to unauthorised change (outside of manual verification by proof reading for example) until the relevant Bitcoin transaction is confirmed, and the Chainpoint network is able to return a proof.

Some way should be found to preclude this from happening using perhaps an alternative 3rd party service to temporarily provide tamper evident services such as a simple AWS lambda - a "poor mans hash verification service" if you like. This would only be in service while the chainpoint network is unable to provide a proof response from the Bitcoin network.

Trillian is a Merkle Tree storage system developed my Google. It could be used as an alternative to blockchain storage. Although of course data submitted to it would be much less publicly verifiable, it would be useful for enterprise use, where the data has no need of being publicly verifiable - the submission of automated or IoT device audit-data for example.

This issue suggest building an alternative backend, that implements BackendProvider and is confiured in YML as per the docs. Tips on installing Trillian (It's a Go app) can be found on https://github.com/google/trillian and in this module's "docs" dir.

As a developer, I would like a flexible API, so that I can submit verification requests without having to rely on any publish or write actions.

Replace the dependency on silverstripe/cms in composer. The module should only need to depend on silverstripe/versioned.

As a translator, I would like to be able to easily submit translation files for translation, so that the module can be used by used by a non-English speaking audience.

This story is not strictly about translation itself, more the infrastructure to allow translations to occur. In short: Convert all inline English text into YML lang files and use _t() throughout.