Code Monkey home page Code Monkey logo

cleargdpr's Introduction

ClearGDPR Platform Build Status

Logo

Introduction

ClearGDPR is a backend framework that allows web applications to comply with the new General Data Protection Regulation - GDPR.

ClearGDPR focuses on data privacy, transparency, integrity and availability for the users of a web application, and applies Blockchain technology to achieve those at a high level.

ClearGDPR was built on top of Quorum, a Blockchain that extends Ethereum and is focused in privacy and transparency. In ClearGDPR, Blockchain technology is used as an immutable audit log.

ClearGDPR offers a front-end SDK and an HTTP API in order to allow your web application to manage personal user data according to the GDPR articles.

ClearGDPR can be implemented in 2 different ways:

  • ElementSDK - set of ReactJS component, recommended, simplest.
  • HTTP API - There's a Postman HTTP requests collection to facilitate the initial interaction with the HTTP API.

Finally, ClearGDPR is a containerized solution, applying Docker containers, images, services, volumes and networks.

Table of contents

Architecture

ClearGDPR Architecture

Project structure

project root
├─ admin                      # Admin UI to manage the controller
├─ api                        # Code for the ClearGDPR API implementation
├─ cg                         # Code for the ClearGDPR API implementation
├─ docker                     # Helpers for docker run, stop, start.
├─ frontend                   # demo front-end integrating ClearGDPR ElementSDK
├─ packages
  ├─ js-sdk                   # Code for the ClearGDPR JS SDK implementation
  └─ element-sdk              # Code for the ClearGDPR ElementSDK implementation
├─ quorum                     # Blockchain related docker images and helper tools
├─ travis                     # Scripts used on the CI server
├─ website                    # ClearGDPR public website

Requirements

  • NodeJS v7.6+
  • Yarn
  • Docker
  • Docker-Compose

Before proceeding, follow the installation guide to install all requirements.

Quick Start

The docker-compose.yml contains configuration for a reference development environment of one Data Controller and one Data Processor and all required dependencies.

Full Installation Setup

The wizard will setup all the dependencies and demo website for you. Please follow the command line prompt.

node setup.js

Once the setup is done, you will have access to:

Development Mode

Docker is used for each part of the project, you can run all of them in watch (with nodemon) and development mode:

docker/run

Please make sure you ran node setup.js before.

Integrations

Element SDK

Demo

Coming soon.

Installation

Coming soon.

Code Example

Coming soon.

HTTP API

Example

The API uses "Bearer Authentication", in which the users must bear a valid Jason Web Token - JWT, in order to access the HTTP API. The system considers 3 types of users:

  • subject. A user of your website/app who will store his personal data into the website/app.
  • manager. A management user who has admin access to the system in you company.
  • processor. A management user who has admin access to the system in a 3rd party data partner.

Here's an example HTTP request to list all the companies who are 3rd party data partners (a.k.a processors) with your company:

curl -X GET \
  <ClearGDPR_API_URL>/api/subject/processors \
  -H 'Authorization: Bearer <JWT>' \
  -H 'Cache-Control: no-cache' \
  -H 'Content-Type: application/json'

Documentation

Website

Soon.

Postman Collections

Further reading

Each part of the project has additional README documentation in its subfolder:

Troubleshooting

See the Troubleshooting guide

Roadmap

  • Development deploy
  • HTTP API support for the right to consent and right to be forgotten(complete erasure)
  • All events related to users data are written succesfully to the blockchain
  • Quorum smart contract that stores the state of processors and controllers
  • Processor run mode and controller run mode, with events propagated between nodes through the blockchain smart contract
  • Example UI with registration/consent and erasure abilities
  • Encryption of all personal data on CG nodes
  • Robust integration test suite of CG nodes for both processor and controller mode
  • Admin dashboard design + front-end code (configuration, subject & processor status)
  • Automataic Quick Start script
  • Interactive Quick Start wizard
  • Admin dashboard hooked up to controller state via the CG api
  • HTTP API support for the remaining GDPR article actions/rights
  • More granular controls of consent/revoking of data (ie. which data can be shared specifically)
  • End-2-End test suite of controller/processor interactions via blockchain
  • SDK for implementing ClearGDPR from your frontend
  • Evolving functionalities, upgradability and security in the smart contract
  • Stage/Production deploy
  • Usage of Quorum’s custom privateFor method to whitelist nodes that are privy to specific events
  • Complete documentation
  • Drop in wordpress plugin
  • Commercial middleware plugin(s)

License

This project is licensed under GNU LGPL LICENSE

Contributing

See Contributing

cleargdpr's People

Contributors

fjcero avatar harlemcake avatar jakedavies avatar juanmnl avatar maksymilian-majer avatar nelrohd avatar segoddnja avatar sindelio avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cleargdpr's Issues

OG crashing after a while due to scrypt dependency

Issue by maksymilian-majer
Wednesday Apr 25, 2018 at 14:52 GMT
Originally opened as https://github.com/clevertech/OpenGDPR/issues/101


After some time OG app crashes and until I restart the containers it no longer works.
Here's the console log for the error:

screen shot 2018-04-25 at 16 49 26

Not sure we need scrypt any longer, it's either a peer dependency of some other npm package we're actually using (web3?) or one that we are not using (like solc for compiling Solidity, that's not relevant now).

Add a public build / publish process

Goals:

  1. Run tests for master + PR branches
  2. Track test coverage
  3. Publish docker images (only on tags?)

500 Error when exporting data and erased data

Issue by maksymilian-majer
Wednesday May 23, 2018 at 13:33 GMT
Originally opened as https://github.com/clevertech/OpenGDPR/issues/235


Steps to reproduce:

  1. Give consent for a user
  2. Export data (to see if works)
  3. Erase data
  4. Export data again

Expected result:
There's a nice user-friendly error.

Actual result:
There's an HTTP 500 error.

Additional info:
Probably it's like that with other endpoints as well for both processors, subjects, management. We need to test it thoroughly and if it breaks then write test cases reproducing the error and handle it properly.

Updating the architecture diagram

Possible changes:

[done] CG Controller Node 2 => CG Processor Node 2
[done] GDPR Actions Inovactions / payloads to encrypt & store => GDPR action invocations
[done] Blockchain contract writes => Smart Contract state update or Smart Contract function invocation
[done] User facing site => www.times.com
[too long and unnecessary] Syncs user data when Blockchain notifies of updates to state => Syncs user data when Quorum Node notifies of updates to the Blockchain state
[unnecessary] Blockchain state Sync => Smart Contract state sync
[unnecessary] Blockchain Events => Smart Contract events
[unnecessary] The Quorum Blockchain is in the middle in a separate box, but in reality it's inside the Quorum Nodes, each with a copy

Interactive quick start script

The Quick Start guide can be mostly converted into a wizard like interactive shell script and generate the right config, .env files, docker-compose settings, etc.

Relates to #24

Deploy a demo of the front-end

We need to have a demo deployed to either CT's production AWS or some other cloud infrastructure.

This is subject of discussions with DevOps.

Add/remove nodes dynamically in a Quorum network

The system must have the ability to add or remove nodes (aka processors) from the Quorum blockchain network after it is deployed and set for the first time.

After some research, found a way to add a node in the node Geth (Quorum node) console:

raft.addPeer($enodeInfo)

Network must be in Raft consensus. Can later check connectivity through:

admin.peers

Need more testing now.

JS SDK: foundation, elements and minified version

Issue by fjcero
Wednesday Apr 18, 2018 at 16:05 GMT
Originally opened as https://github.com/clevertech/OpenGDPR/issues/67


We need a way to provide devs and customers an easy integration with the ClearGDPR platform. Since the customer-facing API provides the basic actions to process different requests from the user, we need a layer that allows the user to execute different flows in a seamless way.

So far, this responsibility is on the side of js-sdk which is a functional wrapper of the API, abstracting some processing. Next step is providing devs and customers a way to start using this wrapper in their HTML. To solve this issue, we defined a layer of predefined elements that could be extended by the user (if necessary) which allows to set up a basic implementation of og in the minimum time.

To complete this task we need to:

  • Split basic sdk from elements framework
  • Basic scaffold for elements framework
  • Basic elements to scaffold the "give consent" flow
  • Complete "give consent" flow
  • Allow devs to inject og into their code seamless (window.og)
  • Build the sdk libraries
  • Disclaimer about dependecies with react for MVP
  • Distribute a minified version of og

Design the blockchain event auditing tool

Is your feature request related to a problem? Please describe.
No.

Describe the solution you'd like
Show the top 3 most important informations from the blockchain events in the Admin UI.

Describe alternatives you've considered
None.

Additional context
Define the top 3. @sindelio @fjcero @juanmnl

Document how to deploy ClearGDPR

Is your feature request related to a problem? Please describe.
No.

Describe the solution you'd like
Create documentation on how to deploy ClearGDPR.

Describe alternatives you've considered
None.

Additional context
The deployment is pretty complex and involves sharing permanent volumes between quorum nodes (constellation and geth). Ideally we'll also have an interactive wizard / tutorial for generating deployment configs (ex. for Kubernetes) on a popular cloud platform (AWS / Google Cloud, etc.)

Create documentation for quorum image configuration

Is your feature request related to a problem? Please describe.
No.

Describe the solution you'd like
Clear documentation on quorum image configuration.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
The Quorum image can be ran in two modes:

  1. Constellation
  2. Geth

Both have different configurations options that are required to be set as ENV variables. Some of these variables are base64 encoded. We have scripts to generate those, but for real deployments using infrastructure uncovered by our helper scripts we need to include documentation that can guide people.

Create documentation for healthchecks

Is your feature request related to a problem? Please describe.
No.

Describe the solution you'd like
The health checks for OG and OG processor will be useful for deployment (#46). It needs documenting and including in the quick start wizard (#42).

Describe alternatives you've considered
None.

Additional context
None.

Attempt a private transaction with a local quorum cluster

Issue by maksymilian-majer
Wednesday Apr 11, 2018 at 11:06 GMT
Originally opened as https://github.com/clevertech/OpenGDPR/issues/36


Now that we have quorum working locally we should attempt a private transaction between 2 nodes in a 3 node setup. We should use web3js and the test should attempt to read the consent status from the blockchain on both nodes: one that is privy to the transaction and one that is not privy to the transaction.

Useful example is included here:
https://github.com/ConsenSys/quorum-docker-Nnodes/blob/master/Nnodes/templates/contract_pri.js

Create documentation / Prepare for open sourcing

Issue by Jakedavies
Friday May 18, 2018 at 14:52 GMT
Originally opened as https://github.com/clevertech/OpenGDPR/issues/218


We are looking to open source soon(I assume some time next week). We have been asked to create some documentation.

  • Create a repository
  • Rename repository, og => cg, DBs, configs, etc.
  • Remove boilerplate docs
  • Link between README's for subfolders READMEs (under further reading and other TODO:) places
  • Organize top level README's for a better quick start
  • Script / Guide for generating secrets for local development – not good practice to contain even fake secrets in .env.example
  • OG API documentation - @fjcero
  • Document how to set up quorum for local dev
  • Troublshooting/Common problems section - @sindelio
  • Clean up any github stuff that should be cleaned up
  • What do we do with internal CT deployment source code (terraform, etc.)?
  • URLs on front-end depend on our AWS URL configs.. (config.CG_API_BASE = `${protocol}//og-${host}`;) - @Jakedavies
  • Cleanup boilerplate code (e2e tests, etc.)
  • Remove redis
  • Cleanup of obsolete helper scripts in scripts/ directory
  • Architecture diagram - @Jakedavies
  • Remove or fix of Makefiles
  • Decide on the license - @Jakedavies
  • Thoroughly test Quick Start Guide @Jakedavies @sindelio @fjcero
  • Fix broken tests @Jakedavies
  • Nice looking diagram @juanmnl
  • Generate nice docs using https://docsify.js.org/ @fjcero

For later:

  • Generate wallet private key for the tests (#38, #39)
  • Add a public build / publish process (do we publish images?) (#44)
  • Nice wordpress like installation experience instead of the Quick Start guide (#42)
  • Remove the test:integration differentiator and documentation (#45)
  • Document or remove healthchecks (#47)
  • Add coverage badge (included in #44)
  • Document production-ish quorum deployment (#46)

Preparations for open sourcing the project

Issue by maksymilian-majer
Tuesday May 22, 2018 at 17:07 GMT
Originally opened as https://github.com/clevertech/OpenGDPR/pull/232


Closes #218

Testing instructions:

  1. Ideally reset your docker:
docker stop $(docker ps -q)
docker rm $(docker ps -a -q)
docker rmi $(docker images -q)
docker network rm $(docker network ls -q)
docker volume rm $(docker volume ls -q)
  1. Clone repo to a new directory
git clone [email protected]:clevertech/OpenGDPR.git cleargdpr_os_test
  1. Enter directory
cd cleardgpdpr_os_test/
  1. Checkout open source branch:
git checkout feature/open-source-prep-rebased
  1. Follow the Quick Start guide
  2. Go to: http://localhost:3000
  3. Give consent
  4. Export data
  5. Erase data

maksymilian-majer included the following code: https://github.com/clevertech/OpenGDPR/pull/232/commits

Design Admin Dashboard

Issue by juanmnl
Tuesday May 15, 2018 at 17:31 GMT
Originally opened as https://github.com/clevertech/OpenGDPR/issues/197


As a website/app admin who has implemented Clear GDPR, I want an admin dashboard where I can access and manipulate different user data which has been impacted by GDPR ruling.

  • Design/Wireframe PoC of admin Dashboard
  • Design Dashboard layout
  • Implement Dashboard layout and main block components
  • Design layout and main block components behavior
  • Make layout and main block components responsive
  • Custom Card components for different data visualization
    • Base Card (style - behavior)
    • Basic Number Card
    • Basic Graph Card
    • Composed Number and Graph Card
    • Composed multi [Number and Graph Card]
    • Basic table card
      • Tabular data
    • Processor Preview Card
  • Details Panel behavior
  • Implement React Storybook

😅

Keep the smart contract state across different deployments

Is your feature request related to a problem? Please describe.
If we need to update the smart contract inside the Quorum network for any reason, such as fixing a bug or updating its functionalities, then we will need to import it's state (variable values) to the new contract.

Describe the solution you'd like
To complete... @sindelio maybe?

Describe alternatives you've considered
None.

Additional context
None.

Create CG endpoints for aggregate stats

Maybe just an endpoint called /stats that returns a hard coded set set of numbers as a start. At some point we would like to add timseries sort of data for some graphs? This is probably a good start though.

{
  data:
  {
    controller: {
      consented: 19292
      erased: 10,
    },
    processors: {
     120(:id){
       consented: 10
       erased: 1000
     },
     121(:id){
       consented: 17
       erased: 19922
     }
    }
  }
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.