- File distribution and retrieval use Kademlia nodes to locate target BatNodes
- Kademlia nodes need to send out store rpcs after they receive a shard
- Cli integrates with new file distribution and retrieval methods
- Duplicates of each shard are created and distributed for data redundancy
- Shards can be audited by the data owner*
- Data format of shard transfer is changed to remove the limitations of JSON
- Kademlia nodes can communicate behind different NATs
- Kademlia nodes broker connections between BatNodes so that BatNodes can communicate behind different NATs
- Data redundancy distribution and retrieval w/ kademlia nodes
- Refactor nested callbacks w/ Async control flow
kd = kad node bt = bat node tkd = target kad node tbt = target bat node
- kd performs an iterativeFindNode for <= k kd nodes w/ closest ids to file id
- kd returns those nodes contacts to bt
- bt iterates through contacts until the following subroutine succeeds:
- For each target kad node (tkd):
- bt asks kd to send a broker_connectionRPC to tkd. Broker connection's payload is bt public conect tuple
- tkd receives RPC and tells its target bat node (tbt) to send establish_connection tcp request to bt tuple
- tkd simultaneously sends a success RPC to bt, containing tbt's public tuple
- bt sends establish_connection tcp request to tbt
- if bt receives the establish_connection request, it sends a standard store_file message in response
- if tbt is instead the one to receive establish_connection request, it sends a ready_to_store message in response, and bt responds to that with standard store_file message
- if bt does not receive a response with X amount of time, move onto the next tkd and try again, else, cease iteration
- when tbt finishes receiving file, it initiates an iterative store on its tkd
Edge cases:
- The node doesn't have enough allocated storage
- The nodes are already connected from previous exchange, so both establish_connection requests go through, causing the file to be sent over twice
node1
will write to node2
, which resides in another directory.
Starting condition:
demo/batnode1/hosted
has anexample.txt
file.- Delete any files in
demo/batnode2/hosted
.
- cd into
demo/batnode1
- run
node batnode.js
- open a new terminal session and
cd
intodemo/batnode2
- run
node batnode.js
Ending condition: demo/batnode2/hosted
has a example.txt-1
file.
client node distributes shards to two server nodes
Starting condition:
demo/demo_upload/client-node/personal
has anexample.txt
andtest.pdf
demo/demo_upload/server-node
andserver-node-2
directories have anexample.txt
andtest.pdf
files
cd
intodemo/demo_upload/server-node
, and runnode batnode.js
. That server is now running.- Open a new terminal session, and
cd
intodemo/demo_upload/server-node-2
and do the same. cd
intodemo/demo_upload/client-node
and runnode batnode.js
There should now be:
- shards files in
demo/demo_upload/client-node/shards
- A manifest file in
demo/demo_upload/client-node/manifest
- 4 (with current settings) shards in
demo/demo_upload/server-node/hosted
andserver-node-2/hosted
.
Run git clean -xf
to have all the files generated by the demo wiped.
Run git clean -f -d
to remove untracked directories.
In this demo, all nodes are comprised of a batNode-kademliaNode pair. A bat node is responsible for transferring and storing file data, while a kademlia node is responsible for locating files and nodes on the network.
To run this demo, you will need three terminal windows.
First, clone the repo, cd
into it, and install dependencies. Then, follow the commands below:
In the first terminal window:
cd kad-bat-plugin/node1
rm -rf db
rm hosted/*
node node1.js
In the second terminal window:
cd kad-bat-plugin/node2
rm -rf dbb
rm hosted/*
node node2.js
In the third terminal window:
cd kad-bat-plugin/node3
rm -rf dbbb
rm manifest/*
rm shards/*
node node3.js
What you will see:
In node3's folder, you should see 8 shards created in the shards folder and a single file generated in the manifest folder. You should then notice that the shards are distributed across nodes 1 and 2. Specifically, the shards in node3 should be found in the hosted
folders of nodes 1 and 2.
-
Same steps for node1 & node2 in above "3. Distribute shards to multiple server nodes (batnodes use kadnodes to locate viable hosts)".
In the third terminal window:
cd kad-bat-plugin/node3
(system can't verify correct file path if you don't go to the client's directory)rm -rf dbbb
rm manifest/*
rm shards/*
node clinode3.js
-
Open another(4th) terminal window, select the options to upload/download files while connecting to node1
batchain-sample -u <filePath>
:batchain-sample -u './personal/example.txt'
batchain-sample -d <manifestFile>
(make sure don't modify the db folders under node1~3)
- If your server window keeps running, you can view your current uploaded lists in another window
batchain -l
- You can always run
batchain -h
to review available command and options
For npm
:
- Run
npm install -g
before running anybatchain
option or command, make sure to - Need to run
npm install -g
when making bin changes - If "chalk" is not working for you, run
npm insatll chalk --save
to make the command line more colorful
For yarn
:
- Run
yarn link
to create a symbolic link between project directory and executable command - Open another terminal window, run
batchain
and you should see:
Usage: batchain [options] [command]
Commands:
sample see the sample nodes running
help [cmd] display help for [cmd]
Options:
-h, --help output usage information
-l, --list view your list of uploaded files in BatChain network