storj-archived / bridge Goto Github PK

Before MetaDisk is production ready, we need to review some of the following:

• Indexing and schema design
• Replication vs sharding for our use case
• TODO

These fields are documented, but missing from the actual response. The fields are present in the responses for GET /buckets and GET /buckets/{id}.

To ensure that only the requester can issue the retrieval payload that is presigned by MetaDisk, it should only be decrypted with the client's private key.

When uploading a duplicate file, even in a separate bucket I get this error.

Error [Error: E11000 duplicate key error collection: metadisk-api.files index: _id_ dup key: { : "1d1f5504410f40f6628039ca8422ef1d53072248" }]

X-FileSize header must equal the file size in bytes

If you could add an optional 'pubkey' parameter to the /users POST, then clients would never have to use basic auth if they don't want to.

Documentation should be autogenerated and accessible via the API

I'm trying to test out downloading a file as multiple shards from multiple channels, but I can't seem to upload a file that is large enough to be split. After a couple minutes of waiting, I get Connection aborted: Broken pipe.

The public keys in a bucket's pubkeys field should be used to allow bucket access to clients signing their requests with the corresponding private keys.

@gordonwritescode says:

re: bucket specific pubkeys... api is there, but the auth for them is not yet implemented, let's open an issue and i'll tackle it this week

Users section:
Easy to understand and it is possible to send the request direct. Very good.

Keys section:
Basic authentication missing. Please add them to the curl examples of all the api calls.

Buckets section:
Basic authentication missing. Please add them to the curl examples of all the api calls.
Post buckets has some more parameters. A short discription would be nice. What is storage and transfer? What values are allowed?
Post tokens. A list of available operations please. How do I create a token for 2 or more operations?
Put files. curl missing file and filesize. How about Content-Type?

Since yesterday I'm'not able to authenticate to the API anymore

/usr/bin/curl -v -S --basic --user [email protected]:password -X GET --header 'Accept: application/json' https://api.metadisk.org/buckets

give me

• Trying 162.243.72.56...
• Connected to api.metadisk.org (162.243.72.56) port 443 (#0)
• found 173 certificates in /etc/ssl/certs/ca-certificates.crt
• found 692 certificates in /etc/ssl/certs
• ALPN, offering http/1.1
• SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
• server certificate verification OK
• server certificate status verification SKIPPED
• common name: api.metadisk.org (matched)
• server certificate expiration date OK
• server certificate activation date OK
• certificate public key: RSA
• certificate version: #3
• subject: CN=api.metadisk.org
• start date: Thu, 03 Mar 2016 17:02:00 GMT
• expire date: Wed, 01 Jun 2016 17:02:00 GMT
• issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X1
• compression: NULL
• ALPN, server did not agree to a protocol
• Server auth using Basic with user '[email protected]'

  GET /buckets HTTP/1.1
  Host: api.metadisk.org
  Authorization: Basic cmlja2Vubnktc3RvcmphcGlAdm90ZXotY3RodWxodS5uZXQ6cGxpY3Bsb2M=
  User-Agent: curl/7.43.0
  Accept: application/json

  < HTTP/1.1 500 Internal Server Error
  < X-Powered-By: Express
  < Vary: Origin
  < Access-Control-Allow-Credentials: true
  < Content-Type: application/json; charset=utf-8
  < Content-Length: 37
  < ETag: W/"25-06BquQnCwmlHB2RmaLK0OQ"
  < Date: Wed, 16 Mar 2016 08:08:02 GMT
  < Connection: keep-alive
  <
• Connection #0 to host api.metadisk.org left intact
  {"error":"Invalid email or password"}

It was working before yesterday. Is there any changes ?

Thx

Bridge needs to periodically check for BucketEntrys that are close to their renewal time and automatically renegotiate the associated storage contract.

1.)
get buckets, post buckets, get buckets/{id}, patch buckets/{id} different order:

{
    "user":"{e-mail}",
    "created":"2016-03-09T22:47:48.950Z",
    "name":"Skunk first bucket",
    "pubkeys":["{key}"],
    "status":"Active",
    "transfer":30,
    "storage":10,
    "id":"{id}"
}

2.)
post /buckets/{id}/tokens different order and duplicate token (token / id)

{
  "bucket":"{id}",
  "operation":"PUSH",
  "expires":"2016-03-12T15:23:42.813Z",
  "token":"9f7c94825122936aa37f64e5255c11a15b7506c704311e70be167f3141d3229f",
  "id":"9f7c94825122936aa37f64e5255c11a15b7506c704311e70be167f3141d3229f"
}

3.)
get /buckets/{id}/files different order and duplicate hash (hash / id). I guess
put /buckets/{id}/files has the same problem.

{
    "bucket":"{id}",
    "filename":"Storj.jpg",
    "size":123269,
    "mimetype":"image/jpeg",
    "hash":"cc35d79ba935e0e347f6321d0f0010c36387d065",
    "id":"cc35d79ba935e0e347f6321d0f0010c36387d065"
}

4.)
get keys, post keys different order and duplicate key (key / id)

{
    "user":"{e-mail}",
    "key":"{key}",
    "id":"{key}"
}

@super3 reported multiple verification emails. This should not happen, will try and reproduce.

MetaDisk API should expose an administrative programming interface to allow external services (like user billing and subscriptions) to integrate.

buckets/{id}/files/{hash} returns

[
  {
    "hash": "<hash>",
    "token": "<token>",
    "operation": "PULL",
    "channel": "ws://162.243.48.250:4000"
  }
]

Documentation is not updated:
http://storj.github.io/metadisk-api/#!/buckets/get_buckets_id_files_hash

I didn't check the other API calls.

When I am trying to save data (with PUT) on https://api.metadisk.org with Metadisk API I got error from Success callback that says error = "Shard data not found" and the file is not being stored....
I am using Swift with Alamofire and creating a Bucket worked fine...

For large file pointers and assumedly large files, MetaDisk should account for the amount of time it will take a user to download chunks and reassemble them when signing message nonces in the file pointer response.

Currently files are chunked at 8mb, but we should use the Content-Length header to determine the appropriate chunk size for a given upload

Necessary for production, the simple contract and audit managers implemented in node-storj are not suitable.

PULL token ( get buckets/{id}/files/{hash} ) and PUSH token ( put /buckets/{id}/files ) can be used more than once.

One open question: Should the token expire in any case (for example failed upload because of wrong filesize)?

Key already exists:
curl -u user:password -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{ "key": "{key}" }' 'https://api.metadisk.org/keys'
{"error":"E11000 duplicate key error collection: metadisk-api.publickeys index: _id_ dup key: { : \"{key}\" }"}

Unit test create an user with a public key:
https://github.com/littleskunk/bridge/blob/master/test/pubkey.integration.js#L18

Later I use the same key to create a bucket:
https://github.com/littleskunk/bridge/blob/master/test/pubkey.integration.js#L66

I would expect an error message because of duplicate pubkey.

The current documentation and the metadisk-client code both use a unix timestamp as the nonce:

params.__nonce = Date.now();

This means the server only has to check if that nonce has been used in the last n minutes, because it already knows that older timestamps are invalid, but this approach can be disastrous for multi-threaded clients, or even two different clients trying to make requests at the same time. In short, a timestamp is not reliably unique enough to be used as a nonce.

Using a UUID instead of a timestamp would provide reliable uniqueness, but with this approach the server must store a massive amount of information (all the UUIDs ever used to make a request to the API) in order to validate nonces.

Using both a UUID and a timestamp would provide reliable uniqueness and constraints on how much storage is required for validation:

Let's say you decide to allow up to 15 minutes time difference between your clock and the client's and are keeping track of the nonce values in a database table. The unique key for the table is going to be a combination of 'client identifier', 'access token', 'nonce', and 'timestamp'. When a new request comes in, check that the timestamp is within 15 minutes of your clock then lookup that combination in your table. If found, reject the call, otherwise add that to your table and return the requested resource. Every time you add a new nonce to the table, delete any record for that 'client identifier' and 'access token' combination with timestamp older than 15 minutes.

http://stackoverflow.com/a/6876907/1784296

A timestamp in the logs would be helpful for parsing.

Proposed Implementation

The current upload design will not scale and leaves bridge as a bottleneck for distributing files in the Storj network. We can rework the file upload so that bridge negotiates the shard contracts and authorizes data channels with the farmers, but returns that information to clients so they can upload directly.

Upload Flow

1. Client wishes to store a file and starts by issuing a:
   • POST /frames
   • This creates a staging area for a file
   • Returns a unique ID for the frame (staging area)
2. Client performs encryption/sharding and iterates for each encrypted shard:
   • PUT /frames/:frame_id + including metadata about shard
   • Bridge negotiates a contract and returns a datachannel consign token for the shard
   • Client opens datachannel with farmer and consigns the shard
3. Client is finished filling the frame and wishes to create a file pointer in bucket:
   • POST /buckets/:bucket_id/files + including the frame ID
   • Bridge uses the current snapshot of the frame to create a file pointer and bucket entry
   • Bridge responds with the identifier (file hash) for the file in the bucket

Control Flow

1. Client wishes to append more data to the frame and create a new file pointer
   • Follow upload flow steps 2 and 3 again
2. Client is finished with the frame and closes it by issuing a:
   • DELETE /frames/:frame_id
3. Client wishes to see all open frames by issuing a:
   • GET /frames
4. Client wishes to check the status of an individual frame:
   • GET /frames/:frame_id

Tome, when you have some time, will you think about the initial successful sign up landing and how to step the user through bucket creation, pairing their application, and installing client tools/API docs depending on the developer's platform?

I can join a call with you next week to share some ideas.

Currently if MetaDisk restarts, it will have no peers until contacted by others.

Requesting an optional label field in storj/models/pubkey.js, so keys may be managed by alias'.

• MIME type of file (derived from Content-Type header on upload)
• File size
• Optional file name

https://github.com/Storj/metadisk-api/blob/master/doc/auth.md#http-basic

This should work but i get invalid password response:
curl -u user:SHA256 -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' 'https://api.metadisk.org/keys'

If i use my clear type password it is working (I am getting an other error message because authentication was successfull but the api call not):
curl -u user:password -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' 'https://api.metadisk.org/keys'

Hi,

As it was working a week ago, I think it a server side issue:

Getting token :

/usr/bin/curl -v -s -S --basic --user 'xxx:xxx' -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --data '{"operation":"PUSH"}' https://api.storj.io/buckets/57160eac54b452990bec7450/tokens
*   Trying 162.243.72.56...
* Connected to api.storj.io (162.243.72.56) port 443 (#0)
* found 187 certificates in /etc/ssl/certs/ca-certificates.crt
* found 748 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*    server certificate verification OK
*    server certificate status verification SKIPPED
*    common name: api.storj.io (matched)
*    server certificate expiration date OK
*    server certificate activation date OK
*    certificate public key: RSA
*    certificate version: #3
*    subject: CN=api.storj.io
*    start date: Wed, 06 Apr 2016 15:39:00 GMT
*    expire date: Tue, 05 Jul 2016 15:39:00 GMT
*    issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3
*    compression: NULL
* ALPN, server did not agree to a protocol
* Server auth using Basic with user 'xxx'
> POST /buckets/57160eac54b452990bec7450/tokens HTTP/1.1
> Host: api.storj.io
> Authorization: Basic cmlja2Vubnktc2pjeHRlc3RAdm90ZXotY3RodWxodS5uZXQ6NWJmYTczODU3MjI0NThkYzRlMDNhN2Y4YzM1YWJmZmJmZWJlZjQwMDliYzkzM2M3NTYzMWJlNDJiM2ViNTE3ZA==
> User-Agent: curl/7.43.0
> Content-Type: application/json
> Accept: application/json
> Content-Length: 20
> 
* upload completely sent off: 20 out of 20 bytes
< HTTP/1.1 200 OK
< Server: nginx/1.6.2
< Date: Tue, 19 Apr 2016 11:25:24 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 240
< Connection: keep-alive
< X-Powered-By: Express
< Vary: Origin
< Access-Control-Allow-Credentials: true
< ETag: W/"f0-z1Z6Gmwn6vxKW67tc+6kog"
< 
* Connection #0 to host api.storj.io left intact
{"bucket":"57160eac54b452990bec7450","operation":"PUSH","expires":"2016-04-19T11:30:24.706Z","token":"ca1f7de6a6384681454fcbba1686d9300c0aa18f12a45297314bfc32948ef751","id":"ca1f7de6a6384681454fcbba1686d9300c0aa18f12a45297314bfc32948ef751"}

And then trying to upload a file :

/usr/bin/curl -v -s -S --basic --user 'xxx:xxx' -X PUT --header 'Content-Type: multipart/form-data' --header 'Accept: application/json' --header 'x-token: ca1f7de6a6384681454fcbba1686d9300c0aa18f12a45297314bfc32948ef751' --header 'x-filesize: 7' --form "file=@sample/wilcard/coucou.txt" https://api.storj.io/buckets/57160eac54b452990bec7450/files
*   Trying 162.243.72.56...
* Connected to api.storj.io (162.243.72.56) port 443 (#0)
* found 187 certificates in /etc/ssl/certs/ca-certificates.crt
* found 748 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*    server certificate verification OK
*    server certificate status verification SKIPPED
*    common name: api.storj.io (matched)
*    server certificate expiration date OK
*    server certificate activation date OK
*    certificate public key: RSA
*    certificate version: #3
*    subject: CN=api.storj.io
*    start date: Wed, 06 Apr 2016 15:39:00 GMT
*    expire date: Tue, 05 Jul 2016 15:39:00 GMT
*    issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3
*    compression: NULL
* ALPN, server did not agree to a protocol
* Server auth using Basic with user 'xxx'
> PUT /buckets/57160eac54b452990bec7450/files HTTP/1.1
> Host: api.storj.io
> Authorization: Basic cmlja2Vubnktc2pjeHRlc3RAdm90ZXotY3RodWxodS5uZXQ6NWJmYTczODU3MjI0NThkYzRlMDNhN2Y4YzM1YWJmZmJmZWJlZjQwMDliYzkzM2M3NTYzMWJlNDJiM2ViNTE3ZA==
> User-Agent: curl/7.43.0
> Accept: application/json
> x-token: ca1f7de6a6384681454fcbba1686d9300c0aa18f12a45297314bfc32948ef751
> x-filesize: 7
> Content-Length: 195
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=------------------------a0c4fe990d73c249
> 
< HTTP/1.1 100 Continue
< HTTP/1.1 502 Bad Gateway
< Server: nginx/1.6.2
< Date: Tue, 19 Apr 2016 11:27:42 GMT
< Content-Type: text/html
< Content-Length: 172
< Connection: keep-alive
* HTTP error before end of send, stop sending
< 
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.6.2</center>
</body>
</html>
* Closing connection 0

Same with "https://api.metadisk.org" except for the error:

 <html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.6.2</center>
</body>
</html>

If someone can fix it ...

1. Same key can be registered twice.
   { "key": "04:38:74:de:22:53:6d:ec:c5:50:82:57:cc:80:6a:9e:5a:f5:e8:be:6a:80:05:68:43:d5:c0:c2:b1:12:90:34:30:f9:a4:6c:12:8c:a1:7e:30:e2:fb:54:f5:41:41:61:85:dd:a2:df:87:8a:db:b9:0d:66:81:14:52:f4:16:21:25" }
{ "key": "043874de22536decc5508257cc806a9e5af5e8be6a80056843d5c0c2b112903430f9a46c128ca17e30e2fb54f541416185dda2df878adbb90d66811452f4162125" }
2. First key has to be register by using HTTP basic auth. Optional public key for the registration would be better.
3. Same signature can be used more than once. In addition to the parameters required for each individual request, you must also include a __nonce parameter. This value should be an integer and must be incremented with every request. I did not increment the value.

• buckets
• pubkeys
• users

<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)>

Environment Python 3.4 on Windows.

Will update if I find a work around.