Using the node APIs and a tool such as https://oclif.io/ we could create a simple CLI exposing commands such as erebos bzz:upload ./this-folder and erebos pss:sendAsym "text message" --key=<peer public key> --topic=<hex value>

It would be very useful for static websites to support https://swarm-guide.readthedocs.io/en/latest/up-and-download.html#directory-with-default-entry so the user wouldn't need to access /index.html explicitly.

Feedlinks

In order to support the idea of realtime feeds and persistent pss, it is proposed that we gravitate towards an informal standard we have dubbed "feedlinks".

After discussing the use case with the broader Swarm community we have reached general consensus on the first iteration of the implementation.

The Proposed Solution

• The content of messages will be stored in Swarm in represented as JSON.
• The BZZ hash of the stored content will then be posted to Swarm Feeds.
• The content will include a pointer to the previous message hash creating a singly linked list allowing traversal independent of Swarm functionality.

The Proposed Protocol Format

  "protocol": "feedlinks",
  "version": 1.0,
  "id": "ff473a167f456e839186b9ef391d344b7b14fa13877fe2b0075863a4755f2e9f",
  "timestamp": 1545233446,
  "references": [
    "fg584a167f456e839186b9ef391d344b7b14fa13877fe2b0075863a4755f2e9g",
    "f9473a167f456e839186b9ef391d344b7b14fa13877fe2b0075863a9855f2e9f"
  ],
  "previous": "fb473a169f456e839186b9ef391d344b7b14fa13877fe2b0075863a4755f2e9a",
  "type": "application/json",
  "author": "0x2013dfc464e594318cd8353f695e1f2b61f4e824",
  "content": {
    "any": "random",
    "data": "type",
    "we": "support"
  }
}

Content Type

The content type stored in the content attribute will be indicated by the type attribute.

It is the responsibility of the higher level code (not the protocol implementation) to handle the encoding and decoding of content types.

Questions

What is id for if unable to retrieve the hash before upload?
Inclusion of the protocol attribute?

I have seen in the documentation that there are functions that get the privateKey as an argument for signing (for example https://erebos.js.org/docs/api-bzz#signfeeddigest).

It would be better from a security standpoint to not expose the privateKey to libraries that you are using, because there might be dependencies in the code that can potentially leak it by mistake or by malicious purposes.

It's possible to use for example a function that can be called when your data needs to be signed. You can pass that function instead of the privateKey, and when the underlying code needs to sign the data, it can simply call the function with the data as parameter.

It enables the library user to have more control over what happens when the app requires access to the privateKey, for example it can show a confirmation dialog (for example fingerprint check on mobile) or possibly can use an external wallet API .

What do you think?

Not sure if it is problem of Erebos or actually Swarm, but it is not possible to add nested directory structure using uploadDirectory method.

For example

const dir = {
      [`foo-${uploadContent}.txt`]: {
        data: `this is foo-${uploadContent}.txt`,
        contentType: 'plain/text',
      },
      [`bar-${uploadContent}.txt`]: {
        data: `this is bar-${uploadContent}.txt`,
        contentType: 'plain/text',
      },
      [`some/folder/bar-${uploadContent}.txt`]: { // <-------
        data: `this is bar-${uploadContent}.txt`,
        contentType: 'plain/text',
      },
    }

The file some/folder/bar-${uploadContent}.txt is not uploaded, or at least the list() does not return manifest that would either return the whole entry some/folder/bar-${uploadContent}.txt or at least some that would point to nested manifest.

Not sure what is the correct behavior or if it is a problem of Swarm, yet this is definitely unexpected behavior.

"@erebos/swarm-node@^0.5.2" is not distributed in npm, so can not install "@erebos/cli" via npm.
Check for errors

❯ npm install --global @erebos/cli
npm ERR! code ETARGET
npm ERR! notarget No matching version found for @erebos/swarm-node@^0.5.2
npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.
npm ERR! notarget
npm ERR! notarget It was specified as a dependency of '@erebos/cli'
npm ERR! notarget

:)

Hi,

There's currently no method in the bzz-api for updating a resource in an existing manifest at a certain path.

According to the docs this can be done by uploading a file to a url with the manifest and path locator options.

From the HTTP API docs:

Locator: bzz:/<manifest_hash?>/<resource_path?>/<encrypt?>

Locator Parts:

• manifest hash - optional - an existing manifest address to update a resource included in the manifest.
• resource path - optional - which resource to update in the manifest. encrypt - optional flag to enable encryption

I wrote this but don't have time to look into it further at this point. At first glance, it should work.

 updateManifest (
  hash: string,
  path: string,
  data: string | Buffer,
  options?: UploadOptions = {},
): Promise<hexValue> {

  const body = typeof data === 'string' ? Buffer.from(data) : data
    const raw = options.contentType == null

    if (options.headers == null) {
      options.headers = {}
    }
    options.headers['content-length'] = body.length
    if (
      options.headers != null &&
      options.headers['content-type'] == null &&
      !raw
    ) {
      options.headers['content-type'] = options.contentType
    }
    const url = this.getUploadURL({
      manifestHash: hash,
      path
    })

    return this._fetchTimeout(url, options, {method: 'POST' })

}

Could be configured at the instance level (default timeout for all requests) and with per-request option.

Ideally matching the node ones as much as possible, but supporting browser APIs such as Blob, FileReader, etc.

Documented in https://swarm-guide.readthedocs.io/en/latest/pss.html#sending-a-raw-message but not clear what the RPC API is.

ethersphere/swarm#1665

The current logic of adding the "default path" entry to a manifest creates the entry but fails to set the content-type for it.
There might be some additional headers needed to be set with tar-fs or tar-stream.

Description

If the gateway url does not contain a closing slash, the upload request will fail.

Example

const bzz = new BzzAPI({ url:https://swarm-gateways.net` });
bzz.upload('some text');
Fails with network error
const bzz = new BzzAPI({ url: https://swarm-gateways.net/ });
bzz.upload('some text');
`
Upload completes.

Expected behavior

The documentation should be specific about it (at the moment not even the examples contain the slash), or both cases should be handled.

Get feedback from other team members and/or interested third-parties before releasing.

I use @erebos/swarm-browser package in order to communicate with swarm in my app.

const client = new SwarmClient({ws: '...'});

When I restart swarm, erebos package throws an error Connection failed and it is not possible to continue work with app, in order to fix it I need to refresh the app.

Do you have any ideas how to implement automatic re-connection of SwarmClient

reproduce (using node v10.14.2):

1. git clone [email protected]:MainframeHQ/erebos.git
2. yarn install
3. yarn bootstrap
4. git status

It seems that flow was changed to version 0.100 from 0.97, hence the slight change in the generated output, but the files were not committed, or gitignored, not sure what's the intended behavior.

In reference to: https://erebos.js.org/docs/hex - it may seem that docs are wrong or misleading;

Using code:

import { createHex } from '@erebos/hex';

console.log(createHex('foo'));

fails with:

$ ts-node test.ts 

/usr/lib/node_modules/ts-node/src/index.ts:245
    return new TSError(diagnosticText, diagnosticCodes)
           ^
TSError: ⨯ Unable to compile TypeScript:
test.ts:1:10 - error TS2305: Module '"./node_modules/@erebos/hex/types"' has no exported member 'createHex'.

1 import { createHex } from '@erebos/hex';

If I do this instead it works:

import { createHex } from '@erebos/swarm';

console.log(createHex('foo'));

Just trying to import the module causes null reference exceptions:

justin@justin-server:~/code/test$ node -r babel-register .
/home/justin/code/test/node_modules/@erebos/api-bzz-base/cjs/index.js:76
    this._url = new URL(url).toString();
                    ^

ReferenceError: URL is not defined
    at new BaseBzz (/home/justin/code/test/node_modules/@erebos/api-bzz-base/cjs/index.js:76:21)
    at new Bzz (/home/justin/code/test/node_modules/@erebos/api-bzz-node/lib/index.js:24:5)
    at new NodeClient (/home/justin/code/test/node_modules/@erebos/swarm-node/lib/Client.js:60:23)
    at Object.<anonymous> (/home/justin/code/test/index.js:4:16)
    at Module._compile (module.js:635:30)
    at loader (/home/justin/code/test/node_modules/babel-register/lib/node.js:144:5)
    at Object.require.extensions.(anonymous function) [as .js] (/home/justin/code/test/node_modules/babel-register/lib/node.js:154:7)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
    at Function.Module._load (module.js:489:3)

Also I had to manually install rxjs, that appears to be a dependency which isn't actually listed in the project.json file so npm doesn't install it automatically.

The code in index.js:

import { SwarmClient } from '@erebos/swarm-node'

const client = new SwarmClient({ bzz: 'http://localhost:30399' })

client.bzz
  .upload('Hello world!', { contentType: 'text/plain' })
  .then(hash => console.log('hash:', hash) || client.bzz.download(hash))
  .then(res => res.text())
  .then(text => {
    console.log(text) // "Hello world!"
  })

https://erebos.js.org/docs/examples-feeds

System information

OS & Version:Linux 4.19.4-1-MANJARO
Node Version: v11.1.0
Erebos Version: v0.5.1
Swarm Version: v0.3.6 & v0.3.7

Expected behaviour
Fire up two local Swarm nodes (on different ports) and successfully run the PSS example code found here: https://erebos.js.org/docs/examples#communication-between-two-nodes

Actual behaviour
Instantiating a second SwarmClient object with PSS endpoint fails with an ENOENT error:

events.js:167
      throw er; // Unhandled 'error' event
      ^

Error: connect ENOENT ws://localhost:8602
    at PipeConnectWrap.afterConnect [as oncomplete] (net.js:1117:14)
Emitted 'error' event at:
    at emitErrorNT (internal/streams/destroy.js:82:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)
    at process.internalTickCallback (internal/process/next_tick.js:72:19)

It does not seem to be a problem with the nodes themselves. For instance running nolash's pss-js example appears to function as expected.

Steps to reproduce the behaviour

• Spin up two Swarm nodes locally, both with websocket ports exposed*.
• Run the PSS example code found in the Erebos documentation (with ports changed to match respective nodes).
• Observe connect ENOENT error when creating second SwarmClient object.

Switching which node is connected to first doesn't change this behaviour. Instantiating the first SwarmClient object is always successful and the second always fails.

*I have tried spinning up nodes both with swarm-test-cluster (Swarm v0.3.7) and docker image built from this repository (Swarm v0.3.6). The nodes themselves in both cases appear healthy and functioning as expected.

Edit:
A simple little example demonstrating that the error still occurs no matter which node is first:

import { SwarmClient } from '@erebos/swarm-node'

const shuffle = arr => arr.sort(() => Math.random() - 0.5)

const run = async () => {
  let wsPorts = [8601, 8602]
  shuffle(wsPorts)
  console.log(wsPorts)

  const alice = new SwarmClient({ pss: `ws://localhost:${wsPorts[0]}` })

  // Display Alice's public key
  await alice.pss.getPublicKey()
    .then(aliceKey => {
      console.log(`Alice: ${aliceKey}`)
    })

  // This will always error out (ENOENT), no matter which port
  const bob = new SwarmClient({ pss: `ws://localhost:${wsPorts[1]}` })

  // Display Bob's public key
  await bob.pss.getPublicKey()
    .then(bobKey => {
      console.log(`Bob: ${bobKey}`)
    })
}

run().catch(console.error)

The following code fails to run:

import { createKeyPair, sign } from '@erebos/secp256k1'
import { createHex, BzzAPI  } from '@erebos/swarm';
import { pubKeyToAddress } from '@erebos/keccak256'

var keyPairTmp = createKeyPair();
var keyTmpPub = keyPairTmp.getPublic("hex");
var userTmp = pubKeyToAddress(createHex("0x" + keyTmpPub));
var signerTmp = async bytes => sign(bytes, keyPairTmp.getPrivate());
var bz = new BzzAPI({ url: 'http://localhost:8500', signerTmp });
var topic = '0x0000000000000000000000000000000000000000000000000000000000000001';
var feedOptions = {
	user: userTmp,
	topic: topic,
}

bz.upload("foo").then(function(h) {
	console.log("data uploaded to " + h);
	bz.setFeedContentHash(feedOptions, h).then(function(r) {
		console.log("ok " + r.text())
	}).catch(console.error);;
}).catch(console.error)

error:

$ ts-node sign.ts 
data uploaded to 2387e8e7d8a48c2a9339c97c1dc3461a9a7aa07e994c5cb8b38fd7c1b3e6ea48
Error: Missing `signBytes()` function
[...]

Change the signerTmp name to signBytes as follows:

[...]
var signerTmp = async bytes => sign(bytes, keyPairTmp.getPrivate());
var signBytes = signerTmp;
var bz = new BzzAPI({ url: 'http://localhost:8500', signBytes});
[...]

... and it runs without error

From gitter with some additions:

I ran into something weird using swarm CLI vs Erebos for uploading webapps to the swarm using the defaultPath option.

Uploaded with Erebos-JS: http://knuckle-swarm.designisdead.com/bzz:/e5cedbf7fc9a5dd92689d48b309ee2554e7c61591ecf319aa2b35bacb11cfb01/#/
Uploaded with Erebos- CLI : http://knuckle-swarm.designisdead.com/bzz:/818e3913b78905f3a237be58dd8a551d8baf0f77c163e14428587a842c15605f/#/

As you can see it seems like some css is missing (background is different) and some tag for icon is partially displayed in the top corner. This only happens when using my Erebos-js script to upload, not when using Erebos-CLI or using a development server/local file system. I checked the built files and they are fine.

Erebos:CLI: erebos bzz:upload .dist/pwa-mat --http-gateway http://knuckle-swarm.designisdead.com --default-path index.html

Erebos-JS :

const path = require('path')
const open = require('open')
const SwarmClient = require('@erebos/swarm').SwarmClient
const host = process.env.SWARM_HOST ||'http://knuckle-swarm.designisdead.com'
const clientType = process.env.CLIENT_TYPE || 'pwa-mat'

const bzz = new SwarmClient({bzz: host}).bzz
console.log('\x1b[33m%s\x1b[0m', `Bzz client started at ${host}`)

const directoryData = path.join(__dirname, 'dist', clientType)
console.log('\x1b[36m%s\x1b[0m', `Uploading ${clientType} from following path: ${directoryData}`)

const uploadOptions = {
  defaultPath: 'index.html'
}
console.log('\x1b[36m%s\x1b[0m', `Setting default path to resolve:  ${uploadOptions.defaultPath}`)

bzz.uploadDirectoryFrom(directoryData, uploadOptions).then(hash => open(`${host}/bzz:/${hash}`)).catch(e => console.log(e))

CLI renders fine:

Uploaded with the JS lib:

version:
"@erebos/swarm": "^0.4.0",

reproduction:
There is no access control on the nodes so you can just download the build files for now
erebos bzz:download 818e3913b78905f3a237be58dd8a551d8baf0f77c163e14428587a842c15605f --http-gateway http://knuckle-swarm.designisdead.com --path=./download-reproduction-content

Description

I'm developing an application where I'd like to use erebos. There are some issues which I'd like to solve in the library, so I check out the repo to make changes to it. I place a symlink in my project's node_modules folder which points to the specific erebos package (api-bzz-browser) I need.

Disclosure: this is a react-native package which uses the metro bundler and it has an open issue about not being able to handle symlinks facebook/metro#1. I'm using a workaround where I specifically tell it where to look for source files, which solves this issue.

Issues

• doing yarn install && yarn bootstrap && yarn build does not include all the dependencies in the node_modules folder for api-bzz-browser. It seems like there's some kind of hoisting going on, but not by lerna(?) because I don't see the flag for it.
  workaround: I install and build in the specific package manually.
• it seems like dependencies are not always listed in the leaf packages' package.json where they are used, but they pick it up nonetheless from the root node_modules. Example: rxjs in api-bzz-base.

I'm still new to Lerna, and not sure what else influences the build, but it would be nice to have documentation regarding how to set up local development of erebos as a dependency. It's also possible that these issues only arise when it's being consumed by react-native / metro bundler.

Something like puppeteer might be a good option to start with, assuming we can run it in CircleCI.

Feedlinks

In order to support the idea of realtime feeds and persistent pss, it is proposed that we gravitate towards an informal standard we have dubbed "feedlinks".

After discussing the use case with the broader Swarm community we have reached general consensus on the first iteration of the implementation.

The Proposed Solution

• The content of messages will be stored in Swarm in represented as JSON.
• The BZZ hash of the stored content will then be posted to Swarm Feeds.
• The content will include a pointer to the previous message hash creating a singly linked list allowing traversal independent of Swarm functionality.

The Proposed Protocol Format

{
  "protocol": "feedlinks",
  "version": 1,
  "id": "ff473a167f456e839186b9ef391d344b7b14fa13877fe2b0075863a4755f2e9f",
  "timestamp": 1545233446,
  "references": [
    "fg584a167f456e839186b9ef391d344b7b14fa13877fe2b0075863a4755f2e9g",
    "f9473a167f456e839186b9ef391d344b7b14fa13877fe2b0075863a9855f2e9f"
  ],
  "previous": "fb473a169f456e839186b9ef391d344b7b14fa13877fe2b0075863a4755f2e9a",
  "type": "application/json",
  "author": "0x2013dfc464e594318cd8353f695e1f2b61f4e824",
  "content": {
    "any": "random",
    "data": "type",
    "we": "support"
  }
}

Content Type

The content type stored in the content attribute will be indicated by the type attribute.

It is the responsibility of the higher level code (not the protocol implementation) to handle the encoding and decoding of content types.

Questions

What is id for if unable to retrieve the hash before upload?
Inclusion of the protocol attribute?

Following #31 I realized it's not just the HTTP endpoint that are affected but also the WebSocket and IPC ones, so I think the Ethereum and Swarm clients could be completely split.

https://docusaurus.io/ should be good starting point.

I added #19 to the v0.5 milestone but I think the following features could also be useful to add in this release:

• Access control (#64) - I discussed with Daniel last week and from my understanding there is no HTTP API to add a password to a resource, so I think at first our support would be only on the reading side, adding the option to provide a password to retrieve a resource.
• Raw PSS messages (#63) - Louis told me the RPC API should be implemented already so that should be easy to add as an option.

Also nice to have:

• PSS support in CLI (#47) - Currently there is no easy way to interact with PSS even from the Swarm CLI so that might be useful, even if the first implementation is very basic.
• Eth APIs support (#21) - Milos started adding tests for it so it might be reliable soon - essentially we could start replacing web3.js for some use cases.
• High-level "website" support in CLI (#69) - this seems to be a very common use case for Swarm.

@mosic @dleonard00 what do you think please?
Should we ship v0.5 as soon as possible with basic feeds support only and focus on the rest later, or provide a more full-feature release?

The Client constructor has some logic trying to be "smart" regarding what was provided, supporting APIs, RPC instances and plain strings.
The http option is confusing at the moment because it can be used to connect to the Swarm BZZ API, but potentially also an Ethereum HTTP gateway, so this should be refactored to be explicit either way.

https://swarm-guide.readthedocs.io/en/latest/usage.html#feeds
Would be great for things like comments.

Sorry, didn't notice it's already open. Ignore me :) !

At the moment the browser and Node.js versions of the Bzz APIs and therefore clients are not compatible with React Native.
The main issue is that these modules use browser-specific APIs such as Blob and FormData, or Node.js-specific ones such as streams and filesystem APIs. react-native-fs and rn-fetch-blob might provide good alternatives to implement the necessary APIs for React Native.

Better use https://github.com/ethersphere/swarm-docker directly if possible rather than having to maintain our own image.

Added in ethersphere/swarm#1959

Might need to implement a way to compute the chunk address based on the feed metadata, as done in https://github.com/ethersphere/swarm/blob/3cf1468c0f975f6411e3644b380c598dfdd05d69/storage/feed/id.go#L42

Part of this change, the FeedMetadata interface could be renamed to FeedID to match Swarm's nomenclature.

Adding the options to specify the request mode in the browser APIs for Bzz would allow to clients to ignore server CORS settings when needed.

Ideally for both BZZ and PSS, using the browser, node and CLI.

https://unpkg.com/@erebos/[email protected]/dist/erebos.development.js
Appears to be missing a lot of stuff including everything from https://erebos.js.org/docs/wallet-hd

Did you remove these features and not update the docs or did you recently add the features and not update the build? Or am I using the wrong version?

I would really like an HDWallet without the need to use webpack and import half the internet into my project. Erebos looked promising.

geth

geth --syncmode light

swarm

swarm --bzzaccount 4fe9account --corsdomain '*' --maxpeers 100 --ens-api /Users/username/Library/Ethereum/geth.ipc –-ws --wsorigins '*'

import { SwarmClient } from '@erebos/swarm-browser';

this.Client = new SwarmClient({ ws: 'ws://localhost:8546' });
console.log(this.Client);

const [publicKey, overlayAddress] = await Promise.all([
  this.Client.pss.getPublicKey(),
  this.Client.pss.baseAddr(),
]);

console.log(publicKey);
console.log(overlayAddress);

this isn't work. It does not work in any Browser.
However, if you run Onyx, the above code works.
What is the problem? Help me.

version
@erebos/cli/0.10.0 darwin-x64 node-v10.15.1
swarm version 0.5.3-unstable
geth 1.9.6-stable
code
wMacBook-Pro-282:testd wany$ erebos bzz:list da30bb001cc3a8f6537e81e92af3ccd67bbf3c0f51c8e256b75ba0c58ac21c61/master.m3u8
✔ Retrieving list for da30bb001cc3a8f6537e81e92af3ccd67bbf3c0f51c8e256b75ba0c58ac21c61/master.m3u8
{}
wMacBook-Pro-282:testd wany$ curl -s http://localhost:8500/bzz-list:/da30bb001cc3a8f6537e81e92af3ccd67bbf3c0f51c8e256b75ba0c58ac21c61/master.m3u8
{"entries":[{"hash":"7fe9360fb8f3a9f328fccacc3fd03ad9890ba87dbb75c5dab070eb022732d53a","path":"master.m3u8","contentType":"application/vnd.apple.mpegurl","mode":420,"size":117,"mod_time":"2019-10-16T14:53:52+08:00"}]}
summary
upload a directory, when I try to get list of a files in this directory.erebos return a null,but swarm api not yet.
what's the reason?
thx a lot

https://swarm-guide.readthedocs.io/en/latest/usage.html#feeds

The platform started using a few APIs to interact with the Ethereum blockchain.
It would be useful to be able to replace the dependency on web3.js with erebos for these calls.

Hi. Is it possible to add support for the progress in percents when uploading a file?
I'm using @erebos/swarm-browser in https://github.com/IgorShadurin/swarm-social-react. For the visual display of file uploading there is not enough progress callback function or something like that.

https://swarm-guide.readthedocs.io/en/latest/usage.html#access-control
Might need to enforce the security constraint of being connected to a local node.

ethersphere/swarm#1666

Leveraging feeds the CLI could provide a command that would implement the following flow:

1. Generate a key pair or use a provided private key (provided directly, from an env variable or file)
2. Create a feed manifest based on inputs (at least the website name or the manifest hash)
3. Upload folder with defaultpath set to index.html

So the command could be something like erebos website:upload ./dist --name=my-website --key-file=./key

Another website:create command could be used to create the key pair and write it to the file for example depending on the provided flags.

Would be nice to be able to send and receive messages from the CLI.