webtorrent / webtorrent Goto Github PK

chrome-net and chrome-dgram use chrome.socket. Starting with Chrome 33, this API is deprecated in favor of the sockets.udp, sockets.tcp and sockets.tcpServer APIs.

http://developer.chrome.com/apps/socket

Update these packages to use the new api. Looks like it has some nice additions like being able to see how many sockets the app has open to catch leaks.

getting this

0 info it worked if it ends with ok
1 verbose cli [ '/usr/bin/node', '/usr/bin/npm', 'start' ]
2 info using [email protected]
3 info using [email protected]
4 verbose node symlink /usr/bin/node
5 verbose run-script [ 'prestart', 'start', 'poststart' ]
6 info prestart [email protected]
7 info start [email protected]
8 verbose unsafe-perm in lifecycle true
9 info [email protected] Failed to exec start script
10 error [email protected] start: ./bin/cmd.js
10 error Exit status 8
11 error Failed at the [email protected] start script.
11 error This is most likely a problem with the webtorrent package,
11 error not with npm itself.
11 error Tell the author that this fails on your system:
11 error ./bin/cmd.js
11 error You can get their info via:
11 error npm owner ls webtorrent
11 error There is likely additional logging output above.
12 error System Linux 3.11.0-12-generic
13 error command "/usr/bin/node" "/usr/bin/npm" "start"
14 error cwd /home/web/webtorrent
15 error node -v v0.10.26
16 error npm -v 1.4.3
17 error code ELIFECYCLE
18 verbose exit [ 1, true ]

Let me try to clear up a little confusion about what WebTorrent is and how it works.

The goal of the project is to build a browser BitTorrent client that requires no install (no plugin/extension/etc.) and interoperates with the regular BitTorrent network, though it uses WebRTC Data Channels for peer-to-peer transport.

To accomplish that goal, I have two objectives.

The first is to develop a working JavaScript BitTorrent client as a Chrome App. This is now finished, though we still need to add support for additional features and polish to bring it on par with other modern clients. I did this first to learn how existing clients work and because lots of this code will be reused by the WebRTC client.

The second objective is to make the BitTorrent protocol work over WebRTC. The protocol obviously needs some modifications, but all the main concepts will remain intact. This new protocol will be called "the WebTorrent protocol", or "WebTorrent extensions".

We'll use this new protocol to ship a standalone JS file that webmasters can add to their site if they want to fetch files over webtorrent. All they need to provide is the info hash (or a .torrent file) (or a magnet uri) and the WebTorrent script will do the rest. We'll have a good default "viewer" UI depending on the file type: PDF viewer, image viewer, audio/video tag (if the format allows for streaming), etc. and also allow the programmer to get the data programmatically without UI too.

This standalone JS file will eventually be the primary WebTorrent client that most users interact with. Eventually, I hope that popular tracker sites will add it to their site to make consuming torrent content easier.

So, why the Chrome App?

We'll add support for the WebTorrent protocol to the Chrome App so it will speak both protocols, meaning it can connect to both normal and web-based clients. This accomplishes two things:

• With a reasonably large install base using the Chrome App, we'll have lots of users to test out the WebTorrent protocol with. We can fine-tune it on real users before we actually ship anything or make any promises about performance. The Chrome App will simply update with better and better WebTorrent support, and they won't have to do a thing.
• The first time a Chrome App client downloads a torrent that no other WebTorrent client is seeding, they become the first WebTorrent seeder. They essentially "bring" the file into the WebTorrent network.

Remember, until mainstream BitTorrent clients support WebTorrent, the web-based clients using the standalone JS file can only download from from other web-based clients. Browsers can't open arbitrary tcp/udp connections -- just WebRTC connections. So, essentially, we're bootstrapping the WebTorrent network by "bridging" it with the existing BitTorrent network.

Eventually, mainstream BitTorrent clients could add support for WebTorrent by using libjingle or other WebRTC bindings (see node-webrtc) so that WebTorrent clients can directly connect to them.

Questions/feedback welcome.

UPDATE: Now planning to use node-webkit to release native binaries for all major platforms instead of a chrome app, due to excessive chrome app CSP restrictions which make development not fun.

While a torrent is downloading, stream it to VLC, ala peerflix.

• Add an http server (see http-alt for js implementation by substack) since the chrome app apis don't come with one, but they do support tcp.
• Support range requests
• Support seeking & re-prioritize pieces to fetch
• Auto-launch the system's VLC

I have no immediate plans to implement this, but adding an issue here for when I get around to it.

• Use piece size recommendations from here: http://wiki.depthstrike.com/index.php/EAD:Utilities:TorrentBuild

Is anyone aware of how Cinefi was performing plugin-less torrent streaming before they shut down? It seems they may already have a proprietary version of webtorrent working, and their email is [email protected].

The UI currently lags (it's affecting the CSS animations) and I think it's due to SHA1 hashing happening on the main thread. Stick it in a web worker pool.

There's some advice at the end of this BEP http://bittorrent.org/beps/bep_0003.html

We already have auto-reconnect for peers that we connect to, but let's do it for incoming peers too. First we need to discover their listening port -- I think there's a extended message that sends this.

Implement peer exchange protocol for more efficient peer discovery and facilitating healthier swarms.

There is sadly no standardized PEX protocol, but rather 3 competing implementations, This paper, specifically section 2.1.4.3, gives a solid overview of the competing protocols and notes that ut_pex is by far the most popular. It's also the one I've seen supported empirically the most, so let's disregard AZ_PEX and BitComet's PEX for webtorrent.

Resources:

• ut_pex spec (aka LTEP)
• Paper analyzing PEX impact on bittorrent

• Should work over any stream interface
• Use bops
• Use peer-wire-protocol module as a guide

Right now, we don't support metadata fetched via BEP 9 if it's larger than one metadata block size (16KB). Need to fix this ASAP.

Don't open WebRTC connection to the IP's present in block list

How to test this ? I have a seedbox with several fullHD videoclips and want to make in playable in pure browser.

We currently use a hacky solution adapted from http://github.com/bmcmahen/bittorrent-dht

Consider learning from:

• https://github.com/stbuehler/node-dht
• https://github.com/nikhilm/kademlia

Specs here:

• BitTorrent spec: http://www.bittorrent.org/beps/bep_0005.html
• Kademlia paper: http://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf

The bittorrent extension protocol (BEP10) allows clients to innovate and come up with their own extensions to the protocol. A few extensions, like torrent metadata (BEP9), DHT (BEP5), and the PORT message are widely supported. WebTorrent already has basic support for these.

However, there are probably other extensions that are de facto mandatory in modern clients. Once nearly all clients support a feature, I'd guess that clients without the feature are at risk of being ignored, or at a speed disadvantage. Which extensions fit this description?

If you're familiar with bittorrent, please feel free to chime in. I'd love more help from people who know about bittorrent!

https://github.com/shama/workerify is a nice way to write modular code that uses web workers

This is just an idea for something to implement eventually. Got the idea from:

The ipMagnet tool allows BitTorrent users to download a magnet link which they can then load into their BitTorrent client. When the magnet link connects to the tracker, the user’s IP-address will be displayed on the site, alongside a time-stamp and the torrent client version.

http://torrentfreak.com/vpn-proxy-working-check-torrent-ip-address-140419/

WebTorrent needs a website. Something simple and clean like the NodeOS site.

• What is webtorrent
• What are we building
• How to try it out, getting started
• Subscribe for updates (email list)
• Blog (for project updates)
• Documentation for all the webtorrent node modules

Is project have any working chrome apps for now?

It starts with this command returning exit status 127:

stylus --use nib css/main.styl --compress --out chrome/ && ./bin/post-build.js

Then it's all downhill from there.

I haven't yet gotten a chance to dig deeper

Since we're using browserify and we've used the node fs, dgram and net apis with browser-specific shims, webtorrent works in node out-of-the-box. Specifically, the bittorrent client works in node. The webtorrent client will take a bit more work.

We should add an executable so users can do:

$ npm install -g webtorrent
$ webtorrent /path/to/torrent

or

$ webtorrent magnet:?xt=urn:btih:d2474e86c95b19b8bcfdb92bc12c9d44667cfa36&dn=Leaves+of+Grass+by+Walt+Whitman.epub

Trackers can be specified in a magnet URI or in a .torrent file.

I'm interested.
my skills about js and how can i contributing ?
Could you please more describe about your idea, working plan and what you need ?
thank you

Hi,
I'm trying webtorrent cli on OpenBSD and it does not work. Any idea?
Thx!

$ node bin/cmd.js 'http://www.bt-chat.com/download1.php?id=124137&type=torrent' --vlc     

/home/jirib/tmp/webtorrent/index.js:53
      res.pipe(concat(function (torrent) {
               ^
ReferenceError: concat is not defined
    at ClientRequest.<anonymous> (/home/jirib/tmp/webtorrent/index.js:53:16)
    at ClientRequest.g (events.js:180:16)
    at ClientRequest.EventEmitter.emit (events.js:95:17)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (http.js:1688:21)
    at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:121:23)
    at Socket.socketOnData [as ondata] (http.js:1583:20)
    at TCP.onread (net.js:525:27)

jirib:/home/jirib/tmp/webtorrent
$ node bin/cmd.js -v                                                                  
0.2.6
$ node -v                                                                                                                                                            
v0.10.24
$ sysctl kern.version                                                                                                                                                
kern.version=OpenBSD 5.5-current (GENERIC.MP) #126: Mon May 12 22:40:04 MDT 2014
    [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP

I've only tested this on Mac OS - this may not occur in Windows. I suspect the window is just flowing slightly off the screen or perhaps the "hit area" around the border of the window is too small. Essentially, if you maximise WebTorrent, you can only resize the window using the left and right sides of the frame. You must move the window in order to resize from the top, bottom, or any of the four corners.

Hello - awesome project - awesome work by you - awesome speech in Budapest. But when I install under ubuntu 14.04 64 bit, npm start gives me:

npm ERR! weird error 127
npm WARN This failure might be due to the use of legacy binary "node"

on ubuntu, node env var is nodejs but i cant seem to figure out what/how to edit

~/webtorrent/bin/cmd.js has the env var node at the top but changing that to nodejs just brings up Webtorrent help screen in terminal

Would love to make it work - Thx

Now that webtorrent is fs-backed, when a torrent is added, we should check for existing files in the destination directory, verifying piece hashes to properly initialize the storage state, and only then proceed with the download. If the torrent already exists and verifies as completed, no downloading should take place, just seeding.

webtorrent.app will take care of persisting torrent metadata across runs, along with known peers.

By default, browserify's streams implementation uses a fake Buffer object from browserify-buffer that behaves weirdly. We really want it to use bops so it uses the environment's native buffer type.

I think @defunctzombie's approach will work:

  // defunctzombie: https://github.com/alexgorbatchev/node-browser-builtins/tree/master/builtin
  // defunctzombie: take the stream files
  // defunctzombie: stream.js
  // defunctzombie: and the __stream shit
  // defunctzombie: put them all into a repo
  // defunctzombie: maybe called bops-stream I dunno
  // defunctzombie: and update the uses of Buffer('foobar') or whatever to use bops
  // defunctzombie: if the bops was API compatible you could just change require('buffer') to require('bops');
  // feross: right
  // defunctzombie: then in your module just require('bops-stream'); and you will be set
  // feross: and use it in place of require('stream')?
  // defunctzombie: this will be nice since others will be able to use your module too
  // defunctzombie: yep

I'm not clear on how exactly the WebRTC version of webtorrent will interface with the existing browserifiable libraries bittorrent-client/swarm/protocol.

I think progress towards the actual web portion of webtorrent would benefit greatly from a detailed breakdown of the plan to build the webrtc version based on what exists today, including:

• what modules need to be built and their anticipated dependencies
• how the actual peer connection / communication will be handled over webrtc
• the expected browserify process
• will any existing webrtc modules be used for connecting the hybrid bittorrent/webtorrent clients with the webtorrent clients? e.g., wrtc, webrtc, or webrtc.io?

Thanks! 👍

Lets discuss the potential of DHT support. WebRTC has some pretty serious restrictions on data transferred and a bit of de-risking should be taken when planning.

How would the DHT features be delivered without UDP or some sort of hand-rolled proprietary, routing outside of the protocol spec?

Hi @feross,

I just bumped into webtorrent and i'm astonished: I really think you are on top of something big!

I'm co-author (with @jinroh) of KadOH, a (dying?) project that has much in common with webtorrent:

• KadOH was initially a student project started in 2011 around the subject "data privacy protection on mobile" that turned into an implementation of the Kademlia algorithm with web technologies – you can read more at http://jinroh.github.io/kadoh. As demonstration, it later ended up in a implementation of a decentralized twitter-like application built on KadOH.
• to do browser-to-browser communication, we relied mainly on XMPP since, at that time, WebRTC was still a draft initiative. However, it looked like a promising techno that we never took time to experiment later..
• when run in nodejs, KadOH was able to connect and interact with Mainline (the DHT behind bittorrent). We wanted to do that in browser (thru chrome socket API): we had issues with buffer-browserify+bncode, we never investigated further. But it looks like you fixed a lots of stuff!
• we also wanted to write a complete implementation of a bittrorrent client for Node built on top of KadOH. And maybe, make it run on browser too!

Now, it's been a while since last time we did a true commit to KadOH.. One last initiative I wanted to achieve was to split our implementation into several reusable modules (for instance, eventually is our implementation of ~promises) but I never went further.

However, I think there is probably some parts of KadOH that could interesting to pickup to continue building webtorrent. We got for instance an implementation of the Kademlia routing table (with its iterative-lookup algorithm) that webtorrent might be interested in (#11?).

I would be very happy and proud to discuss it and help you building webtorrent by integrating pieces of KadOH!

– Alex

Transmission's UI is beautifully elegant. Let's start with something like that, though we can do much better for the streaming video use case.

Swarmplayer (http://swarmplayer.p2p-next.org/) can play live streams directly in browser, but it needs to be installed as a plugin. Could you add such a thing into Webtorrent ? Swarmplayer is opensource.

I know it's alpha, but here is an error I came across (tip of master).
A couple questions:
Re: Todo for basic bittorrent client as chrome app - Are you talking about a chrome extension? If so, why?
What kind of api do you have in mind for webtorrent-swarm? I assuming that's the bit that sorts out segmented file transfers.

$ npm start

> [email protected] start /home/cagedwisdom/webtorrent
> npm run build && ./bin/start.js

> [email protected] build /home/cagedwisdom/webtorrent
> npm run build-css && npm run build-js

> [email protected] build-css /home/cagedwisdom/webtorrent
> stylus --use nib css/main.styl --compress --out chrome/ && ./bin/post-build.js

  compiled chrome/main.css

> [email protected] build-js /home/cagedwisdom/webtorrent
> browserify --debug . > chrome/bundle.js


events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: spawn ENOENT
    at errnoException (child_process.js:988:11)
    at Process.ChildProcess._handle.onexit (child_process.js:779:34)
npm ERR! weird error 8
npm ERR! not ok code 0

BEP9: Extension for Peers to Send Metadata Files
http://www.bittorrent.org/beps/bep_0009.html

Is it possible to do something like this for torrent ? Similar thing exist for Usenet:
https://www.youtube.com/watch?v=nkEJeDXgCp8

I'm in. I've watched your presentation and this is a very good idea and would give people a good view of what webrtc is capable of.

Do you know any good source to start hacking into how a bittorrent server should work or any idea of how you would do it in the backend?

Thanks.

For the ubuntu 13.10 torrent, pieces are quite large (like 500KB or something) so the progress bar jumps by large increments (or sometimes makes no progress for a while). Use the Piece's blocksWritten as an estimate.

Hi

It looks like you are writing a BitTorrent client as a packaged app as well. Have you taken a look at:

• https://github.com/astro/bitford
• http://youtube.com/watch?v=-O3MZDAZMYk
  ?

Would you like to help out?

http://blog.vline.com/post/52644825765/tunneling-webrtc-over-tcp-and-why-it-matters

Currently working towards this here: https://github.com/feross/chrome-app-socket

Reasons to make it like dgram and net in node:

• re-usable for others via browserify
• familiar, understood semantics
• tcp implementation uses streams
• might be able to re-use existing node bittorrent modules that assume dgram and net api

Issue is here: https://github.com/feross/chrome-app-socket/issues/1

101 looks like an off by one.
And 417 total is just too much. Tell the DHT to slow down.