pmarks-net / dtella Goto Github PK

http://code.google.com/p/dtella-cambridge/source/browse/branches/adc

There are lots of additions, most notably ADC protocol support (with option
to support a hybrid network for backwards compatibility). Other ones include:
- IP-location matcher (see below)
- config system that can handle multiple networks/bridges running under
multiple users
- build system more integrated with distutils, so more extensible to other
OSes.


1.2.4.3 (2009-04-22):
see 
http://code.google.com/p/dtella-cambridge/source/browse/branches/adc/docs/change
log_adc.txt

1.2.4.5 (2009-10-01):
- several uninteresting bug fixes; see `git log`
- overhauled the build system to be more integrated with python's standard
build system (distutils).
  - forked half of local_config into build_config
  - bridge modules are installed by giving setup.py the --bridge option
- new state file format, using the shelve module
- new bridge/network config system, using ConfigParser
  - this allows a single installation of dtella to connect to multiple
networks and multiple bridges for each network
  - each installation has a "default" network config (and optionally a
"default" bridge config) which are used as templates for new configs
  - each config has its own logging/state files
  - also added code to migrate old data into the default network

1.2.6.1 (2010-04-18):
- simplify config system:
  - drop "user default" config (and just use system default)
  - have bridge config use same-named network config, instead of user default
- drop support for Python 2.4
- restructure windows build system
  - main worker scripts moved to installer_win/
  - frontend scripts in the main repo directory
- various debian / POSIX build system tweaks
  - get rid of POSIX !upgrade stuff, it was bloated and pointless
  - shell script installer now installs to ~/.local/share/dtella
- allow locations to be matched on IP; requires python-subnettree module
  - this is now the preferred way to infer locations; it is done on
    the local side and will override any remote claims about location
- new `debug dcfg` command to view dcfg

When a bridge config specifies bind_ip, it needs to make its way to 
listenUDP in dtella/common/core.py

Dtella should not echo valid commands back to the main chat. This is
especially confusing for newbies who may not realise that nobody else will
be able to read it, or Dtella's output.

Instead, Dtella should echo the command as if sent to *Dtella in a private
message, and output everything resulting from the command through private
message.

What steps will reproduce the problem?
1. Configure your nick to be "Aux"
2. Attempt to get the filelist or download a file from another user.

What is the expected output? What do you see instead?
Expected: The connection goes through and the download starts.
Observed: The connection times out.

What version of the product are you using? On what operating system?
1.2.6 and 1.2.7 on Windows and Linux

Please provide any additional information below.
Attached is a CDM Debug capture of two handshakes, one with the nick "Aux" and 
one with "test".  When the nick was "Aux", file connections timed out.  When 
the nick was "test", files could be downloaded.

Attached is a patch that adds local discovery of nodes using mDNS/Bonjour. 
It uses pybonjour and requires Bonjour be present, i.e by installing 
Bonjour standalone or by having iTunes or Safari present on the machine. 

There are two parts - a bonjour 'server' that invokes a mDNS registration 
and a puller which is called from start() in core to search for nodes on 
the local (same broadcast domain/subnet) network.

This is separate from the dconfig puller module.

So far I've tested on Mac OS X, Windows XP and Windows 7. They have been 
able to start from a clean install to network sync within 2 minutes 
without any DNS ipcache help. 

Known issues:
1. Bridge/dconfigpusher has not been tested. We don't use bridge, and 
won't need to with this :)
2. I haven't taken much care programming wise - this is a days worth of 
coding, so there are probably better places to bolt on the server and 
puller modules - also, puller will block during the initial search, but 
this isn't much of a problem
3. If !udp is used, dtella needs to be restarted before the service 
registration is changed.
4. We don't check if bonjour is installed or not

In addition to the patch below, the following needs to be added to 
local_config:

use_bonjour = True

Presently if a required version is set, then a user may simply override the
notice and continue using the older version.

This feature would force the client to re-override at a regular interval to
ensure they are more inclined to follow the required upgrade path.

On newer DC clients, an undocumented advancement to the Hub-Client protocol
is "$ConnectToMe <remote_nick> <myip>:<myport>S|" to signal that the two
clients should use self-signed TLS encrypted transmissions.

Dtella successfully passes the flags correctly, signifying that a client is
TLS capable.  However, the current implementation flags "<port>S" as a
malformed address.

I present herewith a patch that was made against current head revision
(556) that successfully transmits a CS packet.  It has been tested and
shown to work with the StrongDC 2.21 client.

Are there any issues with the patch?

~Andyhhp

Code:

http://code.google.com/p/cam-dc-dtella-patch/source/diff?old=10&r=23&format=unid
iff&path=%2Ftrunk%2Fdtella-cambridge-1.2.3%2Fdtella%2Fclient%2Fdc.py
http://code.google.com/p/cam-dc-dtella-patch/source/diff?old=10&r=23&format=unid
iff&path=%2Ftrunk%2Fdtella-cambridge-1.2.3%2Fdtella%2Fcommon%2Fcore.py
http://code.google.com/p/cam-dc-dtella-patch/source/diff?old=10&r=23&format=unid
iff&path=%2Ftrunk%2Fdtella-cambridge-1.2.3%2Fdtella%2Fcommon%2Fstate.py
http://code.google.com/p/cam-dc-dtella-patch/source/diff?old=10&r=23&format=unid
iff&path=%2Ftrunk%2Fdtella-cambridge-1.2.3%2Fdtella%2Flocal_config.py

Summary:

2 new packets:
WR - sync packet sent with every YR packet, which contains the whole "new
items" list
WN - broadcast packet sent every time someone announces a new item

3 new commands:
ihave [stuff] - announce to the network, ie. broadcast a WN packet
newstuff [args] - show the list of new items dtella currently knows about
notify <on | off> - whether dtella should tell the user when it receives
WR/WN packets

This was reported by Ximin Luo.  Will investigate:

When dtella is logging to a stdout and this is closed (eg. virtual tty),
dtella will produce these errors in subsequent logging attempts. These
errors should be ignored, and dtella should stop logging to stdout.

X

[17:02:18] <*Dtella> Something bad happened.  You might want to email this
to [email protected] so we'll know about it:
Version: Dtella@Cambridge 1.2.3/L
Traceback (most recent call last):
 File "/home/xl269/.dtella/twisted/internet/base.py", line 1048, in run
   self.mainLoop()
 File "/home/xl269/.dtella/twisted/internet/base.py", line 1057, in mainLoop
   self.runUntilCurrent()
 File "/home/xl269/.dtella/twisted/internet/base.py", line 707, in
runUntilCurrent
   log.deferr()
 File "/home/xl269/.dtella/twisted/python/log.py", line 169, in err
   msg(failure=_stuff, why=_why, isError=1, **kw)
--- <exception caught here> ---
 File "/home/xl269/.dtella/twisted/python/log.py", line 239, in msg
   self.observers[i](actualEventDict)
 File ".dtella/dtella.py", line 58, in logObserver
   handler(text)
 File "/usr/lib/python2.5/logging/__init__.py", line 1035, in critical
   apply(self._log, (CRITICAL, msg, args), kwargs)
 File "/usr/lib/python2.5/logging/__init__.py", line 1101, in _log
   self.handle(record)
 File "/usr/lib/python2.5/logging/__init__.py", line 1111, in handle
   self.callHandlers(record)
 File "/usr/lib/python2.5/logging/__init__.py", line 1148, in callHandlers
   hdlr.handle(record)
 File "/usr/lib/python2.5/logging/__init__.py", line 655, in handle
   self.emit(record)
 File "/usr/lib/python2.5/logging/__init__.py", line 757, in emit
   self.handleError(record)
 File "/usr/lib/python2.5/logging/__init__.py", line 706, in handleError
   traceback.print_exception(ei[0], ei[1], ei[2], None, sys.stderr)
 File "/usr/lib/python2.5/traceback.py", line 124, in print_exception
   _print(file, 'Traceback (most recent call last):')
 File "/usr/lib/python2.5/traceback.py", line 13, in _print
   file.write(str+terminator)
exceptions.IOError: [Errno 5] Input/output error

[17:02:18] <*Dtella> Something bad happened.  You might want to email this
to [email protected] so we'll know about it:
Version: Dtella@Cambridge 1.2.3/L
Traceback (most recent call last):
 File ".dtella/dtella.py", line 228, in main
   runClient(dc_port)
 File ".dtella/dtella.py", line 148, in runClient
   reactor.run()
 File "/home/xl269/.dtella/twisted/internet/base.py", line 1048, in run
   self.mainLoop()
 File "/home/xl269/.dtella/twisted/internet/base.py", line 1057, in mainLoop
   self.runUntilCurrent()
--- <exception caught here> ---
 File "/home/xl269/.dtella/twisted/internet/base.py", line 705, in
runUntilCurrent
   call.func(*call.args, **call.kw)
 File "/home/xl269/.dtella/dtella/common/core.py", line 1644, in cb
   self.checkStatus()
 File "/home/xl269/.dtella/dtella/common/core.py", line 1775, in checkStatus
   self.initCompleted(good=False)
 File "/home/xl269/.dtella/dtella/common/core.py", line 1800, in initCompleted
   self.done_callback('no_nodes', None)
 File "/home/xl269/.dtella/dtella/common/core.py", line 4418, in cb
   "No online nodes found.")
 File "/home/xl269/.dtella/dtella/client/main.py", line 278, in showLoginStatus
   LOG.debug(text)
 File "/usr/lib/python2.5/logging/__init__.py", line 971, in debug
   apply(self._log, (DEBUG, msg, args), kwargs)
 File "/usr/lib/python2.5/logging/__init__.py", line 1101, in _log
   self.handle(record)
 File "/usr/lib/python2.5/logging/__init__.py", line 1111, in handle
   self.callHandlers(record)
 File "/usr/lib/python2.5/logging/__init__.py", line 1148, in callHandlers
   hdlr.handle(record)
 File "/usr/lib/python2.5/logging/__init__.py", line 655, in handle
   self.emit(record)
 File "/usr/lib/python2.5/logging/__init__.py", line 757, in emit
   self.handleError(record)
 File "/usr/lib/python2.5/logging/__init__.py", line 706, in handleError
   traceback.print_exception(ei[0], ei[1], ei[2], None, sys.stderr)
 File "/usr/lib/python2.5/traceback.py", line 124, in print_exception
   _print(file, 'Traceback (most recent call last):')
 File "/usr/lib/python2.5/traceback.py", line 13, in _print
   file.write(str+terminator)
exceptions.IOError: [Errno 5] Input/output error

Presently there are some bridge hacks that allow monitoring of stats via
cacti, however, these mechanisms rely heavily on outdated code and custom
scripts on both ends to function.

This enhancement would merge these statistics into a section of the bridge
server where they could be queried via SNMP or other mechanisms. SNMP is
advantageous because it is supported by many graphing applications such as
cacti.

For the purposes of implementing said SNMP server, Dtella Labs has been
assigned the enterprise number "30490" by IANA.

Verify that the changes in InspIRCd 2.0 do not break the Dtella bridge protocol.

Make necessary changes to the bridge to be compatible with 1.2 and 2.0 if 
required.

This should probably be handled more intelligently someday:

Version: Dtella@Purdue 1.2.3/W
Traceback (most recent call last):
 File "dtella.py", line 148, in runClient

 File "twisted\internet\base.pyo", line 1048, in run

 File "twisted\internet\base.pyo", line 1057, in mainLoop

 File "twisted\internet\base.pyo", line 705, in runUntilCurrent

--- <exception caught here> ---
 File "dtella\common\state.pyo", line 112, in cb

exceptions.IOError: [Errno 13] Permission denied:
'C:\\Users\\Bill/.dtella/dtella.state'

Currently moderation of a Dtella network (i.e. kicking and/or banning of
users) is only possible via the IRC Bridge. There should probably be some
way of doing this without the need for IRC.

Some current proposed options include:

1. Web Interface
2. Integration into DC++ client
3. Configuration file

The "not authorized" sometimes shows a strange on-campus IP address for 
off-campus nodes, because the true off-campus IP gets filtered out by 
addMyIPReport.

I'll need to reproduce this case, and figure out how to fix it.

Some campuses change their subnets around somewhat frequently, and so being 
able to update the allowed_subnets by simply changing a dconfig setting in the 
bridge would be extremely useful. Right now, redistributing the client can be 
difficult (especially while Issue 10 is open).

Currently, inspircd.py is still relying on UnrealIRCd's hostname masking 
algorithm, so Dtella users' hostnames are in a different format than normal 
IRC users.

It might be nice to implement InspIRCd's algorithm in Python someday.  
Although their algorithm is slightly crazy, so the implementation needs to 
include unit tests.

We are considering switching the official supported IRCd to InspIRCd from
UnrealIRCd.

The reasons for this are:
1. InspIRCd is a much more actively developed IRCd and is gaining popularity
2. Development of UnrealIRCd has nearly ceased recently.
3. If development of Unreal 4 does pick up again, it is a fork of the
InspIRCd codebase

We will most likely abstract bridge control in such a way that allows us to
support both legacy Unreal as well as InspIRCd (and potentially other
interfaces in the future)

Currently the dtella client responds to ! commands using its virtual bot.

The IRC side should respond to the same set of commands in PM coming from the 
bridge virtual user.

A new config (DNS etc.) format in which all parameters are timestamped,
encrypted (with the network key, or some other kinda-secret symmetric key),
and signed (with an RSA public key in local_config.py).

This would let us store the config in multiple untrusted locations, and
would prevent snooping by people who don't have the right Dtella client.

There was an instance on 2008-04-16 21:10 where a user was kicked by
services (ModMan) and it caused the bridge to lose connection to IRC.


== IRC Transcript ==
...
[9:10pm] <|Rockzo> lol wisconsin law
[9:10pm] <|Rockzo> “IF YOU LEAVE A LOADED FIREARM WITHIN THE
[9:10pm] <|Rockzo> REACH OR EASY ACCESS OF A CHILD YOU MAY BE
[9:10pm] * |Rockzo was kicked by ModMan (Turn caps lock OFF!)
[9:10pm] |NICHOLAS «[email protected]» has Quit iRC
(nucleus.dhirc.com purdue.bridge.dtella.org) «626 people»
[9:10pm] |Face`we-H «[email protected]» has Quit iRC
(nucleus.dhirc.com purdue.bridge.dtella.org) «625 people»
...

== Bridge Log ==
...
2009-04-16 21:10:25,675 - P - <: :536AAAUIK PRIVMSG #dtella :lol wisconsin law
2009-04-16 21:10:26,730 - P - <: :536AAAUIK PRIVMSG #dtella :IF YOU LEAVE A
LOADED FIREARM WITHIN THE
2009-04-16 21:10:26,734 - P - <: :536AAAUIK PRIVMSG #dtella :REACH OR EASY
ACCESS OF A CHILD YOU MAY BE
2009-04-16 21:10:26,762 - P - <: :536AAAUIK PRIVMSG #dtella :FINED OR
IMPRISONED OR BOTH IF THE CHILD
2009-04-16 21:10:26,763 - P - <: :536AAAUIK PRIVMSG #dtella :IMPROPERLY
DISCHARGES, POSSESSES, OR EXHIBITS
2009-04-16 21:10:26,766 - P - <: :536AAAUIK PRIVMSG #dtella :THE FIREARM.
2009-04-16 21:10:26,771 - P - >: :00AAAAAAH KICK #dtella 536AAAUIK :Turn
caps lock OFF!
2009-04-16 21:10:26,773 - D - Sign Time = 0.002079 sec
2009-04-16 21:10:26,774 - P - <: :536AAAUIK QUIT :Kicked
2009-04-16 21:10:26,775 - P - >: :00AAAAAAH KICK #dtella 536AAAUIK :Turn
caps lock OFF!
2009-04-16 21:10:26,796 - C - Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/twisted/python/log.py", line 51,
in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/usr/lib/python2.5/site-packages/twisted/python/log.py", line 36,
in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/usr/lib/python2.5/site-packages/twisted/python/context.py", line
59, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.5/site-packages/twisted/python/context.py", line
37, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File
"/usr/lib/python2.5/site-packages/twisted/internet/selectreactor.py", line
146, in _doReadOrWrite
    why = getattr(selectable, method)()
  File "/usr/lib/python2.5/site-packages/twisted/internet/tcp.py", line
362, in doRead
    return self.protocol.dataReceived(data)
  File "/usr/lib/python2.5/site-packages/twisted/protocols/basic.py", line
148, in dataReceived
    self.lineReceived(line)
  File "/home/dtella/bridge/dtella/bridge/inspircd.py", line 245, in
lineReceived
    f(prefix, args[1:])
  File "/home/dtella/bridge/dtella/bridge/inspircd.py", line 385, in
handleCmd_KICK
    n00b_u = self.ism.findUser(n00b)
  File "/home/dtella/bridge/dtella/bridge/bridge_server.py", line 431, in
findUser
    return self.users[inick.lower()]
exceptions.KeyError: '536aaauik'

2009-04-16 21:10:26,798 - I - Lost IRC connection.
...

Presently all version control (notification of new upgrades to the Dtella
client) are handled by the dynamic configuration record.

I suggest we add an option to tbe bridge to auto kick/kill users with a
version older than a set parameter as an optional feature. This could be
used to remove versions easily in the event a given version has a serious
glitch and the network operator withes to force its removal with no
override from the users.