maddiem4 / ejtp-lib-python Goto Github PK

There's gonna be a lot of common structure between, say, TCPJack, TLSJack, SocketIOJack, etc. Presumably, similar observations could be made of jacks that follow the datagram model, UDPJack being the obvious one, but TwitterDMJack and IRCJack would be other possible examples. What I'd like to see happen is the factoring of as much code as possible out of UDPJack and TCPJack, so that creating new jacks in these patterns is simply a matter of plugging in the appropriate backend code to fill in the blanks of an established class structure.

This ticket is done when UDPJack and TCPJack are subclasses of DatagramJack and StreamJack, respectively, and the code environment is sane and cleaned up afterwards.

EJForward is going to be an integral store-and-forward substrate for other services like EJMail, adding an optional layer of reliability and cloud-like behavior to any projects that want to leverage such things. Normally, I'd put this sort of thing in an entirely separate project, but in this case, I'd prefer to bundle it within the EJTP library. This is the same approach I will later take with EJIdent.

EJForward is based on a pattern of intermediate storage, as follows. It does NOT handle the arrangement of storage space, nor how much space a storage provider offers. There may be future automated administration protocols for this, but at this time, you're expected to have the provider set it up for you based on personal communication, or some automated system provided by the provider which uses the traditional HTTPS, email verification, and CAPTCHA combination.

When this ticket is complete, you should be able to import ejtp.ejforward.ForwardServer, which subclasses the ejtp.client.Client class, and accepts a target interface, and optional storage caps based on size or count. You should also be able to import the ejtp.ejforward.ForwardClient class, which has the same parent as ForwardServer, but operates as the target of a storage provider. This class simply bounces encapsulated messages back out to the EJTP router, which can then deliver them appropriately.

ejforward-notify

Tells a target that it has messages waiting for retrieval. This is used for retries if the target does not acknowledge the receipt of a message.

• total_count
• total_space
• used_space
• hashes (the hashes of a few unretrieved messages. Not necessarily all of them, could be capped at 5 for example)

ejforward-message

Encapsulates a message so that the target will acknowledge it.

• data (string contents of message)

ejforward-ack

Mark some message hashes as arrived. This allows the storage provider to free memory.

• hashes

ejforward-retrieve

The hashes property for this message is optional. If omitted or None, the storage provider responds with a small burst of unretrieved messages, the size of which is up to the storage provider.

• hashes

ejforward-get-status

Respond with ejforward-notify.

ejforward-error

Simple error message, for things like trying to retrieve an unstored hash.

• code
• data
• human (human-readable explanation of the error)

• Modularize Jacks better for plugin extensions
• TLS Jack

It's common practice to use the development branch as base branch and make new stable branches apart from the development one.
Besides it's much easier for merging, patching, creating, etc. of milestones based on a development version.

Currently, s frames are pretty simple. They use the sender's private encryptor, such that the public version can be used to decrypt it - beneficial for data obfuscation and bandwidth savings, in theory. But many types of encryption don't like it when you try to decrypt arbitrary plaintext with them, and even for the ones that don't complain, it's often still blatant misuse of the technology.

Just about every system has some method for signatures, though. So the proper thing to do is to include a signature with every frame. The method I propose is to have the two bytes after the null separator dictate the signature length, followed by the sig itself, and finishing with the payload.

Normal frame:
s["udp4", ["localhost", 9999], "smell"]\x00______<encrypted payload>__________________

New frame:
s["udp4", ["localhost", 9999], "smell"]\x00\x01\x00______<SHA-256 signature>______{"type":"ejmail-message ...}

Note that the 256-bit digest size of the signature is represented by the \x01\x00, and that the 2-byte size allows digests up to 8192 bytes long (exactly 8kb). Note also that this is flexible, because it represents the size of the encrypted digest, not the original digest. Different encryption algorithms may introduce different amounts of padding and such.

One of the requirements, besides the obvious differences in the creation and parsing of s-frames, is that every encryptor needs to have "signature-awareness", meaning that the functions self.sign(plaintext) and self.sig_verify(plaintext, signature) are defined. The base class can define defaults for these that operation on the assumption that the encryptor will not choke when decrypting plaintext data, but for classes like RSA, override functions will be used.

Plan of attack:

1. Add signature awareness to encryptor classes
2. Edit ejtp.frame.Frame to create and interpret s-frames in the new format

The DaemonClient is a reasonably sharp sword - still a bit oddly shaped, but it should straighten out with use and time - but without a pair of eyes, all you can use it for is stabbing about in the dark. Not good. The correct solution, of course, is to create tools with which you can query and explore the landscape of your ejtpd process.

The "describe" action should provide a neat synopsis of an interface. Its Python type, its repr(), and if available, a bit of extra data stored in client._description. This can be of any type, but should be consistent for each class.

The "explore" action should list all the jack and client interfaces. Format ought to be pretty simple.

This was always a planned feature, let's make it happen.

An MOTD server simply reads from a predetermined filename and sends that as the meat of the response. Very simple mechanism, basically just for testing that your settings are right/hosting a textual message for people to find.

Protocol

motd-request

Client->Server. Asks for the current value being hosted.

motd-response {content="..."}

Server->Client. Contains the message of the day.

Command line usage

There should be a script, ejtp-test, which will contain all sorts of tools but for now only have the subcommand "motd". Make that script as well.

Make a TCP Jack as one of the standard jack types. This should provide a guide for all stream-based transports.

TCP overlay format

Each frame is prepended with a number and a period, using the period as a separator. For example,

53892.["tcp",["40.76.128.33", 6666]...

This indicates the total length of the frame, such that after the given X bytes, the receiving jack can start the "read a number, eat some bytes" process again for a new frame. That's the only gimmick, and it's completely invisible to the router.

Connection behavior

The jack should keep a cache of connections, making them as necessary and retaining them until an attempt to use them fails as an IOError. That way you only create connections for hosts you talk to, and only replace them when they have failed.

There's a lot in there that would be more useful as an intermediate subclass between Encryptor and RSAEncryptor, since then AESEncryptor and such could use it too. Copy/paste is bad.

It doesn't bother me if proprietary software companies use this code, as long as they contribute their changes back. According to the GPL how-to, this is as simple as adding a full copy of the license to the codebase, and a notice at the head of every source file.

Below you can see the full traceback.
It seems that the pycrypto Exception message changed and thus the doctest fails because it expects the old one.

File "EJTP/ejtp/crypto/rsa.py", line 86, in ejtp.crypto.rsa.RSA.sig_verify
Failed example:
    public.sign(plaintext)
Expected:
    Traceback (most recent call last):
    TypeError: No private key
Got:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/doctest.py", line 1289, in __run
        compileflags, 1) in test.globs
      File "<doctest ejtp.crypto.rsa.RSA.sig_verify[7]>", line 1, in <module>
        public.sign(plaintext)
      File "EJTP/ejtp/crypto/rsa.py", line 64, in sign
        return self.signer.sign(self.hash_obj(plaintext))
      File "/usr/lib/python2.7/dist-packages/Crypto/Signature/PKCS1_PSS.py", line 143, in sign
        m = self._key.decrypt(em)
      File "/usr/lib/python2.7/dist-packages/Crypto/PublicKey/RSA.py", line 174, in decrypt
        return pubkey.pubkey.decrypt(self, ciphertext)
      File "/usr/lib/python2.7/dist-packages/Crypto/PublicKey/pubkey.py", line 93, in decrypt
        plaintext=self._decrypt(ciphertext)
      File "/usr/lib/python2.7/dist-packages/Crypto/PublicKey/RSA.py", line 239, in _decrypt
        mp = self.key._decrypt(cp)
    TypeError: Private key not available in this object

I used the following RI Blog Post blog post to install the library.

When I run doctestall -l ejtp it hangs testing on jetp.jacks.tcp library

doctestall -l ejtp      
Testing ejtp
Testing ejtp.logging
Testing ejtp.logging.verbose
Testing ejtp.address
Testing ejtp.client
Testing ejtp.crypto
Testing ejtp.crypto.aes
Testing ejtp.crypto.encryptor
Testing ejtp.crypto.rsa
Testing ejtp.crypto.rotate
Testing ejtp.daemon
Testing ejtp.ejforward
Testing ejtp.ejforward.client
Testing ejtp.ejforward.server
Testing ejtp.frame
Testing ejtp.interactive
Testing ejtp.jacks
Testing ejtp.jacks.core
Testing ejtp.jacks.streamjack
Testing ejtp.jacks.tcp

Any help appreciated.

I'm pretty sure that the pattern involves the overriding of kwargs, but until #27 is done, this will be really hard to diagnose. So block until that's done, then figure out WTF is going on here and fix it.

This is the last step before I push v0.9.0 to stable. Shouldn't take too long.

See #47 for further information.

As this project gets bigger I think doctests are a huge disadvantge. In my opinion they are not intuitive to write and make a mess in the docstrings. Another advantage using unittests is that you can write code portable between different python versions much easier.
I propose to add a new directory called "unittests" and placing all test within.
Right now there are 162 doctests. We should rewrite them to unittests before it's too late!

• ejtpd and ejtp-keygen scripts
• various security improvements

For v0.8, make ejtp.crypto point to the ejtp.util.crypto module. This will allow developers time to migrate their code over to use that location. v0.9 deprecates the old location, as #13 covers in more detail.

In #42 we introduced python3 support, so all commits after that one should be compatible with python3.
Some examples:

• e8fc381 uses relative imports with old syntax
• 8020b43 uses u'string' syntax in doctests

Add an elliptic curve encryptor wrapper to the crypto library.

Support ejtp-multipart messages for fragmenting messages when they breach the EJTP MTU. This number is independent of transport medium and exists to promote fairness among competing messages, transport mediums that have a lower MTU must deal with that in their own ways.

Move ejtpd script into new scripts folder, put an ejtp-keygen script in there too.

Jack that supports the IRC protocol (RFC 1459) for both channel and DM-equivalent mediums. The IRC protocol is probably going to be one of the most important ones to support, since it helps enable people to use the same interface address from device to device.

Formerly interactive.py, ejtp-console is fairly ancient, not very modular, and uses some unspeakable hacks regarding encryption data. The primary goal in this issue is to make this script use identity cache files, but there are other goals branching out from that:

• Look for idcache files in paths ~/.ejtp/idents.json, ~/.ejtp/console/idents.json, and finally $EJTP_IDENTITY_CACHE. Do this in such a way that each test-and-deserialize is a distinct iteration of a non-breaking loop, so that for example, data in ~/.ejtp/console/idents.json that collides with data in ~/.ejtp/idents.json will update those specific identities.
• When picking an identity to use as yourself, the script should list all identities in cache that can be used as senders (excluding, for example public-only RSA).
• When sending, any identity is available.
• Identities should always be presented to the user as names, not locations.
• Replace the godawful encryptor hacks I'm pretty sure are still in use. Use Identity encryptors, obviously.

This is mostly to make testing more consistent. Also add a property to the router for the default blocking value.

On sufficiently long inputs, RSA craps out and silently outputs gibberish. Not all long inputs, though, only some. Not sure what's going on or where the bug is, but I built some test code awhile back to find out.

EDIT: PyCrypto folks! Hello! I have some demo code that shows off the error, but it makes use of the EJTP library, rather than operating on PyCrypto directly. I would normally try to strip that stuff out so that the example is as simple as possible, but one of the things the RSA 'encryptor' does is allow you to encode long strings with RSA by breaking up the long string into 128-byte chunks, as you can see in this code. The problems arrive inconsistently, and may very well be in my stitching code, as a result of me making some sort of misunderstanding of the RSA model.

To test it, you'll want to install this EJTP library, and run rsatest.py. There is a possibility you might be able to do this without installing, by hanging out in the top-level directory and invoking it as python ejtp/rsatest.py and praying that the relative imports work, but it's not tested for that and I make no guarantees. Sorry that this is inconvenient, I wish it was less so, and I understand any wariness to install untrusted code on a whim.

Other, dependent projects are starting to get less friendly to debug thanks to the debug lines from EJTP jacks. I'd very much like to use the logging module like a proper project so that debugging information is handled in a logical manner.

Improve the formatting of the codebase by converting from hard tabs to 4 spaces. I'm on my phone right now and can't test this, but I think the correct vim command for this is:

:%s/$\( *\)\t/\1    /g

In each file. In many files, a simpler replacement would probably work, but I'm trying to only edit whitespace at the beginning of each line, to avoid unexpected debugging work. Then again, it's a beginner bug, so maybe a little debugging that makes a new developer explore the codebase is a good thing.

Have all jacks in a package of their own, instead of hanging out in top-level like those weird, leering guys in front of the local Safeway.

Following the directions here: http://roaming-initiative.net/blog/blog/sponsoring-free-software.html

I've installed everything (although through PyPI, if possible, and always with pip). When I get to doctestall ejtp I immediately get the following failed test:

**********************************************************************
File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/ejtp/crypto/rsa.py", line 86, in ejtp.crypto.rsa.RSA.sig_verify
Failed example:
    public.sign(plaintext)
Expected:
    Traceback (most recent call last):
    TypeError: No private key
Got:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.2/lib/python2.7/doctest.py", line 1254, in __run
        compileflags, 1) in test.globs
      File "<doctest ejtp.crypto.rsa.RSA.sig_verify[7]>", line 1, in <module>
        public.sign(plaintext)
      File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/ejtp/crypto/rsa.py", line 64, in sign
        return self.signer.sign(self.hash_obj(plaintext))
      File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/Crypto/Signature/PKCS1_PSS.py", line 143, in sign
        m = self._key.decrypt(em)
      File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/Crypto/PublicKey/RSA.py", line 174, in decrypt
        return pubkey.pubkey.decrypt(self, ciphertext)
      File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/Crypto/PublicKey/pubkey.py", line 93, in decrypt
        plaintext=self._decrypt(ciphertext)
      File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/Crypto/PublicKey/RSA.py", line 239, in _decrypt
        mp = self.key._decrypt(cp)
    TypeError: Private key not available in this object
**********************************************************************
1 items had failures:
   1 of   8 in ejtp.crypto.rsa.RSA.sig_verify
***Test Failed*** 1 failures.

ejtp.identity should probably be a package containing multiple modules, considering the wide range of identity-related features we might want to write. Even breaking out the encryptor cache class into its own module will be a short-term gain. Still trying to think what the filename should be for the core identity object... actually, core.py is not bad, thanks to the surrounding context. There we go.

This is going to be a potentially wide-sweeping innovation across the codebase, but should be simple enough to get going soon. But considering the potential scope of changes, I'm taking personal reins on this one, the only problem being my full-time job and school consuming most of my code time (and energy, since I write code for both). So there's no guarantee this will happen soon, even though it's the next big thing that needs doing. An irritating conundrum, to be sure.

EJTP frames, due to being wrapped so many layers deep in each other, can be a bit hefty for network use. So for version 1.0, I'm introducing a new type of frame for bandwidth optimization.

A g-type frame has a type code of "g", obviously. It has no address data, the g is immediately followed by the null separator. The payload is another EJTP frame string, compressed with the gzip algorithm.

Upon receipt of a type g message, a router should decode the gzipped contents and then attempt to route the underlying frame. The underlying frame must be routable and well-formed, or it will be dropped.

For dummy encryptors like rotate, this is just "yes." For RSA, it means having the private component of the key. Etc. It's simple enough to test for - does the encryptor equal its public self?

Expose this even at the cache level, including a general .all() method to return every identity in the cache.

Hey, I just attempted to install pycrypto, and it seems it is running into some issues while installing.

user@User:~/pycrypto$ python setup.py build
running build
running build_py
running build_ext
running build_configure
warning: GMP or MPIR library not found; Not building Crypto.PublicKey._fastmath.
building 'Crypto.Hash._MD2' extension
gcc -pthread -fno-strict-aliasing -fwrapv -Wall -Wstrict-prototypes -fPIC -std=c99 -Wextra -Wno-missing-field-initializers -Wno-unused-parameter -O3 -fomit-frame-pointer -Isrc/ -I/usr/include/python2.7 -c src/MD2.c -o build/temp.linux-x86_64-2.7/src/MD2.o
src/MD2.c:31:20: fatal error: Python.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1

I installed the apt repository libgmp3-dev, but it seems I get the same error even after installing it. Any ideas on how to fix or resolve this issue?

EJTP should have some basic standard utilities for exchanging encryption information and such. These need to get along with higher-level protocols and their libraries, in well thought-out ways.

ejtp-error

• code
• msg

ejtp-hello

Carries signed data about an interface. Generally the most important property of the data object is "encryptor", though the only required properties are "interface" and "date" (the latter is an ISO 8601 datetime). Each signature is based on the checksum of "data".

• data
• sigs - mapping of strinterface : signature

ejtp-request

Ask for a hello of an interface.

• interface

Error Codes

10 - Hello rejected
11 - Permission denied

I don't use Python 3.X, but lots of people do, and we ought to support that. For now, I'd be happy with just the unit tests passing, even though I'm sure there's still plenty of stuff that will fall through the cracks. We'll just have to catch it later in a more aggressive round of cleanup.

You shouldn't be able to do that by accident/unknowingly. Let's complain a little.

It drives me crazy to have to copy and paste this code everywhere. I want to have one place where recursive doctesting is coded and then just move that completely out of all my other project repos.

Ticket is complete when testing is available through DoctestAll and removed from python-libejtp.

• #28 Refactor RSA and AES code, including more testing (possible WONTFIX, this may be an impossible task)
• #27 Flesh out the ejtp-daemon protocol. (deferring to later version)
• #26 Throw exception when overriding an existing interface in a Router
• #33 Fix the "not making jacks" thing in ejtpd. (deferring to later version)
• Merge all the recent finished work together, and make sure it works.

• #45 Use unittests instead of doctests (and all the subtickets of that)
• #53 Modernize ejtp-console script (ended up done early, releasing in v0.9.2)

Create a TLS Jack based on the TCP Jack.

This is the major change that will break compatibility for library users. Of course, it's nothing a good old-fashioned s/ejtp.util.crypto/ejtp.crypto won't fix, in theory, but it's still a change that will break some dependents.

v0.8 will support access at both locations, probably through an import in ejtp/__init__.py. v0.9 will cut support for ejtp.util.crypto completely. Developers should change their code as soon as possible.

Stayed up tonight to come up with a logo and mascot. I'm pretty happy with both. Need to upload to RI site for hosting, and add references to the README so the pictures show up.

Whatever it entails, get rid of dependencies and references to the ConcurrenTree library. EJTP is shooting for more general purpose usability, and CTree is supposed to depend on it, not the other way round.

This ticket is done when grep ConcurrenTree find ejtp`` turns up nothing.

I'm getting the following error:

python ./ejtp/interactive.py
Traceback (most recent call last):
  File "./ejtp/interactive.py", line 24, in <module>
    from router import Router
  File "/Users/fortiz/development/pythondev/EJTP-lib-python/ejtp/router.py", line 27, in <module>
    from ejtp import logging
  File "/Users/fortiz/development/pythondev/EJTP-lib-python/ejtp/logging/__init__.py", line 23, in <module>
    import logging
  File "/Users/fortiz/development/pythondev/EJTP-lib-python/ejtp/logging/__init__.py", line 47, in <module>
    logging.INFO,
AttributeError: 'module' object has no attribute 'INFO'

I'm using Mac OS X Lion

Move all the splitting code to a separate branch, preserve it there for reference and eventual reintegration. The half-assed splitting code as currently present is more harmful than helpful, and for stability purposes, it's probably a lot better to be naive about size than to try to implement splitting halfway.

See this discussion for further information.

It's silly to have two. It made more sense with the original intended architecture of the project, which was all about "middleware" and modular client chaining, but now the abstraction is dead weight and pointless complexity.