kpdyer / fteproxy Goto Github PK
View Code? Open in Web Editor NEWprogrammable proxy for censorship circumvention
Home Page: https://fteproxy.org/
License: Apache License 2.0
programmable proxy for censorship circumvention
Home Page: https://fteproxy.org/
License: Apache License 2.0
Currently, fte_relay can only be run from its parent directory, because it assumes the location of the *.dfa files. This should not be the case.
Instead of creating our own managed-proxy interface for Tor. Enable an (optional?) integration with pyptlib:
https://gitweb.torproject.org/pluggable-transports/pyptlib.git
Currently we use TCP internally for IPC, for control processes to communicate with worker processes, which is bad. We should investigate more refined methods (e.g. Python's multiprocessing module) for worker threads to receive/process incoming data streams.
The current code release does not include any method to compile custom regular expressions into the necessary DFAs for use in the FTE framework. This issue focuses on adding an easy-to-use workflow for compiling new regular expressions. Should probably include a nice way to manage the regular expressions and add them to the help dialog.
At the moment, the FTE client can support only one FTE server IP address. In many cases the server may be down or inaccessible. Hence, it should be the case that the client can support a list of FTE servers and attempt to connect to them in order.
The command
fteproxy --stop
defaults as
fteproxy --mode client --stop
So, when
fteproxy --stop
is executed on an fteproxy server, the fteproxy server is not terminated and the command fails silently.
We currently specify "#!/usr/bin/python" at the top of scripts. We should instead use "#! /usr/bin/env python"
When executing "fteproxy --stop" the following error occurs:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
self.run()
File "/usr/local/bin/fteproxy", line 110, in run
if fte.conf.getValue('runtime.console.debug'):
File "/usr/local/lib/python2.7/dist-packages/fte/conf.py", line 24, in getValue
return conf[key]
KeyError: 'runtime.console.debug'
The following error can be produced by attempting to bootstrap Tor over FTE.
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
self.run()
File "/usr/local/lib/python2.7/dist-packages/fte/relay.py", line 56, in run
[success, _data] = fte.io.recvall_from_socket(self._socket2)
File "/usr/local/lib/python2.7/dist-packages/fte/io.py", line 48, in recvall_from_socket
_data = sock.recv(bufsize)
File "/usr/local/lib/python2.7/dist-packages/fte/__init__.py", line 78, in recv
frag = self._decoder.pop()
File "/usr/local/lib/python2.7/dist-packages/fte/record_layer.py", line 92, in pop
except fte.encoder.DecodeFailureException:
AttributeError: 'module' object has no attribute 'DecodeFailureException'
The issue is intermittent and only occurs roughly 1 out of 5 times, or so.
Further to Issue #10, we should improve the verbosity of our command line application to let user know that the FTE client/server has successfully started and is listening on the specified port.
The following AssertionError was encountered in normal operation of fteproxy:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
self.run()
File "/usr/local/lib/python2.7/dist-packages/fte/relay.py", line 50, in run
[success, _data] = fte.io.recvall_from_socket(self._socket1)
File "/usr/local/lib/python2.7/dist-packages/fte/io.py", line 48, in recvall_from_socket
_data = sock.recv(bufsize)
File "/usr/local/lib/python2.7/dist-packages/fte/__init__.py", line 78, in recv
frag = self._decoder.pop()
File "/usr/local/lib/python2.7/dist-packages/fte/record_layer.py", line 87, in pop
incoming_msg = self._decoder.decode(self._buffer)
File "/usr/local/lib/python2.7/dist-packages/fte/encoder.py", line 119, in decode
assert len(covertext) >= self._max_len, (len(covertext), self._max_len)
AssertionError: (145, 512)
At the moment our build process is a hybrid between a Makefile and a Python setup.py file. What's more, the build step in the FTE directory depends upon file in "third-party" located in FTE's parent directory. This is bad form. We should organize the project such that either "./configure && make && make install" or "python setup.py install" works from the root directory on all Linux-based systems.
Proxy appears to crash when client is negotiating with it. Could be build issue or incompatibility with client version. Could even be an unintended response to port scanning on HTTP ports (8080 in this case).
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
self.run()
File "/usr/local/lib/python2.7/dist-packages/fte/relay.py", line 112, in run
conn = self.onNewIncomingConnection(conn)
File "/usr/local/lib/python2.7/dist-packages/fte/server.py", line 30, in onNewIncomingConnection
socket = fte.wrap_socket(socket)
File "/usr/local/lib/python2.7/dist-packages/fte/__init__.py", line 307, in wrap_socket
K1, K2)
File "/usr/local/lib/python2.7/dist-packages/fte/__init__.py", line 130, in __init__
[negotiate_cell,remaining_buffer] = self._accept_negotiation(data)
TypeError: 'bool' object is not iterable
The current TBB integration is rough: we have a hard-coded IP:port to our FTE server in Wisconsin.
We need to play nicer with the TBB and make it such that users can configure their Tor+FTE bridge via the Vidalia GUI.
Placeholder for general code quality and best practices.
We should add a command-line switch to enable FTE to run in the background. At the moment we run FTE in the background by using "&" on Linux. As a byproduct, we don't output much information unless the "--debug" switch is added.
We should:
(1) Add a "-d" switch for FTE to start as a daemon.
(2) When "-d" is not specified we should be a bit more chatty, so the user knows what is going on.
The build script should determine the number of cores on the build system and not naively pass "-j8" to make.
When starting up as part of daemon I get this error:
Traceback (most recent call last):
File "/usr/local/bin/fteproxy", line 30, in
import fte.conf
File "/usr/local/lib/python2.7/dist-packages/fte/__init__.py", line 29, in
import fte.client.managed
ImportError: No module named managed
Traceback (most recent call last):
File "/usr/local/bin/fteproxy", line 30, in
import fte.conf
File "/usr/local/lib/python2.7/dist-packages/fte/__init__.py", line 29, in
import fte.client.managed
ImportError: No module named managed
We currently use boost in "FTE/src/fte/cRegex.cc" for a small number of features. This dependency is far too heavyweight and should be removed.
When attempting to compile openfst-1.3.3 under Ubuntu 13.10, the following error occurs.
...
/bin/bash ../../libtool --tag=CXX --mode=link g++ -g -O2 -o fstarcsort fstarcsort.o ../script/libfstscript.la ../lib/libfst.la -lm -ldl
libtool: link: g++ -g -O2 -o .libs/fstarcsort fstarcsort.o ../script/.libs/libfstscript.so ../lib/.libs/libfst.so -lm -ldl
../script/.libs/libfstscript.so: undefined reference to `dlopen'
../script/.libs/libfstscript.so: undefined reference to `dlerror'
configure command:
./configure --enable-fast-install --disable-dependency-tracking --prefix=/usr/local
GCC: 4.8.1
Right now we simply output "Files .no_fte.txt and .yes_fte.txt are identical."
The "systemtests" script should be an opportunity for us to ensure that environmental variables are correctly set, and that fteproxy is installed properly.
Use the Coverity source code analysis service to detect any issues with our C++ source code.
On a default OSX install, Gatekeeper prevents a user from running our Tor Browser Bundle.
http://support.apple.com/kb/ht5290
We either have to (1) jump through the hoops and pay the money to sign our releases or (2) explain to users how to get around this problem, probably by linking to the above Apple KB article.
For now, solution (2) should be fine, I assume.
-Kevin
"make all" on Ubuntu 12.04 64-bit results in the following error:
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/4.6/crtbeginT.o: relocation R_X86_64_32 against `__DTOR_END__' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-linux-gnu/4.6/crtbeginT.o: could not read symbols: Bad value
collect2: ld returned 1 exit status
error: command 'g++' failed with exit status 1
GCC: 4.6
The current build.py is hard-coded, works for Ubuntu and has hacks for OSX.
We should transition to setup.py to facilitate cross-platform builds.
We should fully integrate in the TBB build process and not retrieve the TBB ourselves.
In the recent major refactor I removed the ability for client to dictate (in-band) the regex it wants to use. This is a regression and should be resolved.
Currently, the code has a single hard-coded encryption key that is used for the authenticated encryption scheme. In the future, we want to be able to allow the user to specify and use their own long-term keys.
At the moment the core FTE code (rank/unrank/buildTable), our AE scheme (encrypt/decrypt) and FTE proxy (management of connections, etc.) are all jumbled into one program.
(1) We should separate the FTE code into a library that can be used as a C++ library, or as a Python module. This should be quite easy to do and is mostly a reorganization of directory structures.
(2) We may want to separate out our AE scheme in the same way, too.
Then that will leave use with our FTE proxying code that uses the AE scheme and FTE library to encrypt and encode messages. The main advantage of doing this is to increase transparency and modularity. This refactor should also be done in a way that allows us to swap in/out or disable/enable encryption/encoding for testing purposes.
We should have a good understanding of Tor's existing PyObfsproxy code and keep that in the back of our minds before we attempt this. It would be ideal if we could organize FTE code such that it works well with PyObfsproxy, esp. w.r.t. receiver-side parsing.
Learned formats don't take much disk space, but, given the current buildTable algorithm, they take orders of magnitude more memory than other formats. We should either (1) delete these formats or (2) keep them around, but disable them by default server-side. (This could be achieved by adding a server-side "--experimental" switch that loads learned formats only when the switch is specified.)
At the moment, FTE format names are determined by their filename in the formats directory. This name is the only token used when performing regex negotiation. Hence, if a client has a regex name that matches the server, but the actual regex doesn't match, then the negotiation fails and the client will get unexpected results.
We should make it such that the naming scheme used for format negotiation includes a unique identifier, such as a hash of the regex.
At the moment all FTE configuration parameters must be specified on the command line, which is already getting cumbersome. If we add new features, such as Issue#14, which enables the server admin to specify more than one server-side port, it would be ideal to specify this in a Tor-like configuration file.
The build script should produce (1) FTE binaries and (2) Tor bundles that include the FTE binaries.
In the current release the FTE server binds to 127.0.0.1 on port 8080 and we forward all data to port 8081. This is probably not correct for most deployment. As a few examples of changes we want to make, we could make the following assumptions:
(1) FTE will bind to port 80.
(2) There will be a Tor bridge on the local machine running on port 9001, and we will forward all requests to this port.
However, should the FTE server reject a request to port 80 using a non-HTTP format? Or should the server just allow any supported format on any port?
We don't need the full GPLv3 as a header for each file. We should include a single LICENSE file, then have the standard GPLv3 short boilerplate.
The number of worker processes is currently not configurable and defaults to the number of cores on the system. In many cases it is desirable to restrict the number of worker processes to one, to ensure FTE doesn't consume too much CPU. Hence, the number of worker threads should be a command line configuration parameter.
gcc/g++ support code hardening flags such as -fstack-protector-all
, -D_FORTIFY_SOURCE
, -fPIE
and some more. They should be added to the compiler flags to provide defence in depth.
All Python code from this point should match the pep8 guidelines.
Some versions of tar are not playing well with the current build script. Hence, when everything is tar'd up it includes the "dist" basedir which we don't want.
Placeholder to investigate support for FTE on Windows.
The file
FTE/src/fte/cRegex.cc
should be documented, cleaned up and analyzed by a static analysis tool such as Coverity.
Determine automatic strategy for removing outgoing_max_len and incoming_max_len paramters from fte.wrap_socket interface.
Currently, we use long-term keys for encryption. This may be sufficient if we delegate all responsibility of security goals, such as Perfect Forward Secrecy, to the tunneled protocol. However, if there is something lightweight (e.g., a variant of DH) that could be implemented, then we might want to consider how it would improve the current FTE system.
When installing FTE on the Tor-Cloud AMI instance within Amazon Web Services (AWS), the unit and system tests fail with the following error:
----------------------------------------------------------------------
ERROR: testLTB (fte.tests.bit_ops.TestEncoders)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/fte/tests/bit_ops.py", line 43, in testLTB
self.assertEquals(fte.bit_ops.long_to_bytes(0xff), '\xFF')
File "/usr/local/lib/python2.7/dist-packages/fte/bit_ops.py", line 39, in long_to_bytes
bytestring = binascii.unhexlify(bytestring)
TypeError: Non-hexadecimal digit found
----------------------------------------------------------------------
Python version is 2.7.3
GCC 4.6.3
Ubuntu 12.04
We don't quite have the dependencies right for the Makefile. So "make -jN" for N>1 results in a build failure in some cases.
Currently systemtests depends upon being able to download an external website. These tests should be completely self contained.
At the moment the FTE client listens for incoming FTE connections on port 8079 and forwards them to 8080 on 127.0.0.1. The client uses intersection-http by default.
We should determine if our current intersection-http format is the right choice for a default format. We should also determine the correct default client-side port to listen for incoming Tor connections.
Since our primary use case for FTE is as a type of pluggable transport for Tor, it makes sense to directly (and automatically) create the Tor Browser Bundles during the build process.
At the moment an FTE server can listen on only one port at a time. Hence, to listen on, say, ports 22 and 80, this requires two FTE instances to be running server-side, which requires twice the memory. However, if we setup FTE to take the list of server-side ports as a list, rather than an integer, this would all the FTE server to be much more clever and listen on multiple ports in the ServerDemuxer, without the need to have multiple FTE instances.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.