Code Monkey home page Code Monkey logo

jetforce's Introduction

Jetforce

An experimental TCP server for the new, under development Gemini Protocol. Learn more about Gemini here.

Rocket Launch

pypi version build

Table of Contents

Features

  • A built-in static file server with support for gemini directories and CGI scripts.
  • A complete framework for writing robust gemini applications like astrobotany.
  • A lean, modern codebase with type hints and black formatting.
  • A solid foundation built on top of the twisted asynchronous networking engine.

Installation

Requires Python 3.7 or newer.

The latest stable release can be installed from PyPI:

$ pip install jetforce

Or, install from source:

$ git clone https://github.com/michael-lazar/jetforce
$ cd jetforce
$ pip install .

Or, install into a python virtual environment:

# Create a project directory somewhere
$ mkdir /opt/jetforce

# Activate a virtual environment and install jetforce
$ python3 -m virtualenv /opt/jetforce/venv
$ source /opt/jetforce/venv/bin/activate
$ pip install jetforce

# The launch script will be installed here
$ /opt/jetforce/venv/bin/jetforce

Usage

Use the --help flag to view command-line options:

usage: jetforce [-h] [-V] [--host HOST] [--port PORT] [--hostname HOSTNAME]
                [--tls-certfile FILE] [--tls-keyfile FILE] [--tls-cafile FILE]
                [--tls-capath DIR] [--dir DIR] [--cgi-dir DIR] [--index-file FILE]
                [--default-lang DEFAULT_LANG] [--rate-limit RATE_LIMIT]

An Experimental Gemini Protocol Server

optional arguments:
  -h, --help            show this help message and exit
  -V, --version         show program's version number and exit

server configuration:
  --host HOST           Server address to bind to (default: 127.0.0.1)
  --port PORT           Server port to bind to (default: 1965)
  --hostname HOSTNAME   Server hostname (default: localhost)
  --tls-certfile FILE   Server TLS certificate file (default: None)
  --tls-keyfile FILE    Server TLS private key file (default: None)
  --tls-cafile FILE     A CA file to use for validating clients (default: None)
  --tls-capath DIR      A directory containing CA files for validating clients (default:
                        None)

fileserver configuration:
  --dir DIR             Root directory on the filesystem to serve (default: /var/gemini)
  --cgi-dir DIR         CGI script directory, relative to the server's root directory
                        (default: cgi-bin)
  --index-file FILE     If a directory contains a file with this name, that file will be
                        served instead of auto-generating an index page (default: index.gmi)
  --default-lang DEFAULT_LANG
                        A lang parameter that will be used for all text/gemini responses
                        (default: None)
  --rate-limit RATE_LIMIT
                        Enable IP rate limiting, e.g. '60/5m' (60 requests per 5 minutes)
                        (default: None)

Setting the hostname

The server's hostname should be set to the DNS name that you expect to receive traffic from. For example, if your jetforce server is running on "gemini://cats.com", you should set the hostname to "cats.com". Any URLs that do not match this hostname will be refused by the server, including URLs that use a direct IP address such as "gemini://174.138.124.169".

IDNs (domain names that contain unicode characters) should be defined using their ASCII punycode representation. For example, the domain name café.mozz.us should be represented as --hostname xn--caf-dma.mozz.us.

Setting the host

The server's host should be set to the local socket that you want to bind to:

  • --host "127.0.0.1" - Accept local connections only
  • --host "0.0.0.0" - Accept remote connections over IPv4
  • --host "::" - Accept remote connections over IPv6
  • --host "" - Accept remote connections over any interface (IPv4 + IPv6)

TLS Certificates

The gemini specification requires that all connections be sent over TLS.

If you do not provide a TLS certificate file using the --tls-certfile flag, jetforce will automatically generate a temporary cert for you to use. This is great for making development easier, but before you expose your server to the public internet you should setup something more permanent. You can generate your own self-signed server certificate, or obtain one from a Certificate Authority like Let's Encrypt.

Here's an example openssl command that you can use to generate a self-signed certificate:

$ openssl req -newkey rsa:2048 -nodes -keyout {hostname}.key \
    -nodes -x509 -out {hostname}.crt -subj "/CN={hostname}"

Jetforce also supports TLS client certificates (both self-signed and CA authorised). Requests that are made with client certificates will include additional CGI/environment variables with information about the TLS connection.

You can specify a CA for client validation with the --tls-cafile or --tls-capath flags. Connections validated by the CA will have the TLS_CLIENT_AUTHORISED environment variable set to True. Instructions on how to generate CA's are outside of the scope of this readme, but you can find many helpful tutorials online.

Static Files

Jetforce will serve static files in the /var/gemini/ directory by default. Files ending with *.gmi will be interpreted as the text/gemini type. If a directory is requested, jetforce will look for a file named index.gmi in that directory to return. Otherwise, a directory file listing will be automatically generated.

Virtual Hosting

For the sake of keeping the command line arguments straightforward and easy to understand, configuring virtual hosting is not supported via the command line. However, it is readily available using only a few lines of python and a custom launch script. Check out examples/vhost.py for more information.

Jetforce does not (yet) support virtual hosting at the TLS-layer using SNI. This means that you cannot return different server TLS certificates for different domains. The suggested workaround is to use a single certificate with multiple subjectAltName attributes. There is also an sni_callback() hook in the server codebase that can be subclassed to implement custom TLS behavior.

CGI

Jetforce supports a simplified version of CGI scripting. It doesn't exactly follow the RFC 3875 specification for CGI, but it gets the job done for the purposes of Gemini.

Any executable file placed in the server's cgi-bin/ directory will be considered a CGI script. When a CGI script is requested by a gemini client, the jetforce server will execute the script and pass along information about the request using environment variables.

The CGI script must then write the gemini response to the stdout stream. This includes the status code and meta string on the first line, and the optional response body on subsequent lines. The bytes generated by the CGI script will be forwarded verbatim to the gemini client, without any additional modification by the server.

CGI Environment Variables

Name Example Value
GATEWAY_INTERFACE CGI/1.1 (for compatibility with RFC 3875)
SERVER_PROTOCOL GEMINI
SERVER_SOFTWARE jetforce/0.0.7
GEMINI_URL gemini://mozz.us/cgi-bin/example.cgi/extra?hello%20world
SCRIPT_NAME /cgi-bin/example.cgi
PATH_INFO /extra
QUERY_STRING hello%20world
SERVER_NAME mozz.us
SERVER_PORT 1965
REMOTE_HOST 10.10.0.2
REMOTE_ADDR 10.10.0.2
TLS_CIPHER TLS_AES_256_GCM_SHA384
TLS_VERSION TLSv1.3

CGI Environment Variables - Authenticated

Additional CGI variables will be included only when the client connection uses a TLS client certificate:

Name Example Value
AUTH_TYPE CERTIFICATE
REMOTE_USER mozz123 (the certificate subject CN attribute)
TLS_CLIENT_HASH SHA256:86341FB480BFE333C343530D75ABF99D1437F69338F36C684C8831B63C993A96
TLS_CLIENT_NOT_BEFORE 2020-04-05T04:18:22Z
TLS_CLIENT_NOT_AFTER 2021-04-05T04:18:22Z
TLS_CLIENT_SERIAL_NUMBER 73629018972631
TLS_CLIENT_AUTHORISED 0 (not authorised) / 1 (authorised) †

† Requires the server to be configured with a CA for validating client certificates.

Deployment

Jetforce is intended to be run behind a process manager that handles daemonizing the script, redirecting output to system logs, etc. I prefer to use systemd for this because it's installed on my operating system and easy to set up.

Here's how I configure my server over at gemini://mozz.us:

# /etc/systemd/system/jetforce.service
[Unit]
Description=Jetforce Server

[Service]
Type=simple
Restart=always
RestartSec=5
Environment="PYTHONUNBUFFERED=1"
ExecStart=/usr/local/bin/jetforce \
    --host 0.0.0.0 \
    --port 1965 \
    --hostname mozz.us \
    --dir /var/gemini \
    --tls-certfile /etc/letsencrypt/live/mozz.us/fullchain.pem \
    --tls-keyfile /etc/letsencrypt/live/mozz.us/privkey.pem \
    --tls-cafile /etc/pki/tls/jetforce_client/ca.cer

[Install]
WantedBy=default.target
  • --host 0.0.0.0 allows the server to accept external connections from any IP address over IPv4.
  • PYTHONUNBUFFERED=1 disables buffering stderr and is sometimes necessary for logging to work.
  • --tls-certfile and --tls-keyfile point to my WWW server's Let's Encrypt certificate chain.
  • --tls-cafile points to a self-signed CA that I created in order to test accepting client TLS connections.

With this service installed, I can start and stop the server using

systemctl start jetforce
systemctl stop jetforce

And I can view the server logs using

journalctl -u jetforce -f

WARNING

You are exposing a server to the internet. You (yes you!) are responsible for securing your server and setting up appropriate access permissions. This likely means not running jetforce as the root user. Security best practices are outside of the scope of this document and largely depend on your individual threat model.

Releases

To view the project's release history, see the CHANGELOG file.

License

This project is licensed under the Floodgap Free Software License.

The Floodgap Free Software License (FFSL) has one overriding mandate: that software using it, or derivative works based on software that uses it, must be free. By free we mean simply "free as in beer" -- you may put your work into open or closed source packages as you see fit, whether or not you choose to release your changes or updates publicly, but you must not ask any fee for it.

jetforce's People

Contributors

080h avatar dbandstra avatar dbxnr avatar michael-lazar avatar tallship avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

jetforce's Issues

More efficient TCP packets

I have been running some crawling tests, and I noticed that when jetforce generates menu listing using yield, it will send a separate TCP packet for each iteration of the loop a.k.a for each line in the generated menu.

2020-09-24 02:46:33 [mozz_archiver.downloaders] DEBUG: Creating download request for gemini://mozz.us/jetforce/examples/
2020-09-24 02:46:33 [mozz_archiver.downloaders] DEBUG: gemini://mozz.us/jetforce/examples/: Line received
2020-09-24 02:46:33 [mozz_archiver.downloaders] DEBUG: gemini://mozz.us/jetforce/examples/: Data received (31)
2020-09-24 02:46:33 [mozz_archiver.downloaders] DEBUG: gemini://mozz.us/jetforce/examples/: Data received (16)
2020-09-24 02:46:33 [mozz_archiver.downloaders] DEBUG: gemini://mozz.us/jetforce/examples/: Data received (32)
2020-09-24 02:46:33 [mozz_archiver.downloaders] DEBUG: gemini://mozz.us/jetforce/examples/: Data received (46)
2020-09-24 02:46:33 [mozz_archiver.downloaders] DEBUG: gemini://mozz.us/jetforce/examples/: Data received (44)
2020-09-24 02:46:33 [mozz_archiver.downloaders] DEBUG: gemini://mozz.us/jetforce/examples/: Data received (38)
2020-09-24 02:46:33 [mozz_archiver.downloaders] DEBUG: gemini://mozz.us/jetforce/examples/: Data received (48)
2020-09-24 02:46:33 [mozz_archiver.downloaders] DEBUG: gemini://mozz.us/jetforce/examples/: Data received (50)
2020-09-24 02:46:33 [mozz_archiver.downloaders] DEBUG: gemini://mozz.us/jetforce/examples/: Data received (50)
2020-09-24 02:46:33 [mozz_archiver.downloaders] DEBUG: gemini://mozz.us/jetforce/examples/: Data received (50)
2020-09-24 02:46:33 [mozz_archiver.downloaders] DEBUG: gemini://mozz.us/jetforce/examples/: Data received (46)
2020-09-24 02:46:33 [mozz_archiver.downloaders] DEBUG: gemini://mozz.us/jetforce/examples/: Data received (40)
2020-09-24 02:46:33 [mozz_archiver.downloaders] DEBUG: gemini://mozz.us/jetforce/examples/: Connection lost (Connection was closed cleanly.)

I would like to somehow bundle this response to send all in one packet. However, I also want to respect that if there is no more data available (for example if waiting for more output from a CGI script) then the server should go ahead and send what it has. I was kind of hoping that twisted would handle packet optimization under the hood for me, but apparently not 😒

jetforce not serving the full certificate chain

$ openssl s_client -connect mozz.us:1965
CONNECTED(00000005)
depth=0 CN = mozz.us
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = mozz.us
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = mozz.us
verify return:1

https://testtls.com/mozz.us/1965

Chain Of Trust | CRITICAL | failed (chain incomplete).

I'm using my Let's Encrypt full certificate chain in my jetforce settings

certfile = "/etc/letsencrypt/live/mozz.us/fullchain.pem"
keyfile = "/etc/letsencrypt/live/mozz.us/privkey.pem"

It looks like this twisted method in jetforce

ctx.use_certificate_file(self.certfile)

needs to be switched to

ctx.use_certificate_chain_file(self.certfile)

Add more tests to diagnostics script

Some ideas:

  • Write an EOF halfway through the request URL
  • Try sending a transient client certificate
  • Pin the client to TLS v1.1
  • Send a non-empty request which is not a URL of any kind, like "Hello, Gemini!\r\n"?
  • Send a totally empty request?
  • Send a URL with a query string

Automatically reload TLS certificate

I noticed that my TLS certificate was expired on my jetforce server, even though certbot had already refreshed the certificate in my filesystem.

I want to see if there's any way for python to automatically detect the file change and reload without needing to manually restart the server every couple of months.

I can't make jetforce.service work on raspberrypi 1

systemctl status jetforce.service gives me this:

jetforce.service - Jetforce Server
   Loaded: loaded (/etc/systemd/system/jetforce.service; disabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Mon 2021-02-15 16:30:53 WET; 5s ago
  Process: 2596 ExecStart=/home/pi/.local/bin/jetforce --host 0.0.0.0 --port 1965 --hostname 192.168.1.135 --dir /home/pi/gemini/
 Main PID: 2596 (code=exited, status=1/FAILURE)

journalctl -u jetforce -f gives me this:

fev 15 16:28:23 raspberrypi systemd[1]: Started Jetforce Server.
fev 15 16:28:23 raspberrypi jetforce[2545]: Traceback (most recent call last):
fev 15 16:28:23 raspberrypi jetforce[2545]:   File "/home/pi/.local/bin/jetforce", line 6, in <module>
fev 15 16:28:23 raspberrypi jetforce[2545]:     from jetforce.__main__ import main
fev 15 16:28:23 raspberrypi jetforce[2545]: ModuleNotFoundError: No module named 'jetforce'
fev 15 16:28:23 raspberrypi systemd[1]: jetforce.service: Main process exited, code=exited, status=1/FAILURE
fev 15 16:28:23 raspberrypi systemd[1]: jetforce.service: Failed with result 'exit-code'.
fev 15 16:28:29 raspberrypi systemd[1]: jetforce.service: Service RestartSec=5s expired, scheduling restart.
fev 15 16:28:29 raspberrypi systemd[1]: jetforce.service: Scheduled restart job, restart counter is at 232.
fev 15 16:28:29 raspberrypi systemd[1]: Stopped Jetforce Server.

I just changed this bit to this and when I run that command on the terminal it works fine...

ExecStart=/home/pi/.local/bin/jetforce \
    --host 0.0.0.0 \
    --port 1965 \
    --hostname 192.168.1.135 \
    --dir /home/pi/gemini/capsule \
    --tls-certfile /home/pi/gemini/server/certs/192.168.1.135.crt \
    --tls-keyfile /home/pi/gemini/server/certs/192.168.1.135.key

This must not be anything wrong with the repo, but some configuration I'm missing

Response to PermissionError

Currently, jetforce responds with a “An unexpected error occurred” when stumbling upon a file it cannot read (i.e., PermissionError), and while it may be technically correct, perhaps it would be more meaningful to return “51 Not Found”. That way, the gemini captain can have files that are world-unreadable (say, Makefiles or some shellscripts or notes) and that aren't meant for public consumption. The requesting party needn't know exactly why they can't get the file. Jetforce can inform the captain that somebody has requested an unreadable file so that permissions can be corrected if they were wrong, but basically I feel that it would enhance functionality if jetforce would just ignore unreable files and directories.

py.typed file

It appears the project is missing a py.typed file to indicate that the project supports type checking.

v0.2.1 - Link Not found.

Hello,

i have update my jetforce Server from Version 0.2.0 to 0.2.1.

after the update my links are not working anymore.

log v0.2.0 (works)

Mar 31 09:21:25 xxx jetforce: "gemini://xxx" 31 "gemini://xxx/" 23                                                                                                        
Mar 31 09:21:25 xxx jetforce: "gemini://xxx/" 20 "text/gemini" 949                                                                                                             
Mar 31 09:21:27 xxx jetforce: "gemini://xxx/~foo/" 20 "text/gemini" 303

log v0.2.1 (not work)

Mar 31 09:18:27 xxx jetforce: "gemini://xxx" 31 "gemini://xxx/" 23                                                                                                          
Mar 31 09:18:27 xxx jetforce: "gemini://xxx/" 20 "text/gemini" 949                                                                                                              
Mar 31 09:18:30 xxx jetforce: "gemini://xxx/~foo/" 51 "Not Found" 14

Thanks for Help
Best regards, ~creme

Catch OS Error when loading large file

[URLMaxSize] Send a 1024 byte URL, the maximum allowed size
Requesting URL
  'gemini://mozz.us/00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000\r\n'
Response header
  "42\t[Errno 36] File name too long: '/usr/local/www/mozz/gemini/00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'\r\n"
Status should return code 51 (NOT FOUND)
  x '42'

Jetforce should be catching these types of OS errors and returning a 51 NOT FOUND instead of returning sensitive information about the server's file system.

Does v0.3.2 change CGI?

Will CGI variables like QUERY_STRING now be decoded for you? If so, that could break existing apps and I think would be not with the RFC.

Connecting through Bombadillo throws errors in console.

Through Bombadillo, I cannot connect to my own site. Using other clients like, asuka and AV-98, I can easily connect. When I try to connect using Bombadillo, jetforce throws this into the logs:

May 18 15:49:04 hermes jetforce[31639]: Task exception was never retrieved
May 18 15:49:04 hermes jetforce[31639]: future: <Task finished coro=<GeminiServer.accept_connection() done, defined at /usr/local/bin/jetforce:664> exception=ConnectionResetError('Connection lost')>
May 18 15:49:04 hermes jetforce[31639]: Traceback (most recent call last):
May 18 15:49:04 hermes jetforce[31639]:   File "/usr/local/bin/jetforce", line 490, in handle
May 18 15:49:04 hermes jetforce[31639]:     await self.parse_header()
May 18 15:49:04 hermes jetforce[31639]:   File "/usr/local/bin/jetforce", line 547, in parse_header
May 18 15:49:04 hermes jetforce[31639]:     data = await self.reader.readuntil(b"\r\n")
May 18 15:49:04 hermes jetforce[31639]:   File "/usr/lib/python3.7/asyncio/streams.py", line 585, in readuntil
May 18 15:49:04 hermes jetforce[31639]:     raise IncompleteReadError(chunk, None)
May 18 15:49:04 hermes jetforce[31639]: asyncio.streams.IncompleteReadError: 0 bytes read on a total of None expected bytes
May 18 15:49:04 hermes jetforce[31639]: During handling of the above exception, another exception occurred:
May 18 15:49:04 hermes jetforce[31639]: Traceback (most recent call last):
May 18 15:49:04 hermes jetforce[31639]:   File "/usr/local/bin/jetforce", line 672, in accept_connection
May 18 15:49:04 hermes jetforce[31639]:     await request_handler.handle(reader, writer)
May 18 15:49:04 hermes jetforce[31639]:   File "/usr/local/bin/jetforce", line 494, in handle
May 18 15:49:04 hermes jetforce[31639]:     return await self.close_connection()
May 18 15:49:04 hermes jetforce[31639]:   File "/usr/local/bin/jetforce", line 597, in close_connection
May 18 15:49:04 hermes jetforce[31639]:     await self.flush_status()
May 18 15:49:04 hermes jetforce[31639]:   File "/usr/local/bin/jetforce", line 590, in flush_status
May 18 15:49:04 hermes jetforce[31639]:     await self.writer.drain()
May 18 15:49:04 hermes jetforce[31639]:   File "/usr/lib/python3.7/asyncio/streams.py", line 348, in drain
May 18 15:49:04 hermes jetforce[31639]:     await self._protocol._drain_helper()
May 18 15:49:04 hermes jetforce[31639]:   File "/usr/lib/python3.7/asyncio/streams.py", line 202, in _drain_helper
May 18 15:49:04 hermes jetforce[31639]:     raise ConnectionResetError('Connection lost')
May 18 15:49:04 hermes jetforce[31639]: ConnectionResetError: Connection lost

Bomadillo also complains about an expired cert, and that the cert hostname does not match.

cannot start jetforce after upgrade to v0.3.0

hello,

i have update my jetforce server to the current release. after this step i cannot start the service. :/

May 31 08:46:14 core.envs.net systemd[1]: Started Jetforce Server - gemini protocol server.
May 31 08:46:14 core.envs.net jetforce[21206]: Traceback (most recent call last):
May 31 08:46:14 core.envs.net jetforce[21206]:   File "/usr/local/bin/jetforce", line 6, in <module>
May 31 08:46:14 core.envs.net jetforce[21206]:     from jetforce.__main__ import main
May 31 08:46:14 core.envs.net jetforce[21206]:   File "/usr/local/lib/python3.7/dist-packages/jetforce/__init__.py", line 8, in <module>
May 31 08:46:14 core.envs.net jetforce[21206]:     from .protocol import GeminiProtocol
May 31 08:46:14 core.envs.net jetforce[21206]:   File "/usr/local/lib/python3.7/dist-packages/jetforce/protocol.py", line 8, in <module>
May 31 08:46:14 core.envs.net jetforce[21206]:     from twisted.internet.address import IPv4Address, IPv6Address
May 31 08:46:14 core.envs.net jetforce[21206]:   File "/usr/local/lib/python3.7/dist-packages/twisted/internet/address.py", line 101, in <module>
May 31 08:46:14 core.envs.net jetforce[21206]:     @attr.s(hash=False, repr=False, eq=False)
May 31 08:46:14 core.envs.net jetforce[21206]: TypeError: attrs() got an unexpected keyword argument 'eq'
May 31 08:46:14 core.envs.net systemd[1]: jetforce.service: Main process exited, code=exited, status=1/FAILURE
May 31 08:46:14 core.envs.net systemd[1]: jetforce.service: Failed with result 'exit-code'.

[Feature Request] Default Request Handler

Hi,

One feature it would be nice to have is a default request handler, which, once set up, would be called in last resort before returning a "file not found" error.

The idea behind this is to enable a CGI module to handle "fancy" URL routing inside CGI applications (e.g. gemini://example.com/blog/a-new-post/ instead of gemini://example.com/cgi-bin/blog.cgi?a-new-post ) without having to add URL rewriting on your side.

Kind regards

URLs with spaces

It seems like Jetforce isn't respecting URLs with encoded spaces (%20). I'm working on setting up Cosmic Voyage on gemini and many of the ship names (and directories) have spaces. I've encoded those as %20s and tried encoding them as + as well, but neither works.

Interestingly, if I use the auto-generated directory indexing and the files contain spaces a similar problem occurs. In this case the space is literally there, throwing off the labeling and links both.

I think the server just needs to do a little encode/decode work with %20 and it should be good.

Validate port number in URL

A URL with a different port number in it should not succeed:

[URLWrongPort] A URL with an incorrect port number should be rejected
Requesting URL
  'gemini://mozz.us:443/\r\n'
Response header
  '20\ttext/gemini\r\n'
Status should return a failure code (5X PERMANENT FAILURE)
  x Received status of '20'

separate diagnostic script into own project

Please consider spinning out the test script into its own independent git project; it's pretty important for Gemini to have some kind of server conformance testing suite, and such a thing may attract a different developer group.

Using multiple TLS certificates

I saw this on the README:

Jetforce does not (yet) support virtual hosting at the TLS-layer using SNI. This means that you cannot return different server TLS certificates for different domains.

What work is required to achieve this for Jetforce? I wouldn't mind trying to help with this.

Way to add redirects

It'd be nice if there was an easy way to add redirects for Jetforce when in static file serving mode. There could be something like a redirects.jetforce file at the root, that isn't served but instead consulted for redirect info. Ideally it wouldn't require a server restart when adding a new redirect, so maybe it could reload that file every 5 minutes or something.

Another way to do this could be to have a special .redir file type, that just contains the new URL. So I can create a file called example.gmi.redir, with the content of the file being just the line: /path/to/redirect/file.gmi. And when someone requests example.gmi, Jetforce sees the redirect file, and returns an absolute URL redirect: gemini://myhost.com/path/to/redirect/file.gmi.

Do either of those options sound good to you?

Redirect to canonical URL if missing trailing slash

I wrote about this a bit here

https://portal.mozz.us/gemini/mozz.us/diagnostics/2020-01-08/notes.gmi

[HomepageRedirect]

This is not a hard requirement, but it's
(arguably) a good practice in HTTP to
only have one canonical URL for every
resource, and then leverage redirects to
point to that resource. So if I have
two routes:

  • gemini://mozz.us
  • gemini://mozz.us/

Since the resource is actually a
directory and thus should be
represented with a trailing slash, the
first URL should return a redirect to
the second URL instead of responding
with the content directly.

[HomepageRedirect] A URL with no trailing slash should redirect to the canonical resource
Requesting URL
  'gemini://mozz.us\r\n'
Response header
  '20\ttext/gemini\r\n'
Status should return code 31 (REDIRECT PERMANENT)
  x '20'
Meta should redirect to location "gemini://[hostname]/"
  x 'text/gemini'
Header should end with "\r\n"
  ✓ '\r\n'
Body should be empty
  x '=== WELCOME TO MOZZ.US ===\n___________..._\n_______'

PATH_INFO strips trailing slash

PATH_INFO should be preserving trailing slashes in the URL after the script name.

$ jetforce-client "gemini://mozz.us/cgi-bin/debug.py"
20 text/plain
GEMINI_URL  : gemini://mozz.us/cgi-bin/debug.py
SCRIPT_NAME : /cgi-bin/debug.py
PATH_INFO   :

$ jetforce-client "gemini://mozz.us/cgi-bin/debug.py/"
20 text/plain
GEMINI_URL  : gemini://mozz.us/cgi-bin/debug.py/
SCRIPT_NAME : /cgi-bin/debug.py
PATH_INFO   :

$ jetforce-client "gemini://mozz.us/cgi-bin/debug.py/hello"
20 text/plain
GEMINI_URL  : gemini://mozz.us/cgi-bin/debug.py/hello
SCRIPT_NAME : /cgi-bin/debug.py
PATH_INFO   : /hello

$ jetforce-client "gemini://mozz.us/cgi-bin/debug.py/hello/"
20 text/plain
GEMINI_URL  : gemini://mozz.us/cgi-bin/debug.py/hello/
SCRIPT_NAME : /cgi-bin/debug.py
PATH_INFO   : /hello

Reverse-proxying

Is there an easy way to set up a reverse proxy in Jetforce, maybe using a CompositeApplication? So all requests for example.com use the static file server or some Jetforce app, and all the requests for app.example.com are proxyed to localhost:12345 or whatever.

Would client certs be carried over too?

Missing scheme should be inferred as "gemini://"

1.2 Gemini requests

Gemini requests are a single CRLF-terminated line with the following
structure:

is a UTF-8 encoded absolute URL, of maximum length 1024 bytes.
If the scheme of the URL is not specified, a scheme of gemini:// is
implied.

[URLSchemeMissing] A URL without a scheme should be inferred as gemini
Requesting URL
  '//mozz.us/\r\n'
Response header
  '50\tUnrecognized URL\r\n'
Status should return a success code (20 SUCCESS)
  x Received status of '50'

Binding to IPv4 and IPv6 works differently than in the examples

After some fiddling around I found that:

  • binding to "0.0.0.0" accepts IPv4 connections (as expected
  • binding to "" gives an error (it says that the IPv6 address is already in use)
  • binding to "::" accepts both IPv4 and IPv6 connections (even more surprisingly)

When binding to "::", the IPv4 addresses are displayed as ::ffff:139.162.187.208 in the log.

Maybe we should just update the documentation to reflect this. Or is there something that should be changed in the code?

Response address

I have jetforce v0.2.3 listening on [::] only, and serving contents for a hostname that resolves to two different IPv6 addresses: a traditional public IPv6 address and an yggdrasil address in the CIDR block 0200::/7.

More often than not, when a client is connecting over the yggdrasil address, jetforce will seemingly (if the logfile is any indication) respond over traditional IPv6, and I have also seen it the other way around.

It this just an artifact because jetforce looks up the IP address of its hostname and uses whatever comes first to write its log message, or does it really respond over a different address than the request (that can't happen with TLS, can it?)?

Cheers.

Wrong MIME type ‘text/plain’ assigned to some ebooks

I have a few ebooks for download in EPUB, MOBI and PDF formats over the Gemini protocol.

Jetforce v0.5.0 logs:

 […] "gemini://example.net/ebook.epub" 20 text/plain 1738936
 […] "gemini://example.net/ebook.mobi" 20 text/plain 1597628
 […] "gemini://example.net/ebook.pdf" 20 application/pdf 2930121

However,

>>> import mimetypes
>>> mimetypes.guess_type("gemini://example.net/ebook.epub")
('application/epub+zip', None)
>>> mimetypes.guess_type("gemini://example.net/ebook.mobi")
('application/x-mobipocket-ebook', None)
>>> mimetypes.guess_type("gemini://example.net/ebook.pdf")
('application/pdf', None)

Does jetforce give up too early and wrongfully assigns the MIME type ‘text/plain’ to some formats?

Failure to install pyasn1 / cryptography / setuptools_rust

Following combination of virtualenv and git clone instructions leads to:

error: The 'pyasn1' distribution was not found and is required by service-identity

Commands typed:

cd projects/
git clone https://github.com/michael-lazar/jetforce
cd jetforce/
python3 -m venv venv
. ./venv/bin/activate
python setup.py install

Expected result: installation success, jetforce at venv/bin/jetforce

Actual result: error regarding 'pyasn1'.

Python version used: 3.7.3

Trying the obvious fix (adding pyasn1 to setup.py install_requires) resulted in a second error while building cryptography (3.4.4):

ModuleNotFoundError: No module named 'setuptools_rust'

This is I think an issue with the 'cryptography' package not properly identifying its own build requirements to include setuptools-rust. For jetforce, perhaps either documenting this issue, or requiring setuptools-rust in its own installRequires (undesired) until cryptography fixes this issue.

Sending a client certificate causes an error, and no response

I tried to send a client cert to my server and the page wouldn't load. This is the error left in the log:

May 18 12:02:56 jupiter jetforce[21138]: SSL handshake failed on verifying the certificate
May 18 12:02:56 jupiter jetforce[21138]: protocol: <asyncio.sslproto.SSLProtocol object at 0x7f01f90c9400>
May 18 12:02:56 jupiter jetforce[21138]: transport: <_SelectorSocketTransport fd=9 read=polling write=<idle, bufsize=0>>
May 18 12:02:56 jupiter jetforce[21138]: Traceback (most recent call last):
May 18 12:02:56 jupiter jetforce[21138]:   File "/usr/lib/python3.7/asyncio/sslproto.py", line 625, in _on_handshake_complete
May 18 12:02:56 jupiter jetforce[21138]:     raise handshake_exc
May 18 12:02:56 jupiter jetforce[21138]:   File "/usr/lib/python3.7/asyncio/sslproto.py", line 189, in feed_ssldata
May 18 12:02:56 jupiter jetforce[21138]:     self._sslobj.do_handshake()
May 18 12:02:56 jupiter jetforce[21138]:   File "/usr/lib/python3.7/ssl.py", line 763, in do_handshake
May 18 12:02:56 jupiter jetforce[21138]:     self._sslobj.do_handshake()
May 18 12:02:56 jupiter jetforce[21138]: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
May 18 12:02:56 jupiter jetforce[21138]: SSL error in data received
May 18 12:02:56 jupiter jetforce[21138]: protocol: <asyncio.sslproto.SSLProtocol object at 0x7f01f90c9400>
May 18 12:02:56 jupiter jetforce[21138]: transport: <_SelectorSocketTransport closing fd=9 read=idle write=<idle, bufsize=0>>
May 18 12:02:56 jupiter jetforce[21138]: Traceback (most recent call last):
May 18 12:02:56 jupiter jetforce[21138]:   File "/usr/lib/python3.7/asyncio/sslproto.py", line 526, in data_received
May 18 12:02:56 jupiter jetforce[21138]:     ssldata, appdata = self._sslpipe.feed_ssldata(data)
May 18 12:02:56 jupiter jetforce[21138]:   File "/usr/lib/python3.7/asyncio/sslproto.py", line 189, in feed_ssldata
May 18 12:02:56 jupiter jetforce[21138]:     self._sslobj.do_handshake()
May 18 12:02:56 jupiter jetforce[21138]:   File "/usr/lib/python3.7/ssl.py", line 763, in do_handshake
May 18 12:02:56 jupiter jetforce[21138]:     self._sslobj.do_handshake()
May 18 12:02:56 jupiter jetforce[21138]: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
May 18 12:03:00 jupiter jetforce[21138]: SSL handshake failed on verifying the certificate
May 18 12:03:00 jupiter jetforce[21138]: protocol: <asyncio.sslproto.SSLProtocol object at 0x7f01f90a3a58>
May 18 12:03:00 jupiter jetforce[21138]: transport: <_SelectorSocketTransport fd=9 read=polling write=<idle, bufsize=0>>
May 18 12:03:00 jupiter jetforce[21138]: Traceback (most recent call last):
May 18 12:03:00 jupiter jetforce[21138]:   File "/usr/lib/python3.7/asyncio/sslproto.py", line 625, in _on_handshake_complete
May 18 12:03:00 jupiter jetforce[21138]:     raise handshake_exc
May 18 12:03:00 jupiter jetforce[21138]:   File "/usr/lib/python3.7/asyncio/sslproto.py", line 189, in feed_ssldata
May 18 12:03:00 jupiter jetforce[21138]:     self._sslobj.do_handshake()
May 18 12:03:00 jupiter jetforce[21138]:   File "/usr/lib/python3.7/ssl.py", line 763, in do_handshake
May 18 12:03:00 jupiter jetforce[21138]:     self._sslobj.do_handshake()
May 18 12:03:00 jupiter jetforce[21138]: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
May 18 12:03:00 jupiter jetforce[21138]: SSL error in data received
May 18 12:03:00 jupiter jetforce[21138]: protocol: <asyncio.sslproto.SSLProtocol object at 0x7f01f90a3a58>
May 18 12:03:00 jupiter jetforce[21138]: transport: <_SelectorSocketTransport closing fd=9 read=idle write=<idle, bufsize=0>>
May 18 12:03:00 jupiter jetforce[21138]: Traceback (most recent call last):
May 18 12:03:00 jupiter jetforce[21138]:   File "/usr/lib/python3.7/asyncio/sslproto.py", line 526, in data_received
May 18 12:03:00 jupiter jetforce[21138]:     ssldata, appdata = self._sslpipe.feed_ssldata(data)
May 18 12:03:00 jupiter jetforce[21138]:   File "/usr/lib/python3.7/asyncio/sslproto.py", line 189, in feed_ssldata
May 18 12:03:00 jupiter jetforce[21138]:     self._sslobj.do_handshake()
May 18 12:03:00 jupiter jetforce[21138]:   File "/usr/lib/python3.7/ssl.py", line 763, in do_handshake
May 18 12:03:00 jupiter jetforce[21138]:     self._sslobj.do_handshake()
May 18 12:03:00 jupiter jetforce[21138]: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)

The behaviour I would expect is that the client cert is just ignored, or not verified, etc. This is problem for me right now because the Bombadillo client sends your specified client cert with every request, so when I add a client cert, maybe for astrobotany or something, all Jetforce servers break.

ModuleNotFoundError for packages pip installed system-wide

Given a Python CGI script, containing, e.g.

import qrcode

...Where qrcode was installed with sudo pip install qrcode, and can be found in /usr/local/lib/python3.8/dist-packages, where it is available to an interactive Python session or the same script executed from the command line;

When the CGI script is executed by Jetforce, a ModuleNotFoundError is triggered.

By placing the following code at the top of my script:

import sys
print('20 text/plain\r')
print(sys.path)

...I was able to determine that the Python environment looks like this:
/srv/gemini/cgi-bin/ /lib/python38.zip /lib/python3.8 /lib/python3.8/lib-dynload /lib/python3/dist-packages

See that /usr/local/lib/python3.8/dist-packages is missing.

It seems to be impossible to modify the module search paths with the PYTHONPATH environment variable in the systemd service.

How might i be able to work around the problem and make use of Python modules not in the standard library in my CGI script?

Jetforce allows for arbitrary directory traversal, causing big security issues

Jetforce makes it easy to do arbitrary filesystem traversal by using relative links outside of the root of the gemini home.
For instance, you can browse "gemini://tilde.black/../../etc/motd"

Running the server as a restricted access daemon user helps, but it still has issues. Since the user running jetforce needs to be able to read the public & private keys for the server, and because those people browsing your gemini server are doing so AS that user, if the path is known your gemini server's private key can be read in plain text directly over gemini.

We need a way to hard-limit directory traversal so it cannot go 'up' from the root directory.

Licensing question

I asked some questions about licensing here and it never got answered. Do you think you could take the time to answer them? I appreciate the work you've done with Jetforce, thank you for writing it. I just want to understand some of the things you said.

errors in the systemd service example

systemd on my Rapberry Pi (raspios buster) doesn't understand the [Install] directive.

When I comment out these lines it works:

[Install]
WantedBy=default.target

Are there any consequences of leaving WantedBy out?

Support for "extra-path" component from RFC3875

RFC 3875 supports an "extra-path" component for CGI scripts; i.e. one can call a CGI script with a URL like gemini://mozz.us/cgi-bin/debug.cgi/this/is/extra?foobar and the server would identify debug.cgi as the CGI script to execute, passing the "/this/is/extra" path along as PATH_INFO variable.

Is this something that could be considered for jetforce? Or is this something that would be alien to the Gemini protocol? It would enable the use of a single CGI script that would manage e.g. comments for all blog entries in a personal blog, by passing the blog post in question as extra path.

Jetforce is not (ever?) closing sockets

Hi,
I wrote https://github.com/snoe/deedum and I kept running into sites that would hang, one of them was mozz.us, and the other was tilde.black - when the operator of tilde.black said he was also using jetforce I decided to dig into the problem from the server side, since I had pretty much exhausted figuring it out from the client side.

It seems like (perhaps based on some tls configurations or other voodoo) calling self.transport.loseConnection() in twisted does not always lose the connection. However calling self.transport.transport.loseConnection() seems to bypass this tls shutdown and properly sends a FIN packet.

Here's two wireshark captures with this difference:

diff --git a/jetforce/protocol.py b/jetforce/protocol.py
index dfc53c0..832e305 100644
--- a/jetforce/protocol.py
+++ b/jetforce/protocol.py
@@ -135,7 +135,7 @@ class GeminiProtocol(LineOnlyReceiver):
         finally:
             self.flush_status()
             self.log_request()
-            self.transport.loseConnection()
+            self.transport.transport.loseConnection()

     async def track_deferred(self, deferred: Deferred):
         self._currently_deferred = deferred

BEFORE:
transport

AFTER:
transport transport

Rate-limiting

In light of the recent Gemini attack by 70.113.100.216, it'd be great to have rate-limiting built in to Jetforce, ideally in a configurable way. Maybe something like a limit on requests per second per IP?

Solderpunk mentioned:

serving up code 44 with no response body over and over again rather than serving up actual resources

which seems like a viable course of action to me. I wonder if there's a place for simply not even responding to the request though.

Split out access logs from error logs

I like the simplicity of writing log messages directly to the stream instead of managing log files within the application.

However, I have been thinking that it would be nice to separate the access logs from exceptions and other messages. The access logs should be kept clean so they are easy to parse. My idea is basically to move the access logs to stdout, and keep the startup messages and stack traces on stderr. Supervisors like systemd should make it easy to redirect the different streams to different log managers.

Invalid URLs should return status 59

Any URL that does not conform to RFC-3986, or this section of the gemini spec

<URL> is a UTF-8 encoded absolute URL, of maximum length 1024 bytes.
If the scheme of the URL is not specified, a scheme of gemini:// is
implied.

should return a status of 59 BAD REQUEST. This is more specific than the 50 PERMANENT FAILURE that is currently returned by jetforce.

Add example for binding to IPv6

Currently the documentation gives an example for binding the server to an IPv4 address:

ExecStart=/usr/local/bin/jetforce \
    --host 0.0.0.0 \
    --port 1965 \
    --hostname mozz.us \
    --dir /var/gemini \
    --tls-certfile /etc/letsencrypt/live/mozz.us/fullchain.pem \
    --tls-keyfile /etc/letsencrypt/live/mozz.us/privkey.pem \
    --tls-cafile /etc/pki/tls/jetforce_client/ca.cer

It would be nice if there was also an example of how to bind to a IPv6 or dualstack.

https://docs.python.org/3/library/socket.html#socket.create_server

Status.REDIRECT_TEMPORARY doesn't work when redirecting to another site.

I tried to make an simple link shortener, that reads the file with the link, and redirects the user. Hovewer, all sites, that I tried redirecting to, throw an NOT_FOUND error.

from jetforce import GeminiServer, Response, StaticDirectoryApplication, Status

app = StaticDirectoryApplication(".")
@app.route("/r/(?P<route>.*)")
def redirect(request, route):
	print(f"Route: {route}")
	try:
		with open("r/"+route, "r") as file:
			link = file.read()
		print(f"Responding with {link}")
	except FileNotFoundError:
		print("Responding with 51 NOT_FOUND")
		return Response(Status.NOT_FOUND, "Link file not found. Either the link doesn't exist, or you made an typo.")
	return Response(Status.REDIRECT_TEMPORARY, link)

if __name__ == "__main__":
	server = GeminiServer(app, host="127.0.0.1", hostname="localhost")
	server.run()

add a option to see the 'jetforce' version

I have just looked at jetforce and find it very well done! One thing I noticed is that can we please add an option to display the jetforce version? Like: ‘jetforce -v’

Thanks
~creme

Support python 3.8

Python 3.8 has been released and should be officially supported.

Shebangs that explicitly target python3.7 should be removed in favor of

# /usr/local/env python3

INDEX-FILE not running as CGI

I have a CGI file running smooth if requested directly by the client, but, if set as the INDEX-FILE, its raw code is responded as text/gemini instead.

I guess it has to do with the appended part of the code, as filename being parsed different from index_file, it jumps straight to the INDEX-FILE if-clause.

filename = pathlib.Path(os.path.normpath(str(url_path)))

index_file = filesystem_path / self.index_file

if index_file.exists():

Add automated smoke test

I don't really want to invest in a full test suite just yet, but it would be nice to have a basic smoke test setup w/ Github actions to verify that the code at least runs against different supported versions of python and passes mypy.

Python version check doesn't check

I accidentally ran jetforce with an older version of Python 3 and got a SyntaxError rather than the intended warning message.

Gemini$ git clone https://github.com/michael-lazar/jetforce.git
Cloning into 'jetforce'...
<snip>

Gemini$ cd jetforce/

Gemini/jetforce$ python3 jetforce.py
  File "jetforce.py", line 73
    """
      ^
SyntaxError: invalid syntax

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.