Code Monkey home page Code Monkey logo

aiosmtpd's Introduction

aiosmtpd - An asyncio based SMTP server

Project License on GitHub   PyPI Package   Supported Python Versions   Supported Python Implementations
GitHub CI status   CodeQL status   Code Coverage   Documentation Status

GitHub Discussions

The Python standard library includes a basic SMTP_ server in the smtpd_ module, based on the old asynchronous libraries asyncore_ and asynchat_. These modules are quite old and are definitely showing their age; asyncore and asynchat are difficult APIs to work with, understand, extend, and fix. (And have been deprecated since Python 3.6, and will be removed in Python 3.12.)

With the introduction of the asyncio_ module in Python 3.4, a much better way of doing asynchronous I/O is now available. It seems obvious that an asyncio-based version of the SMTP and related protocols are needed for Python 3. This project brings together several highly experienced Python developers collaborating on this reimplementation.

This package provides such an implementation of both the SMTP and LMTP protocols.

Full documentation is available on aiosmtpd.readthedocs.io_

Requirements

Supported Platforms

aiosmtpd has been tested on CPython>=3.8 and PyPy_>=3.8 for the following platforms (in alphabetical order):

  • Cygwin (as of 2022-12-22, only for CPython 3.8, and 3.9)
  • MacOS 11 and 12
  • Ubuntu 18.04
  • Ubuntu 20.04
  • Ubuntu 22.04
  • Windows 10
  • Windows Server 2019
  • Windows Server 2022

aiosmtpd probably can run on platforms not listed above, but we cannot provide support for unlisted platforms.

Installation

Install as usual with pip:

pip install aiosmtpd

If you receive an error message ModuleNotFoundError: No module named 'public', it likely means your setuptools is too old; try to upgrade setuptools to at least version 46.4.0 which had implemented a fix for this issue.

Project details

As of 2016-07-14, aiosmtpd has been put under the aio-libs_ umbrella project and moved to GitHub.

The best way to contact the developers is through the GitHub links above. You can also request help by submitting a question on StackOverflow.

Building

You can install this package in a virtual environment like so:

$ python3 -m venv /path/to/venv
$ source /path/to/venv/bin/activate
$ python setup.py install

This will give you a command line script called aiosmtpd which implements the SMTP server. Use aiosmtpd --help for a quick reference.

You will also have access to the aiosmtpd library, which you can use as a testing environment for your SMTP clients. See the documentation links above for details.

Developing

You'll need the tox tool to run the test suite for Python 3. Once you've got that, run:

$ tox

Individual tests can be run like this:

$ tox -- <testname>

where <testname> is the "node id" of the test case to run, as explained in the pytest documentation. The command above will run that one test case against all testenvs defined in tox.ini (see below).

If you want test to stop as soon as it hit a failure, use the -x/--exitfirst option:

$ tox -- -x

You can also add the -s/--capture=no option to show output, e.g.:

$ tox -e py311-nocov -- -s

and these options can be combined:

$ tox -e py311-nocov -- -x -s <testname>

(The -e parameter is explained in the next section about 'testenvs'. In general, you'll want to choose the nocov testenvs if you want to show output, so you can see which test is generating which output.)

Supported 'testenvs'

In general, the -e parameter to tox specifies one (or more) testenv to run (separate using comma if more than one testenv). The following testenvs have been configured and tested:

  • {py38,py39,py310,py311,py312,pypy3,pypy37,pypy38,pypy39}-{nocov,cov,diffcov,profile}

    Specifies the interpreter to run and the kind of testing to perform.

    • nocov = no coverage testing. Tests will run verbosely.
    • cov = with coverage testing. Tests will run in brief mode (showing a single character per test run)
    • diffcov = with diff-coverage report (showing difference in coverage compared to previous commit). Tests will run in brief mode
    • profile = no coverage testing, but code profiling instead. This must be invoked manually using the -e parameter

    Note 1: As of 2021-02-23, only the {py38,py39}-{nocov,cov} combinations work on Cygwin.

    Note 2: It is also possible to use whatever Python version is used when invoking tox by using the py target, but you must explicitly include the type of testing you want. For example:

    $ tox -e "py-{nocov,cov,diffcov}"

    (Don't forget the quotes if you want to use braces!)

    You might want to do this for CI platforms where the exact Python version is pre-prepared, such as Travis CI or GitHub Actions_; this will definitely save some time during tox's testenv prepping.

    For all testenv combinations except diffcov, bandit_ security check will also be run prior to running pytest.

  • qa

    Performs flake8_ code style checking, and flake8-bugbear_ design checking.

    In addition, some tests to help ensure that aiosmtpd is releasable to PyPI are also run.

  • docs

    Builds HTML documentation and manpage using Sphinx. A pytest doctest will run prior to actual building of the documentation.

  • static

    Performs a static type checking using pytype.

    Note 1: Please ensure that all pytype dependencies have been installed before executing this testenv.

    Note 2: This testenv will be _SKIPPED on Windows, because pytype currently cannot run on Windows.

    Note 3: This testenv does NOT work on Cygwin.

Environment Variables

ASYNCIO_CATCHUP_DELAY

Due to how asyncio event loop works, some actions do not instantly get responded to. This is especially so on slower / overworked systems. In consideration of such situations, some test cases invoke a slight delay to let the event loop catch up.

Defaults to 0.1 and can be set to any float value you want.

Different Python Versions

The tox configuration files have been created to cater for more than one Python versions `safely`: If an interpreter is not found for a certain Python version, tox will skip that whole testenv.

However, with a little bit of effort, you can have multiple Python interpreter versions on your system by using pyenv. General steps:

  1. Install pyenv from https://github.com/pyenv/pyenv#installation
  2. Install tox-pyenv from https://pypi.org/project/tox-pyenv/
  3. Using pyenv, install the Python versions you want to test on
  4. Create a .python-version file in the root of the repo, listing the Python interpreter versions you want to make available to tox (see pyenv's documentation about this file)

    Tip: The 1st line of .python-version indicates your preferred Python version which will be used to run tox.

  5. Invoke tox with the option --tox-pyenv-no-fallback (see tox-pyenv's documentation about this option)

housekeep.py

If you ever need to 'reset' your repo, you can use the housekeep.py utility like so:

$ python housekeep.py superclean

It is strongly recommended to NOT do superclean too often, though. Every time you invoke superclean, tox will have to recreate all its testenvs, and this will make testing much longer to finish.

superclean is typically only needed when you switch branches, or if you want to really ensure that artifacts from previous testing sessions won't interfere with your next testing sessions.

For example, you want to force Sphinx to rebuild all documentation. Or, you're sharing a repo between environments (say, PSCore and Cygwin) and the cached Python bytecode messes up execution (e.g., sharing the exact same directory between Windows PowerShell and Cygwin will cause problems as Python becomes confused about the locations of the source code).

Signing Keys

Starting version 1.3.1, files provided through PyPI or GitHub Releases will be signed using one of the following GPG Keys:

GPG Key ID Owner Email
5D60 CE28 9CD7 C258 Pandu E POLUAN pepoluan at gmail period com
5555 A6A6 7AE1 DC91 Pandu E POLUAN pepoluan at gmail period com
E309 FD82 73BD 8465 Wayne Werner waynejwerner at gmail period com
5FE9 28CD 9626 CE2B Sam Bull sam at sambull period org

License

aiosmtpd is released under the Apache License version 2.0.

aiosmtpd's People

Contributors

akuchling avatar alefteris avatar cuu508 avatar dependabot[bot] avatar dreamsorcerer avatar dvzrv avatar emersion avatar hrnciar avatar j08ny avatar jaraco avatar kasimov-maxim avatar kozzztik avatar kz26 avatar lgtm-com[bot] avatar manfred-kaiser avatar matrixise avatar maxking avatar mortal avatar pepoluan avatar scop avatar simonklb avatar sirkonst avatar the-login avatar thperret avatar timwolla avatar varbin avatar warsaw avatar waynew avatar wevsty avatar zvyn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aiosmtpd's Issues

Send 550 in response to SMTP RCPT

"Real" SMTP servers often reply with 550 in response to the RCPT command, so that the SMTP client doesn't have to send message data for a recipient that doesn't exist on the server. Would it be possible to add a hook in aiosmtpd.smtp.SMTP.smtp_DATA to validate incoming recipients immediately instead of just adding them blindly to envelope.rcpt_tos?

I've been running an SMTP mailing list forwarder based on smtpd for two years and am looking into porting that to aiosmtpd when 1.0 is released.

500 Error (NameError) name 'error_occurred' is not defined

Using telnet I send a EHLO, I sent a MAIL FROM and RCPT TO... I then send DATA and enter a data line... but as soon as I hit enter it throws this error inside of telnet:

500 Error (NameError) name 'error_occurred' is not defined

The python console shows the following:

SMTP session exception
Traceback (most recent call last):
File "C:\Program Files\Python3.6\python-3.6.1.amd64\aiosmtpd\smtp.py", line 281, in _handle_client
await method(arg)
File "C:\Program Files\Python3.6\python-3.6.1.amd64\aiosmtpd\smtp.py", line 657, in smtp_DATA
status = await self._call_handler_hook('DATA')
File "C:\Program Files\Python3.6\python-3.6.1.amd64\aiosmtpd\smtp.py", line 118, in _call_handler_hook
status = await hook(self, self.session, self.envelope, *args)
File "smtprelay.py", line 13, in handle_DATA
if error_occurred:
NameError: name 'error_occurred' is not defined

The code that is currently being run was pulled straight from the documenation:
http://aiosmtpd.readthedocs.io/en/latest/aiosmtpd/docs/migrating.html

Any idea what is going on here?

Duplicated HELO/EHLO

As already mentioned in source code (

# See issue #21783 for a discussion of this behavior.
, also in original smtpd's code) a second HELO/EHLO is rejected but the standart says it should be handled as a RSET command.

Is there any special reason to not allow duplicated HELO/EHLO? I think the best way is to do the same other SMTP servers do: Allowing a duplicated HELO/EHLO.
Another option is to have a configuration option.

if client closed connection during session, there is error loop

If client close connection during session there is error loop like:

WARNING:asyncio:socket.send() raised exception.
DEBUG:mail.log:b'500 Error: bad syntax\r\n'
INFO:mail.log:Data: ''
WARNING:asyncio:socket.send() raised exception.
DEBUG:mail.log:b'500 Error: bad syntax\r\n'
INFO:mail.log:Data: ''
WARNING:asyncio:socket.send() raised exception.
DEBUG:mail.log:b'500 Error: bad syntax\r\n'
INFO:mail.log:Data: ''
WARNING:asyncio:socket.send() raised exception.
DEBUG:mail.log:b'500 Error: bad syntax\r\n'
INFO:mail.log:Data: ''

I found that, stream reader's read inside a loop, and there is no check for exception. I create a pull request.

#61

Best regards

EHLO command returns all bunch of 250- data, even when the hooks returns otherwise

In the current state of the code, some data are pushed BEFORE calling the hook, including

    250-{hostname}
    250-SIZE ...
    250-8BITMIME
    250-SMTPUTF8
    250-STARTTLS

And AFTER that, the hook is called.

The issue is that if the hook doesn't return a 2xx response, the server sends a mixed response and makes the clients fails.

The EHLO command should call the hook, and depending on the response, send all the push commands afterward.

Connection remains open after `QUIT`

Reproduce steps:

  1. Run basic SMTP server
controller = Controller(Message())
controller.start()
  1. Connect to it with nc -vC 127.0.0.1 8025
  2. Send QUIT command

The server prints 221 Bye, but the connection remains open (server should close connection)
The original smtpd server behaves correctly (python -m smtpd -nd -c DebuggingServer)

If the connection is broken, an exception will be generated

I'm having some problems with my code, check the systemd log as follows.

Feb 06 15:21:23 hostname-vm start_filter.sh[20874]: Exception in handle_exception()
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]: Traceback (most recent call last):
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:   File "/usr/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 315, in _handle_client
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:     await method(arg)
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:   File "/usr/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 710, in smtp_DATA
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:     await self.push('250 OK' if status is MISSING else status)
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:   File "/usr/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 224, in push
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:     await self._writer.drain()
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:   File "/usr/lib/python3.6/asyncio/streams.py", line 339, in drain
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:     yield from self._protocol._drain_helper()
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:   File "/usr/lib/python3.6/asyncio/streams.py", line 210, in _drain_helper
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:     raise ConnectionResetError('Connection lost')
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]: ConnectionResetError: Connection lost
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]: During handling of the above exception, another exception occurred:
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]: Traceback (most recent call last):
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:   File "/usr/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 326, in _handle_client
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:     await self.push(status)
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:   File "/usr/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 224, in push
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:     await self._writer.drain()
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:   File "/usr/lib/python3.6/asyncio/streams.py", line 339, in drain
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:     yield from self._protocol._drain_helper()
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:   File "/usr/lib/python3.6/asyncio/streams.py", line 210, in _drain_helper
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:     raise ConnectionResetError('Connection lost')
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]: ConnectionResetError: Connection lost
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]: Exception in handle_exception()
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]: Traceback (most recent call last):
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:   File "/usr/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 315, in _handle_client
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:     await method(arg)
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:   File "/usr/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 710, in smtp_DATA
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:     await self.push('250 OK' if status is MISSING else status)
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:   File "/usr/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 224, in push
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:     await self._writer.drain()
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:   File "/usr/lib/python3.6/asyncio/streams.py", line 339, in drain
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:     yield from self._protocol._drain_helper()
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:   File "/usr/lib/python3.6/asyncio/streams.py", line 210, in _drain_helper
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:     raise ConnectionResetError('Connection lost')
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]: ConnectionResetError: Connection lost
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]: During handling of the above exception, another exception occurred:
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]: Traceback (most recent call last):
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:   File "/usr/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 326, in _handle_client
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:     await self.push(status)
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:   File "/usr/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 224, in push
Feb 06 15:21:23 hostname-vm start_filter.sh[20874]:     await self._writer.drain()
Feb 06 15:36:31 hostname-vm systemd-journald[379]: Suppressed 7620 messages from start_filter.sh.service
Feb 06 15:36:31 hostname-vm start_filter.sh[20874]: /opt/filter/start_filter.sh: line 10: 20877 Killed                  python ./asyncio_smtpd_filter.py
Feb 06 15:36:31 hostname-vm systemd[1]: start_filter.sh.service: Main process exited, code=exited, status=137/n/a
Feb 06 15:36:31 hostname-vm systemd[1]: start_filter.sh.service: Failed with result 'exit-code'.

We can see a lot of ConnectionResetError, and a ConnectionResetError reports an error multiple times.

Environment:
aiosmtpd version is 1.2

ConnectionResetError when client disconnects abruptly

Hi,

when a client disconnects abruptly, aiosmtpd logs an unhandled ConnectionResetError, that follows an attempt to return a status code to the client once it has already gone.

I managed to reproduce the issue quite consistently with the following handler and CLI command:

import asyncio
from concurrent.futures import CancelledError, ProcessPoolExecutor


def func():
    pass


class Handler:
    def __init__(self):
        self._executor = ProcessPoolExecutor(4)

    async def handle_DATA(self, server, session, envelope):
        loop = asyncio.get_event_loop()
        try:
            await loop.run_in_executor(self._executor, func)
        except CancelledError:
            print('cancelled')
        return '250 OK'

CLI: python -m aiosmtpd -n -c handler.Handler

I used a benchmarking tool (https://github.com/nabeken/go-smtp-source) as client, which sends a user defined number of messages of a certain size, concurrently.
If I stop the test hitting CTRL-C on the client, I get the following output from aiosmtpd (sometimes it takes a couple of retries to reproduce, but it should be quite easy):

cancelled
ERROR:mail.log:SMTP session exception
Traceback (most recent call last):
  File "/home/fabio/.virtualenvs/mailsampler/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 281, in _handle_client
    await method(arg)
  File "/home/fabio/.virtualenvs/mailsampler/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 677, in smtp_DATA
    await self.push('250 OK' if status is MISSING else status)
  File "/home/fabio/.virtualenvs/mailsampler/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 203, in push
    await self._writer.drain()
  File "/opt/pyenv/versions/3.6.2/lib/python3.6/asyncio/streams.py", line 323, in drain
    raise exc
  File "/opt/pyenv/versions/3.6.2/lib/python3.6/asyncio/selector_events.py", line 724, in _read_ready
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
ERROR:mail.log:Exception in handle_exception()
Traceback (most recent call last):
  File "/home/fabio/.virtualenvs/mailsampler/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 292, in _handle_client
    await self.push(status)
  File "/home/fabio/.virtualenvs/mailsampler/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 203, in push
    await self._writer.drain()
  File "/opt/pyenv/versions/3.6.2/lib/python3.6/asyncio/streams.py", line 323, in drain
    raise exc
  File "/home/fabio/.virtualenvs/mailsampler/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 281, in _handle_client
    await method(arg)
  File "/home/fabio/.virtualenvs/mailsampler/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 677, in smtp_DATA
    await self.push('250 OK' if status is MISSING else status)
  File "/home/fabio/.virtualenvs/mailsampler/lib/python3.6/site-packages/aiosmtpd/smtp.py", line 203, in push
    await self._writer.drain()
  File "/opt/pyenv/versions/3.6.2/lib/python3.6/asyncio/streams.py", line 323, in drain
    raise exc
  File "/opt/pyenv/versions/3.6.2/lib/python3.6/asyncio/selector_events.py", line 724, in _read_ready
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer

Am I missing something? Can this be confirmed?

Thank you very much for your work on aiosmtpd!

Set the connection identifier without setting a module global

Currently, there's an __version__ module global in smtp.py that is used when the client connects to the SMTP instance. There's also an unused module global __ident__. It should be possible to set these as attributes or arguments to the SMTP class, or at least without having to modify the module global.

Blocking `start()` command?

Hi!

I plan to run my code on a server where the server would run indefinitely, using Supervisor to handle the start/restart.

Reading the code and documentation, I haven't found a way to call controller.start() and have it blocked "forever". I need to do:

controller.start()
input('Press enter to quit')
controller.stop()

which is counter intuitive for an always running server.

Maybe I'm missing something too.

Taking a look back at the smtplib, there was a way to run for a number of iterations, which when set to None meant "running forever".

Having something similar here would be nice controller.start(blocking=True, loop=5000000) for instance.

Is there a way to achieve something similar?

`handle_tls_handshake()` hook should take a session object

Right now, it is passed the session.ssl attribute, but really it should get passed the whole session object. We can make this change without breaking backward compatibility, since it was added post 1.0a4 and 1.0a5 has not yet been released.

We're messing up STARTTLS somehow

Previously in eof_received() I added a conditional around the return super().eof_received() call to avoid it if self.transport was None. I've since removed that, which has no effect on the test suite, but does cause traceback spew to the console whenever the SMTP client calls .starttls() and the connection is subsequently close. For example:

$ tox -e py35-nocov -- -P test_forget_ehlo -E
py35-nocov develop-inst-noop: /home/barry/projects/aiosmtpd
py35-nocov installed: -e [email protected]:aio-libs/aiosmtpd.git@5da59e96368bdb042f23d5cd8c2cadbcf8180fe4#egg=aiosmtpd,appdirs==1.4.3,atpublic==0.5,flufl.testing==0.7,nose2==0.6.5,packaging==16.8,pkg-resources==0.0.0,six==1.10.0
py35-nocov runtests: PYTHONHASHSEED='3663210934'
py35-nocov runtests: commands[0] | python -m nose2 -v -P test_forget_ehlo -E
test_forget_ehlo (aiosmtpd.tests.test_starttls.TestTLSForgetsSessionData) ... DEBUG:asyncio:Using selector: EpollSelector
INFO:mail.log:Peer: ('::1', 35050, 0, 0)
INFO:mail.log:('::1', 35050, 0, 0) handling connection
DEBUG:mail.log:b'220 presto Python SMTP 1.0a5+\r\n'
DEBUG:asyncio:poll took 13.645 ms: 1 events
DEBUG:mail.log:_handle_client readline: b'ehlo [172.16.114.137]\r\n'
INFO:mail.log:('::1', 35050, 0, 0) Data: b'ehlo [172.16.114.137]'
DEBUG:mail.log:b'250-presto\r\n'
DEBUG:mail.log:b'250-SIZE 33554432\r\n'
DEBUG:mail.log:b'250-STARTTLS\r\n'
DEBUG:mail.log:b'250 HELP\r\n'
DEBUG:asyncio:poll took 45.399 ms: 1 events
DEBUG:mail.log:_handle_client readline: b'STARTTLS\r\n'
INFO:mail.log:('::1', 35050, 0, 0) Data: b'STARTTLS'
INFO:mail.log:('::1', 35050, 0, 0) STARTTLS
DEBUG:mail.log:b'220 Ready to start TLS\r\n'
DEBUG:asyncio:<asyncio.sslproto.SSLProtocol object at 0x7f2e033d7198> starts SSL handshake
DEBUG:asyncio:poll took 0.565 ms: 1 events
DEBUG:asyncio:<asyncio.sslproto.SSLProtocol object at 0x7f2e033d7198>: SSL handshake took 5.9 ms
DEBUG:asyncio:poll took 0.368 ms: 1 events
DEBUG:mail.log:_handle_client readline: b'mail FROM:<[email protected]>\r\n'
INFO:mail.log:('::1', 35050, 0, 0) Data: b'mail FROM:<[email protected]>'
DEBUG:mail.log:b'503 Error: send HELO first\r\n'
DEBUG:asyncio:poll took 0.023 ms: 1 events
DEBUG:mail.log:_handle_client readline: b'QUIT\r\n'
INFO:mail.log:('::1', 35050, 0, 0) Data: b'QUIT'
DEBUG:mail.log:b'221 Bye\r\n'
INFO:mail.log:Connection lost during _handle_client()
INFO:mail.log:('::1', 35050, 0, 0) connection lost
DEBUG:asyncio:<_SelectorSocketTransport fd=10 read=polling write=<idle, bufsize=0>> received EOF
DEBUG:asyncio:<asyncio.sslproto.SSLProtocol object at 0x7f2e033d7198> received EOF
INFO:mail.log:('::1', 35050, 0, 0) EOF received
ERROR:asyncio:Exception in callback None()
handle: <Handle cancelled _SelectorSocketTransport._read_ready() created at /usr/lib/python3.5/asyncio/selector_events.py:262>
source_traceback: Object created at (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/home/barry/projects/aiosmtpd/aiosmtpd/controller.py", line 55, in _run
    self.loop.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever
    self._run_once()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1416, in _run_once
    handle._run()
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 262, in _add_reader
    handle = events.Handle(callback, args, self)
Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 734, in _read_ready
    keep_open = self._protocol.eof_received()
  File "/usr/lib/python3.5/asyncio/sslproto.py", line 535, in eof_received
    keep_open = self._app_protocol.eof_received()
  File "/home/barry/projects/aiosmtpd/aiosmtpd/smtp.py", line 169, in eof_received
    return super().eof_received()
  File "/usr/lib/python3.5/asyncio/streams.py", line 256, in eof_received
    self._stream_reader.feed_eof()
AttributeError: 'NoneType' object has no attribute 'feed_eof'
INFO:mail.log:('::1', 35050, 0, 0) connection lost
DEBUG:asyncio:Close <_UnixSelectorEventLoop running=False closed=False debug=True>
ok

What seems to be happening is that, if we enter smtp_STARTTLS(), some state is messed up, and we eventually get eof_received() after connection_lost(). The latter is where self._stream_reader gets set to None.

Maybe we're doing something wrong when we flip on the SSL transport. Maybe there's a bug in asyncio, but if so, there have been no other reports on bugs.python.org that I can find. I don't believe it has anything to do with our overrides of eof_received() and connect_lost() since if I comment those two methods out, I still see the traceback. I think it's more likely we're messing with the asyncio state machine in problematic ways.

Finding New Stewards

I think it's time for me to admit that I don't have much time to work on aiosmtpd. I really hope it doesn't languish though, given how useful of a library it is.

We need to identify someone else to do most of the project management for this library. Any takers?

incompatible with uvloop because of access to private attribute _protocol

I'm using uvloop because of the significant speed up that it provides. However this seems to be incompatible with using the starttls method.

Can not send email
Traceback (most recent call last):
  File "/git/dnd/aiohttp-login/aiohttp_login/handlers.py", line 94, in registration
    'link': link,
  File "/git/dnd/aiohttp-login/aiohttp_login/utils.py", line 155, in render_and_send_mail
    await send_mail(to, subject.strip(), body)
  File "/git/dnd/aiohttp-login/aiohttp_login/utils.py", line 140, in send_mail
    await smtp.starttls(validate_certs=False)
  File "/git/dnd/lib64/python3.6/site-packages/aiosmtplib/esmtp.py", line 193, in starttls
    tls_context, server_hostname=server_hostname, timeout=timeout)
  File "/git/dnd/lib64/python3.6/site-packages/aiosmtplib/protocol.py", line 209, in starttls
    tls_context, server_hostname=server_hostname, waiter=waiter)
  File "/git/dnd/lib64/python3.6/site-packages/aiosmtplib/protocol.py", line 82, in upgrade_transport
    self.reader._transport._protocol = tls_protocol  # type: ignore
AttributeError: 'uvloop.loop.TCPTransport' object has no attribute '_protocol'

Edit: related Python bug report

Task was destroyed but it is pending

Hi,

checking at my errors when running aiosmtpd in production, I get a lot of "Task was destroyed but it is pending!" errors, with this detail following:

task: <Task pending coro=<AioSMTP._handle_client() running at /var/www/towboat/server/workers/utils/aiosmtp.py:344> wait_for=>

I tried to locate the reason why/when this happens, but so far, no luck.

I'm hoping I'm not the only one here, and maybe someone has a solution?

Protocol violation and message receive when exceeding data limit size

Run a simple server with data size limit:

from aiosmtpd.controller import Controller
from aiosmtpd.handlers import Message
from aiosmtpd.smtp import SMTP

class MessageHandler(Message):
    def handle_message(self, message):
        print(message)

class MyController(Controller):
    def factory(self):
        return SMTP(self.handler, data_size_limit=10)

controller = MyController(MessageHandler())
controller.start()

Connect to it with nc

$ nc -C 127.0.0.1 8025
220 hostname 1.0a1
HELO foo 
250 vpn.bayan.rocks
MAIL FROM: [email protected]
250 OK
RCPT TO: [email protected]
250 OK
DATA
354 End data with <CR><LF>.<CR><LF>
Header: value
552 Error: Too much mail data
Header2: value2
552 Error: Too much mail data
Header3: value3
552 Error: Too much mail data

552 Error: Too much mail data
.
250 OK

The server responds multiple time with error code 552 and finally with 250!
Also server receives message and calls handle_message

Drop Python 3.4 support

We should probably drop Python 3.4 support. For one thing, I don't have a good way to run the 3.4 tests locally. We could of course just rely on CI to verify that continues to work.

More importantly though, I think it would be really nice to be able to use the new Python 3.5 async keywords.

Support for UNIX sockets in the Controller

Currently the Controller only seems to support listening on AF_INET and AF_INET6 ports. It is sometimes (especially for LMTP) desirable to listen on a UNIX socket, so it would be great if support for that would be added to the Controller.

Test failures and DeprecationWarnings in Python 3.8

There are test case failures in Python 3.8 . I think this is due to mocking create_server which is now AsyncMock instead of MagicMock with changes in https://bugs.python.org/issue26467 where coroutines when patched return AsyncMock. Thus there are warnings due to the mock not being awaited. The old behavior can be used with MagicMock explicitly passed while patching. There also seems to be a bug that AsyncMock is not recording call_args and other values for synchronous API.

There is also DeprecationWarning since StreamReader and StreamWriter are merged into Stream class in https://bugs.python.org/issue36889. Attached is a log with 3.9.0a0 but these are reproducible on 3.8.0b1 as well.

aiosmptd.log

[FTBFS into debian] A test randomly fails.

Dear developers,

Sometimes, the testing part of aiosmtpd fails, because of a specific test.
Here is, in that case, the error that occurs.

======================================================================
ERROR: test_debug_3 (aiosmtpd.tests.test_main.TestMain)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python-aiosmtpd-1.1/.pybuild/pythonX.Y_3.6/build/aiosmtpd/tests/test_main.py", line 140, in test_debug_3
    main(('-n', '-ddd'))
  File "/build/python-aiosmtpd-1.1/.pybuild/pythonX.Y_3.6/build/aiosmtpd/main.py", line 140, in main
    loop.create_server(factory, host=args.host, port=args.port))
  File "/usr/lib/python3.6/asyncio/base_events.py", line 465, in run_until_complete
    raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.

----------------------------------------------------------------------

This error does prevent aiosmtpd to build from source into debian, and so prevents the reproductibility tests to succeed.

Can you have a look at it please?

Hooks returning None should make smtp.py behave like nothing happened.

When hooks are called for each steps of the SMTP call, the code check if the MISSING value is returned, which is an instance of object().

I'm curious to know why.

From my perspective, if a call to a hook returns None, the code should continue like if there were no hooks at all.
The reason behind that is if no response is returned from the hook, the server still has to return a status, and the default one is accepted by the developer.

Moreover, if a hook exists, it MUST implement the logics presents in smtp.py when no hooks exists. Let me explain:

Here's the line for HELO commands

If there are no hooks, here's what's happening:

        status = await self._call_handler_hook('HELO', hostname)
        if status is MISSING:
            self.session.host_name = hostname
            status = '250 HELP'

But as soon as I add a HELO handler, the status won't be MISSING, which means I have to implement the two consequent lines in my own code, by copy/pasting it. That's not good.

Instead, if the smtp.py code does the following:

        status = await self._call_handler_hook('HELO', hostname)
        # This expect that the self._call_handler_hook returns also None in case no handler was found
        if status is None:
            self.session.host_name = hostname
            status = '250 HELP'

My handle can return None, saying "everything is good" like all hooks behave, and smtp.py will do the intended work.

But maybe I'm missing a specific reason, so I'd be curious to know.

Otherwise, happy to submit a PR if you want!

RPI handle_data doesnt get called

I have a script that works on multiple machines but on 2 machines, specifically raspbian. It will receive emails so long as they have no attachment. I'm wondering if I could get some help.

It just never calls handle_data and the first command is print new email but that doesnt ever happen.

The handle_rcpt works fine; respond 250 to everything.

async def handle_DATA(self, server, session, envelope):
    print ('New Email \n')

controller = Controller(smtphoney(), hostname = IP,port=25)

250-8BITMIME
250-SMTPUTF8
250 HELP
RSET
250 OK
MAIL FROM: [email protected]
250 OK
RCPT TO: [email protected]
250 OK
DATA
354 End data with .
.
Disconnected.

Error: Connection closed.
Failed to send message

Misbehaving clients are never disconnected

It's possible to exhaust resources on the server in a number of ways:

  • Connect and never send a valid command
  • Connect and repeatedly send NOOP
  • Connect and waste time in the SMTP transaction, e.g. send multiple HELO, reset,

Eventually Python will run out of file handles and exceptions will occur in code that tries to open any file, socket, etc.

No timeout on ready event

If you start an SMTP server using the Controller class using a port that is already in-use you're going to hang forever on the ready event here.

One way to handle it would be to add a timeout to the wait and pass an error to the main thread.

CancelledError and misbehaving clients handling

I tried to write a RCPT handler that accepts connection from various embedded devices and forwards the email address to an external system. However, as the devices are not proper SMTP clients and their quality varies I need the solution to be pretty robust. While developing it I discovered a problem with the way aoismtpd deals with clients who send data and ignore the response. To reproduce:

sleep.py (simple handler class simulating my handler):

import asyncio
from aiosmtpd.handlers import Debugging

class RcptWebhook(Debugging):
    async def handle_RCPT(self, server, session, envelope, address, rcpt_options):
        await asyncio.sleep(10)

        envelope.rcpt_tos.append(address)
        return '250 OK'

Run aiosmtpd

python3 -m aiosmtpd -n -c sleep.RcptWebhook

smtp.txt (sample SMTP session)

HELO there
MAIL FROM: [email protected]
RCPT TO: [email protected]
DATA
test
.

(note the is no QUIT in the data)

Send it witch netcat

nc localhost 8025 < smtp.txt

It sends the content of the file and EOF which causes unhandled CanceledError in aiosmtpd while it is waiting for my sleep handler to finish:


INFO:mail.log:Server listening on localhost:8025
INFO:mail.log:Starting asyncio loop
INFO:mail.log:Peer: ('127.0.0.1', 39306)
INFO:mail.log:('127.0.0.1', 39306) handling connection
INFO:mail.log:('127.0.0.1', 39306) Data: b'HELO there'
INFO:mail.log:('127.0.0.1', 39306) Data: b'MAIL FROM: [email protected]'
INFO:mail.log:('127.0.0.1', 39306) sender: [email protected]
INFO:mail.log:('127.0.0.1', 39306) Data: b'RCPT TO: [email protected]'
INFO:mail.log:('127.0.0.1', 39306) EOF received
ERROR:mail.log:SMTP session exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/aiosmtpd/smtp.py", line 276, in _handle_client
    yield from method(arg)
  File "/usr/local/lib/python3.5/dist-packages/aiosmtpd/smtp.py", line 582, in smtp_RCPT
    'RCPT', address, rcpt_options)
  File "/usr/local/lib/python3.5/dist-packages/aiosmtpd/smtp.py", line 116, in _call_handler_hook
    status = yield from hook(self, self.session, self.envelope, *args)
  File "/home/vaclav/src/smtp-events-receiver/sleep.py", line 6, in handle_RCPT
    await asyncio.sleep(10)
  File "/usr/lib/python3.5/asyncio/tasks.py", line 516, in sleep
    return (yield from future)
  File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 266, in result
    raise CancelledError
concurrent.futures._base.CancelledError
INFO:mail.log:('127.0.0.1', 39306) Data: b'DATA'
INFO:mail.log:('127.0.0.1', 39306) Data: b'test'
INFO:mail.log:('127.0.0.1', 39306) Data: b'.'
INFO:mail.log:('127.0.0.1', 39306) Data: b''
INFO:mail.log:('127.0.0.1', 39306) Data: b''
INFO:mail.log:('127.0.0.1', 39306) Data: b''
INFO:mail.log:('127.0.0.1', 39306) Data: b''
...

aiosmtpd is now 100% busy on CPU and keeps printing "500 Error: bad syntax" to the netcat. It ignores KeyboardInterrupt and to quit it I need to terminate netcat with ^C which stops aiosmtp with "Connection lost".

I think aiosmtpd should handle this more gracefully as you don't want your server to get stuck on CPU due to misbehaving client. I see in RFC 5321 that the server must not close the connection until QUIT from the client so I'm not sure what is the proper way of handling EOF though.

Controller.start uses threading

self._thread = threading.Thread(target=self._run, args=(ready_event,))

It seems to be that Controller.start using Threading is not required and will lead to problems.

Like the asyncio http server starting an SMTP server should not involve threading, but just loop.run_forever().

AUTH LOGIN support

I'm looking to implement auth support. For some of the mechanisms, this requires the client supply additional information on a new line following the command after a prompt is received from the server. LOGIN for example will prompt with 334 Username (in base64) and then expect the username to be supplied after and will repeat this for password. Same for plain, the credentials follow the command as a response.

I am fairly new to asyncio in general, but didn't find an easy way I could implement this in a subclass of SMTP or a handler without having to copy a significant amount of code. Am I missing something here? If not, what would be a reasonable approach to addressing this?

Examples:
C: AUTH PLAIN
S: 334
C: dGVzdAB0ZXN0ADEyMzQ=
S: 235 Authentication successful

C: AUTH LOGIN
S: 334 VXNlcm5hbWU6
C: avlsdkfj
S: 334 UGFzc3dvcmQ6
C: lkajsdfvlj
S: 535 Authentication failed

LMTP server offers HELO, doesn't provide correct LHLO response

As initailly reported here: https://gitlab.com/mailman/mailman/issues/348

Telnet to the LMTP server, and type "HELP".

It responds with
250 Supported commands: EHLO HELO MAIL RCPT DATA RSET NOOP QUIT VRFY

This isn't correct. LMTP uses LHLO; the EHLO and HELO should not be offered.

LHLO foo

Responds with
250 foo.example.net

RFC2033 says the implementation MUST support PIPELINING and ENHANCEDSTATUSCODES, and SHOULD support 8BITMIME

Thus, LHLO should be responding with the ESMTP keywords for these extensions.

8BITMIME (RFC 1652)

ENHANCEDSTATUSCODES (RFC 2034)

PIPELINING (RFC 1854)

Since it also responds to HELP HELP should also be listed (RFC 1869)

Thus, the dialog should look like:

S: 220 foo.example.net GNU Mailman LMTP ...
C: LHLO foo
S: 250-foo.example.net
S: 250-HELP
S: 250-ENHANCEDSTATUSCODES
S: 250-8BITMIME
S: 250 PIPELINING

(The order doesn't matter.)

And, obviously, 'HELP' should respond with the correct list of supported commands - including LHLO.

How to bind all interfaces dual-stack?

I wish to write a server application that binds to all interfaces in a dual-stack IPv4/IPv6 environment.

I thought the example server would be doing that, as it appears to bind to '::0'. In other applications, like CherryPy, binding to that address makes the server available on all interfaces, IPv4 and IPv6, but in aiosmtpd, the server is only reachable on IPv6. As a result, when the application moves to an environment that only supports IPv4 (looking at you, Docker), the application is unreachable.

Reading up on the docs for create_server, I see that passing a None value for the host should bind to all interfaces on both stacks, and indeed, that's the behavior I see.

But aiosmtpd doesn't allow passing None due to the the localhost default behavior.

I did find one interface that worked, but I'm not happy with it:

	cont = controller.Controller(proxy, port=8025)
	# no really, I want all interfaces
	cont.hostname = None
	cont.start()

Write a proper manpage for the CLI

The Debian packaging wants a manpage for aiosmtpd CLI. I can write a proper one using rst2man, but for now they're going to use help2man.

Windows Support

Crashes on Windows

PS C:\Windows\system32> D:\aws-ses-proxy\venv\Scripts\python.exe -m aiosmtpd -n
Traceback (most recent call last):
  File "C:\Python36\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\aws-ses-proxy\venv\lib\site-packages\aiosmtpd\__main__.py", line 5, in <module>
    main()
  File "D:\aws-ses-proxy\venv\lib\site-packages\aiosmtpd\main.py", line 137, in main
    loop.add_signal_handler(signal.SIGINT, loop.stop)
  File "C:\Python36\lib\asyncio\events.py", line 481, in add_signal_handler
    raise NotImplementedError
NotImplementedError

Unicode support

Attempt to send message with non-ascii symbols results in UnicodeEncodeError:

>>> from smtplib import SMTP as Client
>>> client = Client('127.0.0.1', 8025)
>>> r = client.sendmail('[email protected]', ['[email protected]'], "Ї")
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/usr/local/lib/python3.6/smtplib.py", line 854, in sendmail
    msg = _fix_eols(msg).encode('ascii')
UnicodeEncodeError: 'ascii' codec can't encode character '\u0407' in position 0: ordinal not in range(128)

Is there any particular reason to use .encode('ascii') instead of just .encode()?

More silent tracebacks

We get a few of these in the Travis output, but not in local test runs. They don't affect the success of the tests though.

test_debug_0 (aiosmtpd.tests.test_main.TestMain) ... ERROR:asyncio:<CoroWrapper BaseEventLoop.create_server() running at /opt/python/3.5.2/lib/python3.5/asyncio/base_events.py:876, created at /home/travis/build/aio-libs/aiosmtpd/aiosmtpd/main.py:136> was never yielded from
Coroutine object created at (most recent call last):
  File "/opt/python/3.5.2/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/python/3.5.2/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/travis/build/aio-libs/aiosmtpd/.tox/py35-nocov/lib/python3.5/site-packages/nose2/__main__.py", line 12, in <module>
    discover()
  File "/home/travis/build/aio-libs/aiosmtpd/.tox/py35-nocov/lib/python3.5/site-packages/nose2/main.py", line 306, in discover
    return main(*args, **kwargs)
  File "/home/travis/build/aio-libs/aiosmtpd/.tox/py35-nocov/lib/python3.5/site-packages/nose2/main.py", line 100, in __init__
    super(PluggableTestProgram, self).__init__(**kw)
  File "/opt/python/3.5.2/lib/python3.5/unittest/main.py", line 94, in __init__
    self.runTests()
  File "/home/travis/build/aio-libs/aiosmtpd/.tox/py35-nocov/lib/python3.5/site-packages/nose2/main.py", line 271, in runTests
    self.result = runner.run(self.test)
  File "/home/travis/build/aio-libs/aiosmtpd/.tox/py35-nocov/lib/python3.5/site-packages/nose2/runner.py", line 53, in run
    executor(test, result)
  File "/home/travis/build/aio-libs/aiosmtpd/.tox/py35-nocov/lib/python3.5/site-packages/nose2/runner.py", line 41, in <lambda>
    executor = lambda suite, result: suite(result)
  File "/opt/python/3.5.2/lib/python3.5/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/opt/python/3.5.2/lib/python3.5/unittest/suite.py", line 122, in run
    test(result)
  File "/opt/python/3.5.2/lib/python3.5/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/opt/python/3.5.2/lib/python3.5/unittest/suite.py", line 122, in run
    test(result)
  File "/opt/python/3.5.2/lib/python3.5/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/opt/python/3.5.2/lib/python3.5/unittest/suite.py", line 122, in run
    test(result)
  File "/opt/python/3.5.2/lib/python3.5/unittest/case.py", line 648, in __call__
    return self.run(*args, **kwds)
  File "/opt/python/3.5.2/lib/python3.5/unittest/case.py", line 600, in run
    testMethod()
  File "/home/travis/build/aio-libs/aiosmtpd/aiosmtpd/tests/test_main.py", line 120, in test_debug_0
    main(('-n',))
  File "/home/travis/build/aio-libs/aiosmtpd/aiosmtpd/main.py", line 136, in main
    loop.create_server(factory, host=args.host, port=args.port))
ok

100% code coverage

We're currently at about 96% overall for coverage, with aiosmtpd/smtp.py the worst offender. Here's the output of tox -e py35-cov. Let's get this to 100%

Name                           Stmts   Miss Branch BrPart  Cover   Missing
--------------------------------------------------------------------------
aiosmtpd/__init__.py               0      0      0      0   100%
aiosmtpd/controller.py            47      0      4      0   100%
aiosmtpd/handlers.py             108      1     44      0    99%   27
aiosmtpd/lmtp.py                  11      0      2      0   100%
aiosmtpd/main.py                 105      0     22      0   100%
aiosmtpd/smtp.py                 345     22    154     11    93%   92-95, 129-130, 132-133, 145-146, 149-151, 209-211, 227, 384-386, 408-409, 420, 432, 122->exit, 128->129, 131->132, 144->145, 148->149, 226->227, 383->384, 407->408, 414->429, 419->420, 431->432
--------------------------------------------------------------------------
TOTAL                            616     23    226     11    96%

Exception when parsing binary data

aiosmtpd will try to ASCII decode binary data as SMTP commands, raising a UnicodeDecodeError. A simple test case that triggers this is shown at the end of this report. This can be caught by a generic exception handler, however it would be better to do this inside the library.

This can be trivially fixed with a try/except handler. PR incoming.

from base64 import b64decode
import socket


TCP_IP = '127.0.0.1'
TCP_PORT = 8025
BUFFER_SIZE = 1024

RAW = "gUMBAwMBGgAAACAAwDAAwCwAwCgAwCQAwBQAwAoAwCIAwCEAAKMAAJ8AAGsAAGoAADkAADgAAIgAAIcAwBkAwCAAwDIAwC4AwCoAwCYAwA8AwAUAAJ0AAD0AADUAAIQAwBIAwAgAwBwAwBsAABYAABMAwBcAwBoAwA0AwAMAAAoHAMAAwC8AwCsAwCcAwCMAwBMAwAkAwB8AwB4AAKIAAJ4AAGcAAEAAADMAADIAAJoAAJkAAEUAAEQAwBgAwB0AwDEAwC0AwCkAwCUAwA4AwAQAAJwAADwAAC8AAJYAAEEAAAcAwBEAwAcAwBYAwAwAwAIAAAUAAAQFAIADAIABAIAAABUAABIAAAkGAEAAABQAABEAAAgAAAYAAAMEAIACAIAAAP8cXF6WB1DBTAUUZfksmYhwy/mtOvciiLZb+ZNMaF/tYg=="
MESSAGE = b64decode(RAW)

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
data = s.recv(BUFFER_SIZE)
print("Received: {}".format(data))
s.send(MESSAGE)
data = s.recv(BUFFER_SIZE)
print("Received: {}".format(data))
s.close()

LMTP server doesn't support require_starttls; require_starttls not RFC-compliant

smtp.py's _handle_client contains the following code:

   270                  if (self.require_starttls
   271                          and not self._tls_protocol
   272                          and command not in ['EHLO', 'STARTTLS', 'QUIT']):
   273                      # RFC3207 part 4
   274                      await self.push('530 Must issue a STARTTLS command first')
   275                      continue

When subclassed for the LMTP server, 'EHLO' needs to be 'LHLO'.

I'm not enough of a Python person to code the change... (It needs an 'unrestricted commands' list that the subclass can override.)

This bug prevents enabling require_starttls for LMTP in Mailman -- https://gitlab.com/mailman/mailman/issues/365

In addition, RFC3207 section 4 (3 paragraphs from the end) requires that NOOP also be accepted before TLS is established. This change, of course, is trivial...

Need Travis-CI integration

The old GitLab .gitlab-ci.yml is no longer appropriate since the move to GitHub. We need a .travis-ci.yml file for Travis integration.

Add mypy/static type annotations

It seems reasonable to add type annotations since aiosmtpd is a Python 3 application/library and type annotations can provide better documentation to users. With the use of a tool like mypy, those assertions can even be tested.

We'll pretty quickly run into problems though since our Handler classes are duck typed, and mypy does not yet support structural typing. Meaning, examples such as Controller.__init__()'s handler argument does not have to be a subclass of anything; it simply needs to support some methods. Handlers are even weirder than straight up duck types too, because handle_*() methods are optional.

Similarly, SMTP methods smtp_*() are optional too.

Some things that might help include:

  • PEP 544 defines "Protocols" that can be used to structurally type arguments;
  • Techniques such as those described in this blog post;
  • Just using an Any type for the duck typed arguments;
  • Require that handlers derive from a common base class. I don't like this solution much as it was essentially rejected for an earlier PR, and it would be an API change requiring a major version number bump

STARTTLS documentation errors & clarifications

http://aiosmtpd.readthedocs.io/en/latest/aiosmtpd/docs/smtp.html includes:

tls_context and require_starttls are related to the ESMTP STARTTLS command for secure connections to the server, based on RFC 3207. tls_context is used as the SSL protocol context, and there is no default. tls_context must be given and require_starttls must be True for STARTTLS to be supported.

This is only partially correct, and totally confusing. Better would be:

The STARTTLS option of ESMTP (and LMTP), defined in RFC3207, provides for secure connections to the server. For this option to be available, tls_context must be supplied. tls_context is created with the ssl.create_default_context call from the ssl module, as follows:

ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)

The context must be initialized with a server certificate, private key, and/or intermediate CA certificate
chain with the load_cert_chain method. This can be done with separate files, or an all in one file. Files must be in PEM format. e.g.

ctx.load_cert_chain( config.mta.lmtp_certificate )

A number of exceptions can be generated by these methods, and by SSL connections, which you must be prepared to handle. Complete documentation for the ssl module and ssl context methods is located at https://docs.python.org/3.6/library/ssl.html, and should be reviewed before use; in particular if client authentication and/or advanced error handling is desired.

If require_starttls is true, a TLS session must be initiated for the server to respond to any commands other than EHLO/LHLO, NOOP, QUIT, and STARTTLS.

If require_starttls is false (the default), use of TLS is not required; the client MAY upgrade the connection to TLS, or may use any supported command on an insecure connection.

If tls_context is not supplied, the STARTTLS option will not be advertised, and the STARTTLS command will be rejected. require_starttls is meaningless in this case, and should be false.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.