getmail6 / getmail6 Goto Github PK

based on getmail 5.14, with adaptations for python 3

Python 90.82% Makefile 0.54% Shell 8.44% Dockerfile 0.20%

getmail6's Introduction

Description

getmail6 is a flexible, extensible mail retrieval system with support for POP3, IMAP4, SSL variants of both, maildirs, mboxrd files, external MDAs, arbitrary message filtering, single-user and domain-mailboxes, and many other useful features.

Licence

getmail is Copyright (C) 1998-2024 Charles Cazabon and others. getmail is licensed for use under the GNU General Public License version 2 (only). See docs/COPYING for specific terms and distribution information.

Bugs

getmail6 has adaptations to work with Python 3. These changes might still contain some bugs. Please report them at https://github.com/getmail6/getmail6. See docs/BUGS for instructions on reporting bugs.

Installation

To install:

pip install getmail6

To uninstall:

pip uninstall getmail6

You can install getmail6 in your home directory if you add --user.

If getmail6 is available via your Linux distribution, you better use that.

Usage

getmail6 is not a python API. getmail6 provides command line tools:

getmail
getmail_maildir
getmail_mbox
getmail_fetch
getmail-gmail-xoauth-tokens

Configuration

Before using getmail you must configure it. See docs/configuration.txt and docs/getmailrc-examples. An example:

[retriever]
type = SimpleIMAPSSLRetriever
server = imap.gmail.com
port = 993
username = <your_email_here>
#password = ... or
password_command = ("pass", "<your_email_here>")

[destination]
type = Maildir
path = ~/Mail/<your_email_here>/INBOX/

[options]
read_all = true
delete = true

Gmail

For gmail, after having enabled 2-Step Authentication, let google generate an "app password" for you. Then, for the above example, use pass edit <your_email_here> and change to the generate one.

Go to https://mail.google.com
If you are signed in, on the left upper corner there is a cogwheel symbol for settings
Choose "See all Settings"
"Accounts and Imports" tab, then "Other Google Account Settings"/"Security" brings you to https://myaccount.google.com/u/0/security?hl=en
Turn on "2-Step Verification" (also known as 2-factor-authentication or 2FA)
In "App passwords", generate a password for your device
Update this in your password command.

getmail is a native app. See https://developers.google.com/identity/protocols/oauth2/native-app Still, to download your email from gmail to your computer using OAuth2 you need to grant getmail OAuth2 access to the scope https://mail.google.com/, as you would to web apps. Unfortunately, the init step in example 12 in docs/getmailrc-examples has to be repeated regularly. This makes the app password method above a better alternative. Don't forget to remove the use_xoauth2 line, if you switch from Oauth2 to app password.

Tests

There is now a test folder that uses docker-mailserver for black box testing.

This still tests Python 2.7.

Tests are work in progress.

Documentation

See the HTML documentation for details on setting up and using getmail.

docs/documentation.html
docs/configuration.html
docs/faq.html
docs/troubleshooting.html

It is included in the docs subdirectory, and will be installed in <PREFIX>/doc/getmail-<version>/ (by default).

getmail6's People

Contributors

Stargazers

Watchers

Forkers

andram dmytri digifuchsi jidanni remmy lilydjwg sethrobertson mickael067 jirkade libnoon hannob wuestengecko ykasap vgropp adamcandy iustin stuart-mclaren mike-kfed richardweinberger spezifisch jheidemann baro77 tmcqueen-materials zw963 christg742 horrad acloserview beur1x daniel-zullo-frequenz matthiaspetermann fow0ryl knagano cclauss cloudpage jradxl ewft coopermasaaki jerome-pouiller bqleine tomnewchao nuffin

getmail6's Issues

AttributeError: module 'getmailcore.destinations' has no attribute 'SimpleIMAPSSLRetriever'

Cannot run getmail on openbsd 6.8, installed locally via OS's 'pip', without getting the above error. My config file looks like:

$ cat ~/.getmail/getmailrc 
[retriever]
type = SimpleIMAPSSLRetriever
server = old.imap.server
username = law@olddomain
password = xxx
mailboxes = ALL

[destination]
type = SimpleIMAPSSLRetriever
server = new.imap.server
username = law@newdomain
password = xxx

[options]
read_all = False

$ ~/.local/bin/getmail -v

Exception: please read docs/BUGS and include the following information in any bug report:

  getmail version 6.8
  Python version 3.8.6 (default, Oct 13 2020, 09:04:17) 
[Clang 10.0.1 ]

Unhandled exception follows:
    File "/home/law/.local/bin/getmail", line 773, in main
    destination_func = getattr(destinations, destination_type)
  AttributeError: module 'getmailcore.destinations' has no attribute 'SimpleIMAPSSLRetriever'

Please also include configuration information from running getmail
with your normal options plus "--dump".

UnicodeEncodeError: 'ascii' codec can't encode character '\u2013' in position 546: ordinal not in range(128)

Hi, we have installed the getmail6 ver 6.16 with the issue #74 applied and fixed. But this seems to be more of the same ?

messages:Jun  1 17:37:47 rt getmail[2708465]: /opt/rt5/bin/rt-mailgate: connecting to https://rt.centralhelp.org/REST/1.0/NoAuth/mail-gateway
messages:Jun  1 17:37:47 rt getmail[2708465]: ok
messages:Jun  1 17:37:47 rt getmail[2708465]: Ticket: 24982
messages:Jun  1 17:37:47 rt getmail[2708465]: Queue:
messages:Jun  1 17:37:47 rt getmail[2708465]: Owner: Nobody
messages:Jun  1 17:37:47 rt getmail[2708465]: Status: new
messages:Jun  1 17:37:47 rt getmail[2708465]: Subject: Employee Departure: <redacted)
messages:Jun  1 17:37:48 rt getmail[2708465]: Requestor: alerts.us@<redacted>  msg 4/5 (131443 bytes) delivered, deleted
messages:Jun  1 17:37:48 rt getmail[2708465]: Exception: please read docs/BUGS and include the following information in any bug report:
messages:Jun  1 17:37:48 rt getmail[2708465]:  getmail version 6.16
messages:Jun  1 17:37:48 rt getmail[2708465]:  Python version 3.6.8 (default, Aug 24 2020, 17:57:11)
messages:Jun  1 17:37:48 rt getmail[2708465]: [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
messages:Jun  1 17:37:48 rt getmail[2708465]: Unhandled exception follows:
messages:Jun  1 17:37:48 rt getmail[2708465]:    File "/usr/local/bin/getmail", line 932, in main
messages:Jun  1 17:37:48 rt getmail[2708465]:    success = go(configs, options.idle)
messages:Jun  1 17:37:48 rt getmail[2708465]:    File "/usr/local/bin/getmail", line 257, in go
messages:Jun  1 17:37:48 rt getmail[2708465]:    options['delivered_to'], options['received'])
messages:Jun  1 17:37:48 rt getmail[2708465]:    File "/usr/local/lib/python3.6/site-packages/getmailcore/destinations.py", line 113, in deliver_message
messages:Jun  1 17:37:48 rt getmail[2708465]:    return self._deliver_message(msg, delivered_to, received)
messages:Jun  1 17:37:48 rt getmail[2708465]:    File "/usr/local/lib/python3.6/site-packages/getmailcore/destinations.py", line 672, in _deliver_message
messages:Jun  1 17:37:48 rt getmail[2708465]:    self.log.info('command %s: %s' % (self, child.err))
messages:Jun  1 17:37:48 rt getmail[2708465]:    File "/usr/local/lib/python3.6/site-packages/getmailcore/logging.py", line 104, in info
messages:Jun  1 17:37:48 rt getmail[2708465]:    self.log(INFO, msg)
messages:Jun  1 17:37:48 rt getmail[2708465]:    File "/usr/local/lib/python3.6/site-packages/getmailcore/logging.py", line 64, in log
messages:Jun  1 17:37:48 rt getmail[2708465]:    handler['stream'].write(msgtxt)
messages:Jun  1 17:37:48 rt getmail[2708465]:  UnicodeEncodeError: 'ascii' codec can't encode character '\u2013' in position 546: ordinal not in range(128)
messages:Jun  1 17:37:48 rt getmail[2708465]: Please also include configuration information from running getmail
messages:Jun  1 17:37:48 rt getmail[2708465]: with your normal options plus "--dump".

Here is the getmail6 --dump output

/usr/local/bin/getmail --getmaildir=/home/rt-cron/.getmail  --rcfile ithelpbot.us.conf --dump
getmail version 6.16
Copyright (C) 1998-2021 Charles Cazabon and others. Licensed under GNU GPL version 2.
getmail configuration:
  getmail version 6.16
  Python version 3.6.8 (default, Aug 24 2020, 17:57:11) 
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
  retriever:  SimpleIMAPSSLRetriever(ca_certs="None", certfile="None", getmaildir="/home/rt-cron/.getmail", keyfile="None", mailboxes="('INBOX',)", move_on_delete="None", password="*", password_command="('/home/rt-cron/.getmail/outlook-xoauth-tokens', '/home/rt-cron/.getmail/ithelpbot.us.json', 'ithelpbot.us@<redacted>', '<redacted>', port="993", record_mailbox="True", server="outlook.office365.com", ssl_cert_hostname="None", ssl_ciphers="None", ssl_fingerprints="()", ssl_version="None", timeout="180", use_cram_md5="False", use_kerberos="False", use_peek="True", use_xoauth2="True", username="ithelpbot.us@<redacted>")
  destination:  MDA_external(allow_root_commands="False", arguments="('--queue', 'US IT Help', '--action', 'correspond', '--url', 'https://rt.<redacted>', '--debug')", command="rt-mailgate", group="rt", ignore_stderr="True", path="/opt/rt5/bin/rt-mailgate", unixfrom="False", user="rt-cron")
  options:
    delete : True
    delete_after : 0
    delete_bigger_than : 0
    delivered_to : True
    fingerprint : False
    logfile : None
    max_bytes_per_session : 0
    max_message_size : 0
    max_messages_per_session : 0
    message_log : None
    message_log_syslog : True
    message_log_verbose : False
    read_all : True
    received : True
    verbose : 1

one of my colleagues is wondering if it is related to this issue on stackoverflow? And if getmail6 has to choose and set an encoding for out streams? Since it seems to fail while doing a stream write 'handler['stream'].write(msgtxt)' ?

https://stackoverflow.com/questions/492483/setting-the-correct-encoding-when-piping-stdout-in-python

There is a secondary error that I open another issue for. And that is that getmail6 says it has retrieved and deleted messages when it clearly has not done so. In our case this leads to multiplying tickets as the ticket system inbox mail does not get deleted after having been retrieved.

I can make the offending email available, if required, but can't post it publicly here for data protection reasons.

gnomekeyring functionality broken

Looks like gnomekeyring has been removed and is no longer available.

This project shoulod be updated to use keyring instead. https://pypi.org/project/keyring/

RuntimeError: dictionary changed size during iteration

There is an exceptions while reading emails. I checked several accounts, behaviour is inconsistent. One gmail account passes (it has no new email), the other gmail account fails (it has new emails). Configuration is the same, only difference in the config file is the user and password (and that one has new email and not the other one). Another separate account from provider laposte.net perfectly reads with or without new emails, while yet another account on office365.com fails with same error as below.

getmail version 6.00
Copyright (C) 1998-2019 Charles Cazabon.  Licensed under the GNU GPL version 2.
SimplePOP3SSLRetriever:[email protected]@pop.gmail.com:995:

Exception: please read docs/BUGS and include the following information in any bug report:

  getmail version 6.00
  Python version 3.7.7 (default, May  3 2020, 15:10:29) 
[GCC 9.3.0]

Unhandled exception follows:
    File "/usr/bin/getmail", line 895, in main
    success = go(configs, options.idle)
    File "/usr/bin/getmail", line 165, in go
    retriever.select_mailbox(mailbox)
    File "/usr/lib/python3.7/site-packages/getmailcore/_retrieverbases.py", line 1013, in select_mailbox
    self._getmsglist()
    File "/usr/lib/python3.7/site-packages/getmailcore/_retrieverbases.py", line 1078, in _getmsglist
    for msgid in self.oldmail.keys():
  RuntimeError: dictionary changed size during iteration

Please also include configuration information from running getmail
with your normal options plus "--dump".

getmail version 6.00
Copyright (C) 1998-2019 Charles Cazabon.  Licensed under the GNU GPL version 2.
getmail configuration:
  getmail version 6.00
  Python version 3.7.7 (default, May  3 2020, 15:10:29) 
[GCC 9.3.0]
  retriever:  SimplePOP3SSLRetriever(ca_certs="None", certfile="None", delete_dup_msgids="False", getmaildir="/home/jerome/.home/.getmail/", keyfile="None", password="*", password_command="()", port="995", server="pop.gmail.com", ssl_cert_hostname="None", ssl_ciphers="None", ssl_fingerprints="()", ssl_version="None", timeout="180", use_apop="False", username="[email protected]")
  destination:  Maildir(filemode="384", path="/home/jerome/donnees/Mail/inbox/", user="None")
  options:
    delete : True
    delete_after : 0
    delete_bigger_than : 0
    delivered_to : True
    fingerprint : False
    logfile : logfile(filename="~/.getmail/log")
    max_bytes_per_session : 0
    max_message_size : 0
    max_messages_per_session : 0
    message_log : ~/.getmail/log
    message_log_syslog : False
    message_log_verbose : False
    read_all : True
    received : True
    verbose : 2

"Return-Path: <unknown>" can confuse MTAs/LDAs down the line

As any MTA should, getmail writes a Return-Path header to every message. It writes Return-Path: <unknown> if it cannot find an appropriate sender address.

I use getmail's MDA_external destination to pass messages to dovecot-lda for final delivery. If a message contains Return-Path: <unknown> (which, surprisingly, is the case for something like 30–50% of my mail), this understandably irritates dovecot-lda into logging Warning: Failed to parse return-path header.

I really want to silence this warning, which is spamming my logs. Currently, I'm employing an ugly hack:

passing arguments = ('-f', '[email protected]') to dovecot-lda, thereby overriding getmail's Return-Path with a lexically valid (but nonsensical) address

That's ugly and painful to configure (because I maintain a large number of getmailrc files). There's a another potential solution, but that would require changing the behavior of dovecot-lda:

calling dovecot-lda with the -e option (which is the way I am doing it) means it will never send bounces, and instead just fail with some stderr output if a message cannot be delivered. In this case, the content of the Return-Path header is irrelevant. dovecot-lda could arguably ignore the Return-Path header when called with -e, and just never attempt to parse it—silencing the warning.

However, it seems unlikely that this corner-case will warrant a change to dovecot, which is a very large project. In my mind, there's more of a chance to change getmail ;) But how?

I've taken a good look at the possibility of adding a new global option like return_path = False. It could prevent modifying or adding any Return-Path header, and especially Return-Path: <unknown>. However, judging from the way delivered_to is implemented (which does nearly the same thing), this would require major code changes. These could cause breakage with some of the more exotic usecases for getmail (like multidrop retrieval), which we're not equipped to adequately test.
Maybe instead of <unknown>, getmail could write something lexically valid but nonsensical like [email protected]? This would prevent tripping up MTAs/LDAs down the line, as happens in my case. However, if bounce mails were really dispatched to this address, even more error spam would ensue, so that probably shouldn't be a default.

I'm curious if anyone has an opinion or idea about this :)

Release 6.5 with OAuth fixes

A FreeBSD user has expressed interest in having the recent OAuth fix included in the coming package update from 6.03. If we can release 6.5 now, that can be included quickly and cleanly.

FreeBSD test and port submission

Thanks for your great work on this! Just a heads-up: I've successfully tested 6.02 in production on FreeBSD 11.4 with Python 3.7. It uses the SimplePOP3SSLRetriever to get mail from 5 different servers and hands them off to dovecot-lda as an MDA_external.

Since getmail-5.14 will be deprecated and removed from the official FreeBSD Ports Collection next month, I've also created a Makefile for the FreeBSD Ports Collection and submitted a bug report to hopefully include getmail6 in the FreeBSD ports tree soon: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248633

setup.py install --show-default-install-dirs TypeError: expected str, bytes or os.PathLike object, not NoneType

OK, let's try #25, even with "install",

$ python3 setup.py install  --show-default-install-dirs
Default installation directories:
  scripts :        /usr/bin
Traceback (most recent call last):
  File "setup.py", line 51, in <module>
    print('  Python modules : %s' % os.path.join(distutils.sysconfig.get_config_var('LIBP'), 'site-packages'))
  File "/usr/lib/python3.8/posixpath.py", line 76, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

Probably not ready for python3.

Auto generate .txt files from .html and lock as readonly, or eliminate altogether

Apparently the .html and .txt files are indeed being maintained by hand separately.
I recommend using
$ w3m -dump file.html > file.txt

Therefore please make the .txt files in this repository read-only,
with a warning at top "Do not edit, created automatically from HTML version."

Sure, one could say "just have setup.py create the .txt files." But the user might not have w3m etc. installed.
So still distribute them.

Or, just consider not maintaining .txt files at all.

Misplaced Content-Length header when passing to procmail

I get automated emails from a specific provider. When asked to pass through procmail, some headers get misplaced and the email is then not properly interpreted by neomutt.

The problem happens when using a simple no-op procmail config. It does not happen if asking getmail to deliver to Maildir directly. The problem only happens with the getmail 6.0 python3 port (synced as of now) and does not happen with getmail 5.14 (python2).

When recieving using getmail 6.0 (master) passing through procmail, the e-mail headers look like:

 0102017367b2723b-6139904f-bdb6-43eb-8b5d-bdbcd4856d51-000000@eu-west-1.amazonses.com  Sun Jul 19 16:30:22 2020
Content-Length: 199664

Return-Path: <
  0102017367b2723b-6139904f-bdb6-43eb-8b5d-bdbcd4856d51-000000@eu-west-1.amazonses.com>
Delivered-To: unknown
Received: from outlook.office365.com (40.101.92.18:995) by jerome.isoxys with
  POP3-SSL; 19 Jul 2020 15:30:22 -0000
Received: from AM0P194MB0388.EURP194.PROD.OUTLOOK.COM (2603:10a6:102:57::16)
...
X-Microsoft-Antispam-Message-Info: 
        =?utf-8?B?Q3JxRkVIZmlFUVNlZFRrcWxQdDIyTTVtZjh6ZWlBcWM1UEZpMkhyM0FGS3gz?=
...
MIME-Version: 1.0

----boundary_260_4a334f9f-8e86-412f-bd2e-3f7071f4e267
Content-Type: multipart/alternative;
 boundary=--boundary_259_847894fc-b9fb-452f-b01b-b152987b68f2

When receiving with getmail 6.0 with direct Maildir delivery, or when using getmail 5.14 passing through procmail, the message looks like (I ask the provider to resend the message, contents are similar but not absolutely identical)

Return-Path: <
  0102017367a3ac51-f7ea524b-3962-463b-ab48-2fb10270dc54-000000@eu-west-1.amazonses.com>
Delivered-To: unknown
Received: from outlook.office365.com (52.97.173.2:995) by jerome.isoxys with
  POP3-SSL; 19 Jul 2020 15:14:14 -0000
Received: from VI1P194MB0191.EURP194.PROD.OUTLOOK.COM (2603:10a6:100:18::28)
...
X-Microsoft-Antispam-Message-Info:      =?utf-8?B?byttQ1JZdnlkZmJJSktDZGxIRDdqRWNjMkZKZjNZem9uV242dEw5WVF0MnFm?=
MIME-Version: 1.0
Content-Length: 187703

----boundary_256_5f87e3f9-3690-41db-a8f2-859d2ff5fed6
Content-Type: multipart/alternative;
 boundary=--boundary_255_8da70375-a5d5-4147-aa2a-63b4d7668131

The difference is that with getmail 6, the Content-Length is placed on top instead of after MIME-Version. It also comes with a timestamp that duplicates the Return-Path contents. I haven't understood the purpose of all the headers. In some cases by moving the Content-Length after MIME-Version, I was able to make mutt understand the message, but in other cases it was not able to understand the From:, Delivered-To or Date fields.

The full headers are attached, with just the username edited to example.com
delivery_Maildir.txt
delivery_procmail.txt

Slow mail retrieval

When fetching new email via getmail 6.7, each new mail causes ~1s delay.

A getmail command without new mails takes 0.5s:

$ time getmail -vvv
getmail version 6.7
Copyright (C) 1998-2019 Charles Cazabon.  Licensed under the GNU GPL version 2.
SimpleIMAPSSLRetriever:[email protected]@mail.cweiske.de:993:
[...]
  0 messages (0 bytes) retrieved, 48 skipped

real	0m0,479s
user	0m0,158s
sys	0m0,013s

But with 5 new test emails (small emails with only "test" in their subject and no body), it takes 5.5 seconds:

$ time getmail -vvv
getmail version 6.7
Copyright (C) 1998-2019 Charles Cazabon.  Licensed under the GNU GPL version 2.
SimpleIMAPSSLRetriever:[email protected]@mail.cweiske.de:993:
[...]
  msg 44/48 (1821 bytes) from <[email protected]> delivered to MDA_external command maildrop (b'')
  msg 45/48 (1821 bytes) from <[email protected]> delivered to MDA_external command maildrop (b'')
  msg 46/48 (1821 bytes) from <[email protected]> delivered to MDA_external command maildrop (b'')
  msg 47/48 (1821 bytes) from <[email protected]> delivered to MDA_external command maildrop (b'')
  msg 48/48 (1821 bytes) from <[email protected]> delivered to MDA_external command maildrop (b'')
  5 messages (9105 bytes) retrieved, 43 skipped
Summary:
Retrieved 5 messages (9105 bytes) from SimpleIMAPSSLRetriever:[email protected]@mail.cweiske.de:993

real	0m5,652s
user	0m0,296s
sys	0m0,128s

I did not have this slow behavior with getmail 5.

What could the cause for the slowness be?

Issue with procmail

I have been trying getmail 6 as a drop-in replacement for getmail 5.14, installing with python 3.7.7. I get an error when passing to procmail.

[destination]
type = MDA_external
path = /usr/bin/procmail
arguments = ("-f", "%(sender)", "-m", "~/.procmail/config.rc")

I send myself a simple plain text email and try to get it. (email address redacted)

Delivery error (command procmail 3177 error (127, exec of command procmail failed (replace() argument 2 must be str, not bytes)))
  msg 621/621 (33354 bytes) from <[email protected]>, delivery error (command procmail 3177 error (127, exec of command procmail failed (replace() argument 2 must be str, not bytes)))
  1 messages (33354 bytes) retrieved, 620 skipped

The message is actually not retrieved (procmail does not log any activity) and getmail does not mark it as seen. If I remove reference to procmail and deliver directly to the Maildir, it gets delivered.

Shutdown in the middle of a fetch does not write .getmail-data/ file.

I'm running getmail in a systemd service. On the first fetch (30,000 mail) I had to shutdown my computer. I assumed getmail would have written the .getmail-data/ file to keep track of the already delivered mail, and restarting would resume where it left off. However it looks like that file wasn't written and restarting the fetch will download everything again. Is there any way I can generate that file from the already delivered mail?

Traceback while trying to reap child process

I observe the following traceback:

getmail version 6.12                                                                                                                     
Copyright (C) 1998-2020 Charles Cazabon and others. Licensed under GNU GPL version 2.                                                    
SimpleIMAPSSLRetriever:[email protected]:993:                                                                                     
  0 messages (0 bytes) retrieved, 1170 skipped                                                                                           
SimpleIMAPSSLRetriever:forsch***@imap.fz-juelich.de:993:                                                                           
stopped waiting for child 1033                                                                                                           
Exception: please read docs/BUGS and include the following information in any bug report:                                                
                                                                                                                                         
  getmail version 6.12                                                                                                                   
  Python version 3.8.5 (default, Jul 28 2020, 12:59:40)                                                                                  
[GCC 9.3.0]                                                                                                                              
                                                                                                                                         Unhandled exception follows:                                                                                                                 File "/usr/local/bin/getmail", line 912, in main                                                                                         success = go(configs, options.idle)                                                                                                  
    File "/usr/local/bin/getmail", line 247, in go                                                                                           r = destination.deliver_message(msg,                                                                                                 
    File "/usr/local/lib/python3.8/dist-packages/getmailcore/destinations.py", line 111, in deliver_message                              
    return self._deliver_message(msg, delivered_to, received)                                                                            
    File "/usr/local/lib/python3.8/dist-packages/getmailcore/destinations.py", line 647, in _deliver_message                             
    child = self.forkchild(                                                                                                              
    File "/usr/local/lib/python3.8/dist-packages/getmailcore/baseclasses.py", line 481, in forkchild                                     
    child.exitcode = self._wait_for_child(child.childpid)                                                                                
    File "/usr/local/lib/python3.8/dist-packages/getmailcore/baseclasses.py", line 437, in _wait_for_child                               
    if os.WIFSTOPPED(self.__child_status):                                                                                               
  TypeError: an integer is required (got type NoneType)                                                                                  
                                                                                                                                         
Please also include configuration information from running getmail                                                                       
with your normal options plus "--dump".

The exit code is 4.

The option --dump yields:

getmail version 6.10                                                                                                                     
Copyright (C) 1998-2020 Charles Cazabon and others. Licensed under GNU GPL version 2.                                                    
getmail configuration:                                                                                                                   
  getmail version 6.10                                                                                                                   
  Python version 3.8.5 (default, Jul 28 2020, 12:59:40)       
[GCC 9.3.0]             
  retriever:  SimpleIMAPSSLRetriever(ca_certs="None", certfile="None", getmaildir="/var/lib/getmail", keyfile="None", mailboxes="('INBOX'
,)", move_on_delete="None", password="*", password_command="()", port="993", record_mailbox="True", server="imap.fz-juelich.de", ssl_cert
_hostname="None", ssl_ciphers="None", ssl_fingerprints="()", ssl_version="None", timeout="180", use_cram_md5="False", use_kerberos="False
", use_peek="True", use_xoauth2="False", username="juser")
  destination:  MDA_external(allow_root_commands="True", arguments="('bronger',)", command="my_procmail.py", group="None", ignore_stderr=
"True", path="/my_procmail.py", unixfrom="False", user="None")
  options:                   
    delete : False              
    delete_after : 0                                                
    delete_bigger_than : 0     
    delivered_to : False
    fingerprint : False
    logfile : logfile(filename="~/getmail.log")
    max_bytes_per_session : 0
    max_message_size : 0
    max_messages_per_session : 0
    message_log : ~/getmail.log
    message_log_syslog : False
    message_log_verbose : True
    read_all : False
    received : False
    verbose : 1

getmail seems to lose its child process and hangs forever

While retrieving messages from an IMAP server, getmail sometimes loses track of the child process (in my case /usr/local/lib/mailagent/filter), and hangs forever.

I'm using getmail 6.11 on FreeBSD 12-STABLE (installed via Ports Collection).
Here is my configuration. I'm using OAuth 2.0 for Exchange Online, but I believe it is irrelevant for this problem.

% getmail --dump
getmail version 6.11
Copyright (C) 1998-2020 Charles Cazabon and others. Licensed under GNU GPL version 2.
getmail configuration:
  getmail version 6.11
  Python version 3.7.9 (default, Oct 17 2020, 01:23:59) 
[Clang 8.0.1 (tags/RELEASE_801/final 366581)]
  retriever:  SimpleIMAPSSLRetriever(ca_certs="None", certfile="None", getmaildir="/home/kasahara/.getmail", keyfile="None", mailboxes="('INBOX',)", move_on_delete="None", password="*", password_command="('getmail-gmail-xoauth-tokens', '/home/kasahara/.getmail/exchange.json')", port="993", record_mailbox="True", server="outlook.office365.com", ssl_cert_hostname="None", ssl_ciphers="None", ssl_fingerprints="()", ssl_version="None", timeout="180", use_cram_md5="False", use_kerberos="False", use_peek="True", use_xoauth2="True", username="email@domainname")
  destination:  MDA_external(allow_root_commands="False", arguments="()", command="filter", group="None", ignore_stderr="False", path="/usr/local/lib/mailagent/filter", unixfrom="True", user="None")
  options:
    delete : False
    delete_after : 0
    delete_bigger_than : 0
    delivered_to : False
    fingerprint : False
    logfile : logfile(filename="~/.getmail/getmail.log")
    max_bytes_per_session : 0
    max_message_size : 0
    max_messages_per_session : 0
    message_log : ~/.getmail/getmail.log
    message_log_syslog : False
    message_log_verbose : False
    read_all : False
    received : True
    verbose : 0

% cat getmailrc
[retriever]
type = SimpleIMAPSSLRetriever
server = outlook.office365.com
username = email@domainname
use_xoauth2 = True
password_command = ("getmail-gmail-xoauth-tokens", "/home/kasahara/.getmail/exchange.json")

[destination]
type = MDA_external

path = /usr/local/lib/mailagent/filter
unixfrom = true

[options]
verbose = 0
read_all = false
delete = false
delete_after = 0
delete_bigger_than = 0
max_bytes_per_session = 0
max_message_size = 0
max_messages_per_session = 0
delivered_to = false
received = true
message_log = ~/.getmail/getmail.log
message_log_syslog = false
message_log_verbose = false

I tried getmail --trace and got the following output.

Normal case:

...(omit)
deliver_message() [destinations.py:107] trace
_deliver_message() [destinations.py:644] trace
msginfo "{'sender': 'someone@somedomain'}"
_prepare_child() [baseclasses.py:416] 
spawned child 67316
_wait_for_child() [baseclasses.py:426] waiting for child 67316_child_handler() [baseclasses.py:400] handler called for signal 20_child_handler() [baseclasses.py:410] handler reaped child 67316 with status 0
command filter 67316 exited 0
    delivered to MDA_external command filter (b"about to execl() with args ['/usr/local/lib/mailagent/filter', '/usr/local/lib/mailagent/filter']")
  msg 8078/8125 (93434 bytes) delivered
....(omit)

Blocked case:

...(omit)
deliver_message() [destinations.py:107] trace
_deliver_message() [destinations.py:644] trace
msginfo "{'sender': 'someone@somedomain'}"
_prepare_child() [baseclasses.py:416] 
spawned child 76563
_wait_for_child() [baseclasses.py:426] waiting for child 76563_child_handler() [baseclasses.py:400] handler called for signal 20_child_handler() [baseclasses.py:410] handler reaped child 76563 with status 0_wait_for_child() [baseclasses.py:426] waiting for child 76563_wait_for_child() [baseclasses.py:426] waiting for child 76563_wait_for_child() 
(repeated indefinitely)

When I killed the blocked getmail process and invoked again, I got a duplicate message, so I believe that the external filter program was successfully called and the message was delivered locally, but getmail couldn't detect it. I have no idea if the problem is in Python, getmail6, or FreeBSD.... I tried to call /usr/sbin/sendmail (with options needed) instead of mailagent's filter program, but the same problem occured.

Is it possible to let getmail spawn the external filter, immediately forget about it, and proceed to the next message (maybe with a configurable delay) ? I think getmail doesn't need to wait until mailagent's filter is finished. It handles locking by itself, and it also tries to go into daemon state. It is fine to invoke multiple 'filter' simultaneously.

unresolved reference to `getmail.spec` in `setup.py`

Hey, thx a lot for porting this to Python 3, well done !

During my attempts to package this for the AUR, I noticed that you removed the getmail.spec file (which actually seems to be used for rpm packaging, not sure though). However, I guess you forgot to remove the reference to that file in setup.py, causing the build process to bail out.

FWIW, this file is also referenced in MANIFEST.in, which is likely just an aesthetic issue though.

I have pushed a -git package btw, where those references are removed in the prepare() function, and mailwizard{,-git} now depend on this new package. :)

Add tests

I added a test folder, which contains some code that sets up docker-mailserver to run black box tests on getmail6.

There are almost no tests so far.

@emtiu Would you review the test setup and add one test regarding #67 (LMTP), as you said, you would look into that, when you have some time.

Unhandled Exception with Python3.8

Experienced the following when running getmail6 with Python 3.8.2:

Exception: please read docs/BUGS and include the following information in any bug report:

  getmail version 6.00
  Python version 3.8.2 (default, Apr 28 2020, 12:54:13)
[GCC 9.3.0]

Unhandled exception follows:
    File "/usr/bin/getmail", line 895, in main
    success = go(configs, options.idle)
    File "/usr/bin/getmail", line 165, in go
    retriever.select_mailbox(mailbox)
    File "/usr/lib/python3.8/site-packages/getmailcore/_retrieverbases.py", line 1484, in select_mailbox
    self._getmsglist(count)
    File "/usr/lib/python3.8/site-packages/getmailcore/_retrieverbases.py", line 1516, in _getmsglist
    for msgid in self.oldmail:
  RuntimeError: dictionary changed size during iteration

Solution:
Line 1516 in _retrieverbases.py had to be changed to:
for msgid in list(self.oldmail):

6.16 release file changed after first tag

After v6.16 was first tagged and signed, the release zip archive has changed once more:

Before:
SHA256 (getmail6-getmail6-v6.16_GH0.tar.gz) = 4f87f9e3c111005ac13df93a7d9cf7572da2da98e771607b2948aed638833795
SIZE (getmail6-getmail6-v6.16_GH0.tar.gz) = 194300

After:
SHA256 (getmail6-getmail6-v6.16_GH0.tar.gz) = b92b9f3367d2f26b4fc9a53785d8f960f587e38d513d77bd3bc430ca4ea39d2a
SIZE (getmail6-getmail6-v6.16_GH0.tar.gz) = 194301

V 6.00 SSL issues?

I downloaded the getmail-master.zip file containing version 6.00. I persuaded the gentoo linux package manager to install it and gave it a try.

An unencrypted connection to a POP3 server was successful.
Attempting to connect to different IMAP servers gave different errors.
2.1 Output

getmail version 6.00
Copyright (C) 1998-2019 Charles Cazabon.  Licensed under the GNU GPL version 2.
SimpleIMAPSSLRetriever:[email protected]:993:

Exception: please read docs/BUGS and include the following information in any bug report:

  getmail version 6.00
  Python version 3.7.8 (default, Jul 10 2020, 10:50:58) 
[GCC 9.3.0]

Unhandled exception follows:
    File "/usr/lib/python-exec/python3.7/getmail", line 895, in main
    success = go(configs, options.idle)
    File "/usr/lib/python-exec/python3.7/getmail", line 156, in go
    retriever.initialize(options)
    File "/usr/lib/python3.7/site-packages/getmailcore/_retrieverbases.py", line 1698, in initialize
    self._connect()
    File "/usr/lib/python3.7/site-packages/getmailcore/_retrieverbases.py", line 692, in _connect
    except socket.sslerror as o:
  AttributeError: module 'socket' has no attribute 'sslerror'

Please also include configuration information from running getmail
with your normal options plus "--dump".

2.2 Output

getmail version 6.00
Copyright (C) 1998-2019 Charles Cazabon.  Licensed under the GNU GPL version 2.
SimplePOP3SSLRetriever:[email protected]:995:

Exception: please read docs/BUGS and include the following information in any bug report:

  getmail version 6.00
  Python version 3.7.8 (default, Jul 10 2020, 10:50:58) 
[GCC 9.3.0]

Unhandled exception follows:
    File "/usr/lib/python-exec/python3.7/getmail", line 895, in main
    success = go(configs, options.idle)
    File "/usr/lib/python-exec/python3.7/getmail", line 156, in go
    retriever.initialize(options)
    File "/usr/lib/python3.7/site-packages/getmailcore/_retrieverbases.py", line 1152, in initialize
    self._connect()
    File "/usr/lib/python3.7/site-packages/getmailcore/_retrieverbases.py", line 437, in _connect
    ssl_version, ca_certs, ssl_ciphers
    File "/usr/lib/python3.7/site-packages/getmailcore/_retrieverbases.py", line 393, in __init__
    self.certfile, **extra_args)
    File "/usr/lib/python3.7/site-packages/getmailcore/_retrieverbases.py", line 126, in _wrap_socket
    return ssl.SSLSocket(**kwargs)
    File "/usr/lib/python3.7/ssl.py", line 808, in __init__
    f"{self.__class__.__name__} does not have a public "
  TypeError: SSLSocket does not have a public constructor. Instances are returned by SSLContext.wrap_socket().

Please also include configuration information from running getmail
with your normal options plus "--dump".

At this point I backed out my trial and reinstalled getmail-5.14 which handles all three of the above servers correctly. Without running the --dump option unfortunately.

Anyway the errors seem to point to porting issues not yet handled.

I will look into installing getmail-6.00 outside the package manager so I can play with it while my production version does its job.

Thanks for starting this port to python 3.

DaveF

mailbox not selectable when maildir name contains space

Hallo,

don't know if its my fault ...
Never used getmail before. I'm running getmail 6.4 on manjaro linux the first time.

`[retriever]
type = SimpleIMAPSSLRetriever
server = imap.goneo.de
port = 993
username = ...
password = ...
mailboxes = ALL

[destination]
type = Maildir
path = ~/mail/

[options]
verbose = 1
read_all = false
delete = false
delete_after = 7
delete_bigger_than = 1000000
received = false
delivered_to = false
`

When creating folders in the web-interface (roundcube) of my provider, and the folder name contains a space like this one INBOX/one two, I get an error when running getmail.

mailbox b'INBOX/one two' not selectable (INBOX/one two) - verify the mailbox exists and you have sufficient permissions

Henning

python setup.py install --show-default-install-dirs: Don't say "install"

docs/documentation.html says

You can see a list of the default installation locations by running:

# python setup.py install --show-default-install-dirs

Please just say to do

# python setup.py --show-default-install-dirs

As we are not trying to seemingly install anything, but just show things. So no need for the whole scary line.

(Sure, I could fix the code directly,
but one cannot tell if documentation.html is automatically produced from documentation.txt, or visa versa.
Or maybe you are maintaining them both by hand.)

Need to refresh the OAuth access token when reconnecting (IMAP IDLE)

My configuration:

getmail version 6.12
Copyright (C) 1998-2020 Charles Cazabon and others. Licensed under GNU GPL version 2.
getmail configuration:
  getmail version 6.12
  Python version 3.9.1 (default, Dec 12 2020, 09:45:22) 
[Clang 8.0.1 (tags/RELEASE_801/final 366581)]
  retriever:  SimpleIMAPSSLRetriever(ca_certs="None", certfile="None", getmaildir="/home/kasahara/.getmail", keyfile="None", mailboxes="('INBOX',)", move_on_delete="None", password="*", password_command="('getmail-gmail-xoauth-tokens', '/home/kasahara/.getmail/exchange.json')", port="993", record_mailbox="True", server="outlook.office365.com", ssl_cert_hostname="None", ssl_ciphers="None", ssl_fingerprints="()", ssl_version="None", timeout="180", use_cram_md5="False", use_kerberos="False", use_peek="True", use_xoauth2="True", username="(hidden)")
  destination:  MDA_external(allow_root_commands="False", arguments="()", command="filter", group="None", ignore_stderr="False", path="/usr/local/lib/mailagent/filter", unixfrom="True", user="None")
  options:
    delete : False
    delete_after : 0
    delete_bigger_than : 0
    delivered_to : False
    fingerprint : False
    logfile : logfile(filename="~/.getmail/getmail.log")
    max_bytes_per_session : 0
    max_message_size : 0
    max_messages_per_session : 0
    message_log : ~/.getmail/getmail.log
    message_log_syslog : True
    message_log_verbose : False
    read_all : False
    received : True
    verbose : 1

I'm using getmail6 with IMAP IDLE and OAuth 2.0 (Exchange Online). The access token is generated by getmail-gmail-xoauth-tokens. It seems that the server terminates the IMAP session when the access token is expired (the lifetime of the token is 3600 secs). getmail6 tries to reconnect, but conf['password'] was populated during the first iteration, so getmail6 uses the expired token again and fails to login (and exit).

getmail version 6.11
Copyright (C) 1998-2020 Charles Cazabon and others. Licensed under GNU GPL version 2.
SimpleIMAPSSLRetriever:(hidden)@outlook.office365.com:993:
  0 messages (0 bytes) retrieved, 9658 skipped
IDLE message received
  msg 9659/9659 (77823 bytes) delivered
  1 messages (77823 bytes) retrieved, 9658 skipped
(omit)
  0 messages (0 bytes) retrieved, 9663 skipped
  0 messages (0 bytes) retrieved, 9663 skipped
SimpleIMAPSSLRetriever:(hidden)@outlook.office365.com:993:
getmailrc: credential/login error (AUTHENTICATE failed.)
  0 messages (0 bytes) retrieved, 0 skipped

I believe that with use_xoauth2 = True, run_password_command() should be called every time before authentication to ensure using the up-to-date token.

Also, when python warning was enabled, I got a message about unclosed files. I think run_command() didn't close temporary files. Because it will be called many times after this fix, I think it is better to close them.

/usr/local/lib/python3.9/site-packages/getmailcore/_retrieverbases.py:931: ResourceWarning: unclosed file <_io.BufferedRandom name=4>
  rc, stdout, stderr = run_command(command,args)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.9/site-packages/getmailcore/_retrieverbases.py:931: ResourceWarning: unclosed file <_io.BufferedRandom name=5>
  rc, stdout, stderr = run_command(command,args)
ResourceWarning: Enable tracemalloc to get the object allocation traceback

Don't encode pure ASCII X-getmail-retrieved-from-mailbox

@jidanni has reported in Debian that even pure ASCII "INBOX" now gets encoded. and has requested to encode only when necessary.

Bug report - https://bugs.debian.org/982396

Copying from the bug report:

There was some change made in the last few months,

$ grep getmail *
37237:X-getmail-retrieved-from-mailbox: INBOX
37238:X-getmail-retrieved-from-mailbox: INBOX
37436:X-getmail-retrieved-from-mailbox: INBOX
38080:X-getmail-retrieved-from-mailbox: INBOX
38623:X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?=
38625:X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?=
38639:X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?=
38640:X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?=

that even pure ASCII "INBOX" now gets encoded.

Please don't just blindly encode pure ASCII. Only encode when actually
necessary. Thanks.

Regression: IDLE not working (and probably Google Extensions as well)

I just tried getmail6 for the first time and I'm getting the message that IDLE is not supported - at least for the server that I'm using at the moment: --idle given, but retriever does not support IDLE

With the old getmail this was working, so I had a look and it seems related to line 1658 in _retrieverbases.py

if 'IDLE' in self.conn.capabilities:

With python3 capabilities is a tuple of binary objects, but 'IDLE' is a string. If I change that to b'IDLE' or use tostr() in when parsing, IDLE is working again. The same thing is likely to happen with google extensions ('X-GM-EXT-1' in self.conn.capabilities) but I'm not using GMail so I wouldn't know.

I guess using tostr to make conn.capabilities a tuple of strings is the better approach - instead of having to prefix all capabilities with b?

Cheers,
Boris

getmail6: sometimes dies on running external filter program in version 6.13

Its a bug report in https://bugs.debian.org/980056 after upgrade to 6.13 from 6.11.

Copying relevant part here:

A snip of log:

 2021-01-13 16:23:09 getmailOperationError error (child pid 83739 killed by signal 8)
 2021-01-13 16:23:09   1 messages (19425 bytes) retrieved, 0 skipped
 2021-01-13 16:24:50 Initializing SimpleIMAPSSLRetriever:[email protected]@imap.gmail.com:993:
 2021-01-13 16:24:55 getmailOperationError error (child pid 84030 killed by signal 8)
 2021-01-13 16:24:55   1 messages (19425 bytes) retrieved, 0 skipped
 2021-01-13 16:26:37 Initializing SimpleIMAPSSLRetriever:[email protected]@imap.gmail.com:993:
 2021-01-13 16:26:42 getmailOperationError error (child pid 84309 killed by signal 8)
 2021-01-13 16:26:42   1 messages (19425 bytes) retrieved, 0 skipped

corresponding rc section:

 [filter-1]
 type = Filter_external
 path = /usr/bin/spamassassin
 ignore_stderr = True

Same bug report shows when the user downgraded to 6.11 it works again.

Retriever info in summary with --idle

Hi,

i use several imap --idle accounts and syslog logging and systemd services. Currently I am missing the account name or something else to have some more info:

A typical log extract currently looks like

Jan  4 18:11:59 host getmail[1989578]: IDLE message received
Jan  4 18:11:59 host getmail[1989578]:   0 messages (0 bytes) retrieved, 2449 skipped
Jan  4 18:11:59 host getmail[2001160]: IDLE message received
Jan  4 18:12:01 host getmail[2001160]:   0 messages (0 bytes) retrieved, 25746 skipped

I would like to have something like
Jan 4 18:11:59 host getmail[1989578]: 0 messages (0 bytes) retrieved, 2449 skipped from account

Document how to upgrade getmail6

The user successfully installs getmail6.
But sooner or later it comes time to upgrade to a new release.
OK, once #12 is finished, he will know how to remove getmail6, so he can install the new version from scracth.
However, maybe it is possible to "upgrade in place" without removing the old version of getmail6.
Please document how.
Yes, many users will be using e.g., Debian, where they can just do "apt-get upgrade". But before this is possible, it would still be best to document how users could upgrade getmail6 manually themselves.

getmail-gmail-xoauth-tokens: AttributeError: 'dict' object has no attribute 'iteritems'

In order to retrieve emails from google the XOAUTH2 protocol needs to be setup. This was working for me with getmail5 but not with getmail6 anymore.

I tried to setup it up from scratch following this tutorial: https://www.bytereef.org/howto/oauth2/getmail.html
But it is not working.

The gmail.json file I am using:

{"scope": "https://mail.google.com/",
 "user": "[email protected]",
 "client_id": "new-client-id-created-for-getmail6",
 "client_secret": "new-secret-coming-with-new-client-id",
 "token_uri": "https://accounts.google.com/o/oauth2/token",
 "auth_uri": "https://accounts.google.com/o/oauth2/auth",
 "redirect_uri": "urn:ietf:wg:oauth:2.0:oob",
 "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs"}

The output of getmail-gmail-xoauth-tokens --init gmail.json :

Visit this url to obtain a verification code:
Traceback (most recent call last):
  File "/usr/bin/getmail-gmail-xoauth-tokens", line 110, in <module>
    print("    %s\n" % auth.code_url())
  File "/usr/bin/getmail-gmail-xoauth-tokens", line 57, in code_url
    return '%s?%s' % (self.data['auth_uri'], self.query(params))
  File "/usr/bin/getmail-gmail-xoauth-tokens", line 49, in query
    for param in sorted(params.iteritems(), key=lambda x: x[0]):
AttributeError: 'dict' object has no attribute 'iteritems'

This is with Manjaro testing, newest getmail 6.4 and Python 3.8.5

These are the XOAUTH2 instructions from google:

https://developers.google.com/identity/protocols/oauth2/native-app

Following the link to the "Credentials Page" to create a new client id + secret for getmail6

unclosed file warnings

When I run getmail with python warnings enabled (PYTHONWARNINGS=d) I get messages about unclosed file handlers:

/usr/lib/python3.8/site-packages/getmailcore/_retrieverbases.py:965: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/[user]/.getmail/oldmail-pop.example-995-[mail]' mode='r' encoding='UTF-8'>
  self.read_oldmailfile(mailbox)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python-exec/python3.8/getmail:943: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/[user]/.getmail/getmailrc' mode='r' encoding='UTF-8'>
  main()
ResourceWarning: Enable tracemalloc to get the object allocation traceback

IMAP requests size, making `davmail` download all the messages every time, making it slow

I have been using getmail over davmail to connect to a Microsoft mailbox. I noticed that IMAP retrieval is very slow, even when there are no new messages. It seems getmail retrieves the entire message body even when it already saw them before (although it detects they were already seen and does not duplicate them in the local mailbox). With an inbox containing 438 messages, all seen, and it takes about 4-6 min for getmail to execute in IMAP, while the POP3 implementation takes a fraction of second. Other IMAP software also took 4-6 min to synchronise the full mailbox, and then a fraction of second to check for new messages.

fetchmail uses the trick of searching for unread messages only (which is a major limitation of fetchmail, does not keep an independent list of which messages were read)
thunderbird sends a NOOP to keep connection alive and ask for updates.

Would there be a way to avoid getmail downloading the body of each UID everytime, since it knows which UID have been seen already?

Below the davmail logs (simplified for 2 messages out of 438, the first and the last).

getmail configuration

verbose = 3
read_all = false
delete = false
delete_after = 50
message_log = ~/.getmail/log

recieved = true
delivered = true

#[retriever]
#type = SimplePOP3Retriever
#server = localhost
#username = [email protected]
#password = ********
#port = 1110

[retriever]
type = SimpleIMAPRetriever
server = localhost
username = [email protected]
password = ********
port = 1143

[destination]
type = MDA_external
path = /usr/bin/procmail
arguments = ("-f", "%(sender)", "-m", "~/.procmail/procmail.rc")

getmail / SimplePOP3Retriever

2021-02-21 02:00:42,270 DEBUG [davmail.pop.PopServer] davmail  - Connection from /0:0:0:0:0:0:0:1 on port 1110
2021-02-21 02:00:42,270 INFO  [davmail.pop.PopServer] davmail.connection  - CONNECT - 0:0:0:0:0:0:0:1:47776
2021-02-21 02:00:42,460 DEBUG [PopConnection-47776] davmail.exchange.ExchangeSession  - Test configuration status: 401
2021-02-21 02:00:42,460 DEBUG [PopConnection-47776] davmail  - > +OK DavMail 5.5.1-3299 POP ready at Sun Feb 21 02:00:42 WET 2021
2021-02-21 02:00:42,460 DEBUG [PopConnection-47776] davmail  - < USER [email protected]
2021-02-21 02:00:42,460 DEBUG [PopConnection-47776] davmail  - > +OK USER : [email protected]
2021-02-21 02:00:42,460 DEBUG [PopConnection-47776] davmail  - < PASS ********
2021-02-21 02:00:42,460 DEBUG [PopConnection-47776] davmail.exchange.ExchangeSession  - Got session davmail.exchange.ews.EwsExchangeSession@239cd145 from cache
2021-02-21 02:00:42,508 INFO  [PopConnection-47776] davmail.connection  - LOGON - 0:0:0:0:0:0:0:1:47776 [email protected]
2021-02-21 02:00:42,508 DEBUG [PopConnection-47776] davmail  - > +OK PASS
2021-02-21 02:00:42,508 DEBUG [PopConnection-47776] davmail  - < UIDL
2021-02-21 02:00:42,603 DEBUG [PopConnection-47776] davmail  - Downloaded 128 KBytes from /ews/exchange.asmx
2021-02-21 02:00:42,626 DEBUG [PopConnection-47776] davmail  - Downloaded 256 KBytes from /ews/exchange.asmx
2021-02-21 02:00:42,634 DEBUG [PopConnection-47776] davmail.exchange.ExchangeSession  - Folder INBOX - Search items current count: 438 fetchCount: 500 highest uid: 74132 lowest uid: 73170
2021-02-21 02:00:42,636 DEBUG [PopConnection-47776] davmail.exchange.ExchangeSession  - Message IMAP uid: 74132 uid: AAAAAJzu0G7W0uhCtiCT9NHNMTIBACDiK+8eQQtEtlMP5JWHh+QAAcBiJP4AAA== ItemId: AAMkADJhMjg5ZWUwLTIwNzctNGQwNS1hOTUwLWJkYzcyZmQ0NWY2MwBGAAAAAACc7tBu1tLoQrYgk/TRzTEyBwBORi0QtGx8RKJz6lAORO9kAAAAAAAOAAAg4ivvHkELRLZTD+SVh4fkAAHAYiT+AAA= ChangeKey: CwAAABYAAAAg4ivvHkELRLZTD+SVh4fkAAG/fmaT
2021-02-21 02:00:42,636 DEBUG [PopConnection-47776] davmail  - > +OK 438 messages (243095857 octets)
2021-02-21 02:00:42,636 DEBUG [PopConnection-47776] davmail  - > 1 AAAAAJzu0G7W0uhCtiCT9NHNMTIBACDiK+8eQQtEtlMP5JWHh+QAAbqgu4EAAA==
[...]
2021-02-21 02:00:42,639 DEBUG [PopConnection-47776] davmail  - > 438 AAAAAJzu0G7W0uhCtiCT9NHNMTIBACDiK+8eQQtEtlMP5JWHh+QAAcBiJP4AAA==
2021-02-21 02:00:42,639 DEBUG [PopConnection-47776] davmail  - > .
2021-02-21 02:00:42,679 DEBUG [PopConnection-47776] davmail  - < LIST
2021-02-21 02:00:42,679 DEBUG [PopConnection-47776] davmail  - > +OK 438 messages (243095857 octets)
2021-02-21 02:00:42,679 DEBUG [PopConnection-47776] davmail  - > 1 72188
[...]
2021-02-21 02:00:42,685 DEBUG [PopConnection-47776] davmail  - > 438 49233
2021-02-21 02:00:42,685 DEBUG [PopConnection-47776] davmail  - > .
2021-02-21 02:00:42,688 DEBUG [PopConnection-47776] davmail  - < UIDL
2021-02-21 02:00:42,689 DEBUG [PopConnection-47776] davmail  - > +OK 438 messages (243095857 octets)
2021-02-21 02:00:42,689 DEBUG [PopConnection-47776] davmail  - > 1 AAAAAJzu0G7W0uhCtiCT9NHNMTIBACDiK+8eQQtEtlMP5JWHh+QAAbqgu4EAAA==
[...]
2021-02-21 02:00:42,693 DEBUG [PopConnection-47776] davmail  - > 438 AAAAAJzu0G7W0uhCtiCT9NHNMTIBACDiK+8eQQtEtlMP5JWHh+QAAcBiJP4AAA==
2021-02-21 02:00:42,693 DEBUG [PopConnection-47776] davmail  - > .
2021-02-21 02:00:42,694 DEBUG [PopConnection-47776] davmail  - < LIST
2021-02-21 02:00:42,694 DEBUG [PopConnection-47776] davmail  - > +OK 438 messages (243095857 octets)
2021-02-21 02:00:42,694 DEBUG [PopConnection-47776] davmail  - > 1 72188
[...]
2021-02-21 02:00:42,696 DEBUG [PopConnection-47776] davmail  - > 438 49233
2021-02-21 02:00:42,696 DEBUG [PopConnection-47776] davmail  - > .
2021-02-21 02:00:42,766 DEBUG [PopConnection-47776] davmail  - < QUIT
2021-02-21 02:00:42,766 DEBUG [PopConnection-47776] davmail.exchange.ExchangeSession  - Delete messages in Trash not modified since Sat Jan 02 02:00:42 WET 2021
2021-02-21 02:00:42,816 DEBUG [PopConnection-47776] davmail  - > +OK Bye
2021-02-21 02:00:42,816 INFO  [PopConnection-47776] davmail.connection  - DISCONNECT - 0:0:0:0:0:0:0:1:47776

Thunderbird IMAP

2021-02-20 23:10:27,853 DEBUG [ImapConnection-38306] davmail  - < 8 noop
2021-02-20 23:10:27,909 DEBUG [ImapConnection-38306] davmail  - noop on INBOX
2021-02-20 23:10:27,959 DEBUG [ImapConnection-38306] davmail  - > 8 OK noop completed
2021-02-20 23:10:27,959 DEBUG [ImapConnection-38306] davmail  - < 9 UID fetch 74133:* (FLAGS)
2021-02-20 23:10:28,008 DEBUG [ImapConnection-38306] davmail  - > 9 OK UID FETCH completed

Fetchmail IMAP

fetchmail -vvv -f ~/.fetchmailrc

set logfile fetchmail.log
poll localhost protocol IMAP port 1143
user "[email protected]" pass "********" is "USERNAME" preconnect "date >> fetchmail.log"
no ssl
sslproto ''
keep
no rewrite
mda "/usr/bin/procmail -f %F -m ~/.procmail/procmail.rc";

2021-02-21 01:46:27,951 DEBUG [davmail.imap.ImapServer] davmail  - Connection from /0:0:0:0:0:0:0:1 on port 1143
2021-02-21 01:46:27,951 INFO  [davmail.imap.ImapServer] davmail.connection  - CONNECT - 0:0:0:0:0:0:0:1:42112
2021-02-21 01:46:28,135 DEBUG [ImapConnection-42112] davmail.exchange.ExchangeSession  - Test configuration status: 401
2021-02-21 01:46:28,135 DEBUG [ImapConnection-42112] davmail  - > * OK [CAPABILITY IMAP4REV1 AUTH=LOGIN MOVE SPECIAL-USE] IMAP4rev1 DavMail 5.5.1-3299 server ready
2021-02-21 01:46:28,135 DEBUG [ImapConnection-42112] davmail  - < A0001 CAPABILITY
2021-02-21 01:46:28,135 DEBUG [ImapConnection-42112] davmail  - > * CAPABILITY IMAP4REV1 AUTH=LOGIN MOVE SPECIAL-USE
2021-02-21 01:46:28,136 DEBUG [ImapConnection-42112] davmail  - > A0001 OK CAPABILITY completed
2021-02-21 01:46:28,136 DEBUG [ImapConnection-42112] davmail  - < LOGIN ********
2021-02-21 01:46:28,136 DEBUG [ImapConnection-42112] davmail.exchange.ExchangeSession  - Got session davmail.exchange.ews.EwsExchangeSession@239cd145 from cache
2021-02-21 01:46:28,320 INFO  [ImapConnection-42112] davmail.connection  - LOGON - 0:0:0:0:0:0:0:1:42112 [email protected]
2021-02-21 01:46:28,320 DEBUG [ImapConnection-42112] davmail  - > A0002 OK Authenticated
2021-02-21 01:46:28,320 DEBUG [ImapConnection-42112] davmail  - < A0003 SELECT "INBOX"
2021-02-21 01:46:28,528 DEBUG [ImapConnection-42112] davmail  - Downloaded 128 KBytes from /ews/exchange.asmx
2021-02-21 01:46:28,548 DEBUG [ImapConnection-42112] davmail  - Downloaded 256 KBytes from /ews/exchange.asmx
2021-02-21 01:46:28,568 DEBUG [ImapConnection-42112] davmail  - Downloaded 384 KBytes from /ews/exchange.asmx
2021-02-21 01:46:28,585 DEBUG [ImapConnection-42112] davmail.exchange.ExchangeSession  - Folder INBOX - Search items current count: 438 fetchCount: 500 highest uid: 74132 lowest uid: 73170
[...]
2021-02-21 01:46:28,585 DEBUG [ImapConnection-42112] davmail.exchange.ExchangeSession  - Message IMAP uid: 73170 uid: AAAAAJzu0G7W0uhCtiCT9NHNMTIBACDiK+8eQQtEtlMP5JWHh+QAAbqgu4EAAA== ItemId: AAMkADJhMjg5ZWUwLTIwNzctNGQwNS1hOTUwLWJkYzcyZmQ0NWY2MwBGAAAAAACc7tBu1tLoQrYgk/TRzTEyBwBORi0QtGx8RKJz6lAORO9kAAAAAAAOAAAg4ivvHkELRLZTD+SVh4fkAAG6oLuBAAA= ChangeKey: CQAAABYAAAAg4ivvHkELRLZTD+SVh4fkAAG5v7lm
2021-02-21 01:46:28,587 DEBUG [ImapConnection-42112] davmail.exchange.ExchangeSession  - Message IMAP uid: 74132 uid: AAAAAJzu0G7W0uhCtiCT9NHNMTIBACDiK+8eQQtEtlMP5JWHh+QAAcBiJP4AAA== ItemId: AAMkADJhMjg5ZWUwLTIwNzctNGQwNS1hOTUwLWJkYzcyZmQ0NWY2MwBGAAAAAACc7tBu1tLoQrYgk/TRzTEyBwBORi0QtGx8RKJz6lAORO9kAAAAAAAOAAAg4ivvHkELRLZTD+SVh4fkAAHAYiT+AAA= ChangeKey: CwAAABYAAAAg4ivvHkELRLZTD+SVh4fkAAG/fmaT
2021-02-21 01:46:28,588 DEBUG [ImapConnection-42112] davmail  - > * 438 EXISTS
2021-02-21 01:46:28,588 DEBUG [ImapConnection-42112] davmail  - > * 0 RECENT
2021-02-21 01:46:28,588 DEBUG [ImapConnection-42112] davmail  - > * OK [UIDVALIDITY 1]
2021-02-21 01:46:28,588 DEBUG [ImapConnection-42112] davmail  - > * OK [UIDNEXT 74134]
2021-02-21 01:46:28,588 DEBUG [ImapConnection-42112] davmail  - > * FLAGS (\Answered \Deleted \Draft \Flagged \Seen $Forwarded Junk)
2021-02-21 01:46:28,588 DEBUG [ImapConnection-42112] davmail  - > * OK [PERMANENTFLAGS (\Answered \Deleted \Draft \Flagged \Seen $Forwarded Junk \*)]
2021-02-21 01:46:28,588 DEBUG [ImapConnection-42112] davmail  - > A0003 OK [READ-WRITE] SELECT completed
2021-02-21 01:46:28,588 DEBUG [ImapConnection-42112] davmail  - < A0004 SEARCH UNSEEN UNDELETED
2021-02-21 01:46:28,762 DEBUG [ImapConnection-42112] davmail  - > * SEARCH
2021-02-21 01:46:28,762 DEBUG [ImapConnection-42112] davmail  - > A0004 OK SEARCH completed
2021-02-21 01:46:28,804 DEBUG [ImapConnection-42112] davmail  - < A0005 LOGOUT
2021-02-21 01:46:28,804 DEBUG [ImapConnection-42112] davmail  - > * BYE Closing connection
2021-02-21 01:46:28,804 DEBUG [ImapConnection-42112] davmail  - > A0005 OK LOGOUT completed
2021-02-21 01:46:28,804 INFO  [ImapConnection-42112] davmail.connection  - DISCONNECT - 0:0:0:0:0:0:0:1:42112

getmail IMAP

2021-02-21 01:23:57,027 DEBUG [davmail.imap.ImapServer] davmail  - Connection from /0:0:0:0:0:0:0:1 on port 1143
2021-02-21 01:23:57,027 INFO  [davmail.imap.ImapServer] davmail.connection  - CONNECT - 0:0:0:0:0:0:0:1:41838
2021-02-21 01:23:57,218 DEBUG [ImapConnection-41838] davmail.exchange.ExchangeSession  - Test configuration status: 401
2021-02-21 01:23:57,219 DEBUG [ImapConnection-41838] davmail  - > * OK [CAPABILITY IMAP4REV1 AUTH=LOGIN MOVE SPECIAL-USE] IMAP4rev1 DavMail 5.5.1-3299 server ready
2021-02-21 01:23:57,219 DEBUG [ImapConnection-41838] davmail  - < PPPN0 CAPABILITY
2021-02-21 01:23:57,219 DEBUG [ImapConnection-41838] davmail  - > * CAPABILITY IMAP4REV1 AUTH=LOGIN MOVE SPECIAL-USE
2021-02-21 01:23:57,219 DEBUG [ImapConnection-41838] davmail  - > PPPN0 OK CAPABILITY completed
2021-02-21 01:23:57,220 DEBUG [ImapConnection-41838] davmail  - < LOGIN ********
2021-02-21 01:23:57,220 DEBUG [ImapConnection-41838] davmail.exchange.ExchangeSession  - Got session davmail.exchange.ews.EwsExchangeSession@239cd145 from cache
2021-02-21 01:23:57,267 INFO  [ImapConnection-41838] davmail.connection  - LOGON - 0:0:0:0:0:0:0:1:41838 [email protected]
2021-02-21 01:23:57,267 DEBUG [ImapConnection-41838] davmail  - > PPPN1 OK Authenticated
2021-02-21 01:23:57,267 DEBUG [ImapConnection-41838] davmail  - < PPPN2 CAPABILITY
2021-02-21 01:23:57,267 DEBUG [ImapConnection-41838] davmail  - > * CAPABILITY IMAP4REV1 AUTH=LOGIN MOVE SPECIAL-USE
2021-02-21 01:23:57,267 DEBUG [ImapConnection-41838] davmail  - > PPPN2 OK CAPABILITY completed
2021-02-21 01:23:57,269 DEBUG [ImapConnection-41838] davmail  - < PPPN3 SELECT INBOX
2021-02-21 01:23:57,495 DEBUG [ImapConnection-41838] davmail  - Downloaded 128 KBytes from /ews/exchange.asmx
2021-02-21 01:23:57,524 DEBUG [ImapConnection-41838] davmail  - Downloaded 256 KBytes from /ews/exchange.asmx
2021-02-21 01:23:57,554 DEBUG [ImapConnection-41838] davmail  - Downloaded 385 KBytes from /ews/exchange.asmx
2021-02-21 01:23:57,578 DEBUG [ImapConnection-41838] davmail.exchange.ExchangeSession  - Folder INBOX - Search items current count: 438 fetchCount: 500 highest uid: 74132 lowest uid: 73170
2021-02-21 01:23:57,578 DEBUG [ImapConnection-41838] davmail.exchange.ExchangeSession  - Message IMAP uid: 73170 uid: AAAAAJzu0G7W0uhCtiCT9NHNMTIBACDiK+8eQQtEtlMP5JWHh+QAAbqgu4EAAA== ItemId: AAMkADJhMjg5ZWUwLTIwNzctNGQwNS1hOTUwLWJkYzcyZmQ0NWY2MwBGAAAAAACc7tBu1tLoQrYgk/TRzTEyBwBORi0QtGx8RKJz6lAORO9kAAAAAAAOAAAg4ivvHkELRLZTD+SVh4fkAAG6oLuBAAA= ChangeKey: CQAAABYAAAAg4ivvHkELRLZTD+SVh4fkAAG5v7lm
2021-02-21 01:23:57,580 DEBUG [ImapConnection-41838] davmail.exchange.ExchangeSession  - Message IMAP uid: 74132 uid: AAAAAJzu0G7W0uhCtiCT9NHNMTIBACDiK+8eQQtEtlMP5JWHh+QAAcBiJP4AAA== ItemId: AAMkADJhMjg5ZWUwLTIwNzctNGQwNS1hOTUwLWJkYzcyZmQ0NWY2MwBGAAAAAACc7tBu1tLoQrYgk/TRzTEyBwBORi0QtGx8RKJz6lAORO9kAAAAAAAOAAAg4ivvHkELRLZTD+SVh4fkAAHAYiT+AAA= ChangeKey: CwAAABYAAAAg4ivvHkELRLZTD+SVh4fkAAG/fmaT
2021-02-21 01:23:57,581 DEBUG [ImapConnection-41838] davmail  - > * 438 EXISTS
2021-02-21 01:23:57,581 DEBUG [ImapConnection-41838] davmail  - > * 37 RECENT
2021-02-21 01:23:57,581 DEBUG [ImapConnection-41838] davmail  - > * OK [UIDVALIDITY 1]
2021-02-21 01:23:57,581 DEBUG [ImapConnection-41838] davmail  - > * OK [UIDNEXT 74134]
2021-02-21 01:23:57,581 DEBUG [ImapConnection-41838] davmail  - > * FLAGS (\Answered \Deleted \Draft \Flagged \Seen $Forwarded Junk)
2021-02-21 01:23:57,581 DEBUG [ImapConnection-41838] davmail  - > * OK [PERMANENTFLAGS (\Answered \Deleted \Draft \Flagged \Seen $Forwarded Junk \*)]
2021-02-21 01:23:57,581 DEBUG [ImapConnection-41838] davmail  - > PPPN3 OK [READ-WRITE] SELECT completed
2021-02-21 01:23:57,625 DEBUG [ImapConnection-41838] davmail  - < PPPN4 FETCH 1:438 (UID RFC822.SIZE)
2021-02-21 01:23:57,673 DEBUG [ImapConnection-41838] davmail.imap.ImapConnection  - * 1 FETCH (UID 73170
2021-02-21 01:23:57,757 DEBUG [ImapConnection-41838] davmail.exchange.ExchangeSession  - Downloaded full message content for IMAP UID 73170 (12399 bytes)
2021-02-21 01:23:57,757 DEBUG [ImapConnection-41838] davmail  - >  RFC822.SIZE 12399)
2021-02-21 01:24:02,732 DEBUG [ImapConnection-41838] davmail.exchange.MessageLoadThread  - Load large message 5035KB uid AAAAAJzu0G7W0uhCtiCT9NHNMTIBACDiK+8eQQtEtlMP5JWHh+QAAbqgu6UAAA== imapUid 73252 in a separate thread
2021-02-21 01:24:03,075 DEBUG [ImapConnection-41838-LoadMessage] davmail  - Downloaded 128 KBytes from /ews/exchange.asmx
2021-02-21 01:24:07,810 DEBUG [ImapConnection-41838] davmail.exchange.MessageLoadThread  - Still loading uid AAAAAJzu0G7W0uhCtiCT9NHNMTIBACDiK+8eQQtEtlMP5JWHh+QAAbqgu6UAAA== imapUid 73252
[...]
2021-02-21 01:27:36,402 DEBUG [ImapConnection-41838-LoadMessage] davmail  - Downloaded 7175 KBytes from /ews/exchange.asmx
2021-02-21 01:27:37,135 DEBUG [ImapConnection-41838] davmail.imap.ImapConnection  - * 438 FETCH (UID 74132
2021-02-21 01:27:37,203 DEBUG [ImapConnection-41838] davmail.exchange.ExchangeSession  - Downloaded full message content for IMAP UID 74132 (14731 bytes)
2021-02-21 01:27:37,203 DEBUG [ImapConnection-41838] davmail  - >  RFC822.SIZE 14731)
2021-02-21 01:27:37,203 DEBUG [ImapConnection-41838] davmail  - > PPPN4 OK FETCH completed
2021-02-21 01:27:37,243 DEBUG [ImapConnection-41838] davmail  - < PPPN5 CLOSE
2021-02-21 01:27:37,296 DEBUG [ImapConnection-41838] davmail  - > PPPN5 OK CLOSE completed
2021-02-21 01:27:37,305 DEBUG [ImapConnection-41838] davmail  - < PPPN6 LOGOUT
2021-02-21 01:27:37,305 DEBUG [ImapConnection-41838] davmail  - > * BYE Closing connection
2021-02-21 01:27:37,305 DEBUG [ImapConnection-41838] davmail  - > PPPN6 OK LOGOUT completed
2021-02-21 01:27:37,305 INFO  [ImapConnection-41838] davmail.connection  - DISCONNECT - 0:0:0:0:0:0:0:1:41838

Hotmail SimpleIMAPSSLRetriever started failing

Started getting this:

getmail version 6.9
Copyright (C) 1998-2020 Charles Cazabon and others. Licensed under GNU GPL version 2.
SimpleIMAPSSLRetriever:[email protected]@imap-mail.outlook.com:993:

Exception: please read docs/BUGS and include the following information in any bug report:

  getmail version 6.9
  Python version 3.9.0 (default, Oct  6 2020, 00:00:00)
[GCC 10.2.1 20200826 (Red Hat 10.2.1-3)]

Unhandled exception follows:
    File "/usr/local/bin/getmail", line 907, in main
    success = go(configs, options.idle)
    File "/usr/local/bin/getmail", line 213, in go
    msg = retriever.getmsg(msgid)
    File "/usr/local/lib/python3.9/site-packages/getmailcore/_retrieverbases.py", line 911, in getmsg
    return self._getmsgbyid(msgid)
    File "/usr/local/lib/python3.9/site-packages/getmailcore/_retrieverbases.py", line 1599, in _getmsgbyid
    return self._getmsgpartbyid(msgid, part)
    File "/usr/local/lib/python3.9/site-packages/getmailcore/_retrieverbases.py", line 1531, in _getmsgpartbyid
    msg = Message(fromstring=sbody)
    File "/usr/local/lib/python3.9/site-packages/getmailcore/message.py", line 134, in __init__
    self.sender = address_no_brackets(self.__msg['Return-Path']
    File "/usr/local/lib/python3.9/site-packages/getmailcore/utilities.py", line 383, in address_no_brackets
    if addr.startswith('<') and addr.endswith('>'):
  AttributeError: 'Header' object has no attribute 'startswith'

Please also include configuration information from running getmail
with your normal options plus "--dump".

getmail version 6.9
Copyright (C) 1998-2020 Charles Cazabon and others. Licensed under GNU GPL version 2.
getmail configuration:
  getmail version 6.9
  Python version 3.9.0 (default, Oct  6 2020, 00:00:00)
[GCC 10.2.1 20200826 (Red Hat 10.2.1-3)]
  retriever:  SimpleIMAPSSLRetriever(ca_certs="None", certfile="None", getmaildir="/var/mail/vmail/.getmail/", keyfile="None", mailboxes="('INBOX', 'Junk', 'Sent')", move_on_delete="None", password="*", password_command="()", port="993", record_mailbox="True", server="imap-mail.outlook.com", ssl_cert_hostname="None", ssl_ciphers="None", ssl_fingerprints="()", ssl_version="None", timeout="180", use_cram_md5="False", use_kerberos="False", use_peek="True", use_xoauth2="False", username="[email protected]")
  destination:  MDA_external(allow_root_commands="False", arguments="('-e', '-d', '[email protected]')", command="deliver", group="None", ignore_stderr="False", path="/usr/libexec/dovecot/deliver", unixfrom="False", user="None")
  options:
    delete : True
    delete_after : 0
    delete_bigger_than : 0
    delivered_to : False
    fingerprint : False
    logfile : logfile(filename="/var/log/getmail.log")
    max_bytes_per_session : 0
    max_message_size : 0
    max_messages_per_session : 0
    message_log : /var/log/getmail.log
    message_log_syslog : False
    message_log_verbose : False
    read_all : False
    received : False
    verbose : 1

Bring back "getmails" wrapper

Hi,

Is there a particular reason "getmails" wrapper script has been dropped in version 6? I have been using it on all my computers for years (with version 5) to fetch mail from all my many email accounts and found it very useful. It is a simple Bash wrapper, so was easy to copy from an old package but some users may be unpleasantly surprised by finding their mail system completely broken after the upgrade just by this small omission.

Thanks,
Bamyasi

current version numbering is incompatible with setuptools

When using setuptools to install getmail6, the script spouts this warning:

/usr/local/lib/python3.7/site-packages/setuptools/dist.py:476: UserWarning: Normalizing '6.03' to '6.3'`

This affects the location of metadata handled by setuptools, like these files:

/usr/local/lib/python3.7/site-packages/getmail-6.3-py3.7.egg-info/PKG-INFO
/usr/local/lib/python3.7/site-packages/getmail-6.3-py3.7.egg-info/SOURCES.txt
/usr/local/lib/python3.7/site-packages/getmail-6.3-py3.7.egg-info/dependency_links.txt
/usr/local/lib/python3.7/site-packages/getmail-6.3-py3.7.egg-info/top_level.txt

Judging from the answer to this StackOverflow question, we're probably in the wrong here, because 6.03 is never acceptable to Python's setuptools.

I strongly suggest that we move the versioning scheme to 6.0.4 at the next opportunity, because this could cause us all sorts of trouble down the line (like when an actual version 6.3 normalizes to the same version string as 6.03).

Also note that for Python internally, the package is still called getmail and not getmail6. I'm not sure if that's a problem. The last major revision that Charles Cazabon published was 5.0 in 2017. (If this is a concern, we might move it to a separate issue.)

After upgrading to Fedora 33 with python 3.9 getmail6 stopped working

systemd[1]: Started Getmail on xxxxxx.
Oct 30 19:56:03 wintermute getmail[26896]: Exception: please read docs/BUGS and include the following information in any bug report:
Oct 30 19:56:03 wintermute getmail[26896]:  getmail version 6.8
Oct 30 19:56:03 wintermute getmail[26896]:  Python version 3.9.0 (default, Oct  6 2020, 00:00:00)
Oct 30 19:56:03 wintermute getmail[26896]: [GCC 10.2.1 20200826 (Red Hat 10.2.1-3)]
Oct 30 19:56:03 wintermute getmail[26896]: Unhandled exception follows:
Oct 30 19:56:03 wintermute getmail[26896]:    File "/usr/local/bin/getmail", line 889, in main
Oct 30 19:56:03 wintermute getmail[26896]:    success = go(configs, options.idle)
Oct 30 19:56:03 wintermute getmail[26896]:    File "/usr/local/bin/getmail", line 173, in go
Oct 30 19:56:03 wintermute getmail[26896]:    retriever.select_mailbox(mailbox)
Oct 30 19:56:03 wintermute getmail[26896]:    File "/usr/local/lib/python3.9/site-packages/getmailcore/_retrieverbases.py", line 1393, in select_mailbox
Oct 30 19:56:03 wintermute getmail[26896]:    codecs.encode(
Oct 30 19:56:03 wintermute getmail[26896]:  LookupError: unknown encoding: imap4-utf-7

getmail configuration:
  getmail version 6.8
  Python version 3.9.0 (default, Oct  6 2020, 00:00:00)
[GCC 10.2.1 20200826 (Red Hat 10.2.1-3)]
  retriever:  SimpleIMAPSSLRetriever(ca_certs="None", certfile="None", getmaildir="/var/mail/vmail/.getmail/", keyfile="None", mailboxes="('[Gmail]/All Mail',)", move_on_delete=""[Gmail]/Trash"", password="*", password_command="()", port="993", record_mailbox="True", server="imap.gmail.com", ssl_cert_hostname="None", ssl_ciphers="None", ssl_fingerprints="()", ssl_version="None", timeout="180", use_cram_md5="False", use_kerberos="False", use_peek="True", use_xoauth2="False", username="[email protected]")
  destination:  MDA_external(allow_root_commands="False", arguments="('-e', '-d', '[email protected]')", command="deliver", group="None", ignore_stderr="False", path="/usr/libexec/dovecot/deliver", unixfrom="False", user="None")
  options:
    delete : True
    delete_after : 0
    delete_bigger_than : 0
    delivered_to : False
    fingerprint : False
    logfile : logfile(filename="/var/log/getmail.log")
    max_bytes_per_session : 0
    max_message_size : 0
    max_messages_per_session : 0
    message_log : /var/log/getmail.log
    message_log_syslog : False
    message_log_verbose : False
    read_all : False
    received : False
    verbose : 1

TypeError: 'gaierror' object is not subscriptable

When connected to the Internet, getmail6 works fine.
But when not, the error message doesn't say
"PLUG THE INTERNET CORD IN AND TRY AGAIN"
but instead it says

  getmail version 6.4
  Python version 3.8.5 (default, Aug  2 2020, 15:09:07) 
[GCC 10.2.0]

Unhandled exception follows:
    File "/home/jidanni/tmp/getmail/bin/getmail", line 881, in main
    success = go(configs, options.idle)
    File "/home/jidanni/tmp/getmail/bin/getmail", line 156, in go
    retriever.initialize(options)
    File "/home/jidanni/tmp/getmail/lib/python3.8/site-packages/getmailcore/_retrieverbases.py", line 1601, in initialize
    self._connect()
    File "/home/jidanni/tmp/getmail/lib/python3.8/site-packages/getmailcore/_retrieverbases.py", line 605, in _connect
    errcode = o[0]
  TypeError: 'gaierror' object is not subscriptable

The problem is due to the error not being ready for python3
https://stackoverflow.com/a/6800507/1118719

Document how to cleanly remove getmail6

README and docs/documentation.html mention

installation

but neglect mentioning

updating
removal

(Maybe instead of tinkering with those files, just put all that bonus info into a new README.getmail6 .)

oldmail files created with single quote in filename

The oldmail index files are being created with single single quotes around the filename.
Example: '[email protected]'

non-compliant e-mail header (in multidrop configuration) causes getmail to stop working

The attached email causes getmail (in a multidrop configuration) to stop working. The root cause is probably the non-compliant email header:
Urgent Notification Of Company Business Name Changes Asia.txt

“X-AntiAbuse: This header was added to track abuse, please include it with any abuse
report
X-AntiAbuse: Primary Hostname - hst.sunucuyeri.net
X-AntiAbuse: Original Domain - test.at
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - alkawtharnational.com
X-Get-Message-Sender-Via: hst.sunucuyeri.net: none
X-Authenticated-Sender: hst.sunucuyeri.net:
X-Source:
X-Source-Args:
X-Source-Dir: ”
Content-Type: multipart/mixed; boundary="----=_NextPart_000_0012_A796884C.DCABE8FF"
X-Envelope-To: [email protected]

I suspect that the two "DOUBLE QUOTATION MARK's" at the beginning and the end of the "anti-abuse-header" causing that the "python email.header.decode_header() function" cannot find the neccesary header field "envelope_recipient", which is located at the end of the header. Then getmail stopps working with the following error message:

Mar 16 04:56:46 Mailserver getmail[985]: Configuration error: envelope_recipient specified header missing (X-Envelope-To:1)
Mar 16 04:56:46 Mailserver systemd[1]: mail.getmail.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 16 04:56:46 Mailserver systemd[1]: mail.getmail.service: Failed with result 'exit-code'.

The problem is that getmail gets stuck on this e-mail and therefore does not fetch any further e-mails from the server.
Are there any suggestions how to solve this problem?
Best regards,
valbewe

Release?

Hi.
I am a packager for the Linux distro Mageia, and I just stumbled across your fork.

Do you plan to do a release / tag at any point? If so, when? :-)

'ascii' codec can't encode characters in position 4017-4018

I'm getting the following error:

getmail version 6.15
Copyright (C) 1998-2020 Charles Cazabon and others. Licensed under GNU GPL version 2.
SimpleIMAPSSLRetriever:[email protected]@mail.gandi.net:993:
Delivery error (command procmail 29762 error (127, exec of command procmail failed ('ascii' codec can't encode characters in position 4017-4018: ordinal not in range(128))))
  msg 1/6 (12391 bytes), delivery error (command procmail 29762 error (127, exec of command procmail failed ('ascii' codec can't encode characters in position 4017-4018: ordinal not in range(128))))

I can make the email available if someone can tell me how to download it.

Update URL on github repository

On our github main page, the URL for the website still points to Charles' website. We probably want to update that to either getmail6.org or this github page.

Migrating a getmail installation

I am migrating an old getmail4 installation, which is part of a mailman2 setup to a new server.
The work flow is:
email -> gmail -> collected by getmail -> server -> mailman2 -> postfix -> users

Have rebuilt this ok for a test list mailman list. getmail6 works seamlessly as a drop in replacement.
I am concerned when migrating the two main lists that old messages should not be re-downloaded.

The larger of the two lists has over 2500 messages on the gmail server
The messages are left on the server, and are not marked as read
- except for the occasional instance when I pop in directly to see the messages on gmail)
I had thought that the oldmail lists were there to stop multi-downloading
BUT, the oldmail files are very small. For example
The [email protected] for the list with 2500 mail entries has only seven (7) lines in it
The other oldmail one is slightly larger and has thirty six (36) lines in it.

But reading the documentation, I don't see anything that gives me confidence that getmail will only download relatively recent emails, so that the size of the oldmail files does not have to be very large.

A bit confused.

Make a .deb so we can continue to use getmail on debian

Can you please make a .deb available?
As you can see I can no longer use getmail on debian:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=968194

Unhandled exceptions with IMAP IDLE + OAuth2.0

My configuration:

% getmail -iINBOX --dump
getmail version 6.12
Copyright (C) 1998-2020 Charles Cazabon and others. Licensed under GNU GPL version 2.
getmail configuration:
  getmail version 6.12
  Python version 3.9.1 (default, Dec 12 2020, 09:45:22)
[Clang 8.0.1 (tags/RELEASE_801/final 366581)]
  retriever:  SimpleIMAPSSLRetriever(ca_certs="None", certfile="None", getmaildir="/home/kasahara/.getmail", keyfile="None", mailboxes="('INBOX',)", move_on_delete="None", password="*", password_command="('getmail-gmail-xoauth-tokens', '/home/kasahara/.getmail/exchange.json')", port="993", record_mailbox="True", server="outlook.office365.com", ssl_cert_hostname="None", ssl_ciphers="None", ssl_fingerprints="()", ssl_version="None", timeout="180", use_cram_md5="False", use_kerberos="False", use_peek="True", use_xoauth2="True", username="(hidden)")
  destination:  MDA_external(allow_root_commands="False", arguments="()", command="filter", group="None", ignore_stderr="False", path="/usr/local/lib/mailagent/filter", unixfrom="True", user="None")
  options:
    delete : False
    delete_after : 0
    delete_bigger_than : 0
    delivered_to : False
    fingerprint : False
    logfile : logfile(filename="~/.getmail/getmail.log")
    max_bytes_per_session : 0
    max_message_size : 0
    max_messages_per_session : 0
    message_log : ~/.getmail/getmail.log
    message_log_syslog : True
    message_log_verbose : False
    read_all : False
    received : True
    verbose : 1

While fetching messages from Exchange Online with IMAP IDLE + OAuth2.0, I occasionally got unhandled exceptions.

IDLE message received
  msg 10023/10023 (266372 bytes) delivered
  1 messages (266372 bytes) retrieved, 10022 skipped from SimpleIMAPSSLRetriever:(hidden)@outlook.office365.com:993
  0 messages (0 bytes) retrieved, 10023 skipped from SimpleIMAPSSLRetriever:(hidden)@[email protected]:993

Exception: please read docs/BUGS and include the following information in any bug report:

  getmail version 6.12
  Python version 3.9.1 (default, Dec 12 2020, 09:45:22)
[Clang 8.0.1 (tags/RELEASE_801/final 366581)]

Unhandled exception follows:
    File "/usr/local/bin/getmail", line 916, in main
    success = go(configs, options.idle)
    File "/usr/local/bin/getmail", line 439, in go
    retriever.close_mailbox()
    File "/usr/local/lib/python3.9/site-packages/getmailcore/_retrieverbases.py", line 1359, in close_mailbox
    self.conn.close()
    File "/usr/local/lib/python3.9/imaplib.py", line 475, in close
    typ, dat = self._simple_command('CLOSE')
    File "/usr/local/lib/python3.9/imaplib.py", line 1230, in _simple_command
    return self._command_complete(name, self._command(name, *args))
    File "/usr/local/lib/python3.9/imaplib.py", line 1049, in _command_complete
    raise self.abort('command: %s => %s' % (name, val))
  imaplib.IMAP4.abort: command: CLOSE => Session invalidated - AccessTokenExpired

Please also include configuration information from running getmail
with your normal options plus "--dump".

IDLE message received

Exception: please read docs/BUGS and include the following information in any bug report:

  getmail version 6.12
  Python version 3.9.1 (default, Dec 12 2020, 09:45:22)
[Clang 8.0.1 (tags/RELEASE_801/final 366581)]

Unhandled exception follows:
    File "/usr/local/bin/getmail", line 916, in main
    success = go(configs, options.idle)
    File "/usr/local/bin/getmail", line 441, in go
    idling = retriever.go_idle(idle)
    File "/usr/local/lib/python3.9/site-packages/getmailcore/_retrieverbases.py", line 1757, in go_idle
    self.conn.send(b'DONE\r\n')
    File "/usr/local/lib/python3.9/imaplib.py", line 332, in send
    self.sock.sendall(data)
    File "/usr/local/lib/python3.9/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
    File "/usr/local/lib/python3.9/ssl.py", line 1173, in send
    return self._sslobj.write(data)
  BrokenPipeError: [Errno 32] Broken pipe

Please also include configuration information from running getmail
with your normal options plus "--dump".

It seems that the server invalidates the session due to token expiration where getmail6 doesn't expect it.

Also I got this once.

IDLE message received
Retrieval error: server for SimpleIMAPSSLRetriever:(hidden)@outlook.office365.com:993 is broken; offered message 14/238306 but failed to provide it.  Please notify the administrator of the server.  Skipping message...
  0 messages (0 bytes) retrieved, 9870 skipped
IMAP error during logout (Session invalidated - AccessTokenExpired)

I'm afraid that I don't know how to handle these exceptions properly in these contexts. I guess it is ok to catch these exceptions and re-initialize the retriever to go idle again.

Unhandled exception, UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 1031: invalid start byte

Hello everyone...

I am calling getmail from IMAP Grab 0.1.4 on Raspian jessie and get the following error:

Downloading mailbox "Gesendet"

Exception: please read docs/BUGS and include the following information in any bug report:

getmail version 6.10
Python version 3.4.2 (default, Sep 16 2019, 19:58:00)
[GCC 4.9.2]

Unhandled exception follows:
File "/usr/local/bin/getmail", line 809, in main
destination = destination_func(**destination_args)
File "/usr/local/lib/python3.4/dist-packages/getmailcore/destinations.py", line 86, in init
ConfigurableBase.init(self, **args)
File "/usr/local/lib/python3.4/dist-packages/getmailcore/baseclasses.py", line 356, in init
self.checkconf()
File "/usr/local/lib/python3.4/dist-packages/getmailcore/baseclasses.py", line 365, in checkconf
self.conf[item.name] = item.validate(self.conf)
File "/usr/local/lib/python3.4/dist-packages/getmailcore/baseclasses.py", line 307, in validate
first_line = f.readline()
File "/usr/lib/python3.4/codecs.py", line 313, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 1871: invalid start byte

ocbak@hugo:~ $ getmail --dump
Error: Default config/data dir "/share/ocbak/.getmail/" does not exist - create or specify alternate directory with --getmaildir option

=====================================

It happens only in 2 of 20 mail folders.
Is this an Issue of IMAP Grab, getmail or Python itself?

What can I do?

Thanks a lot

Please sign your release tags

@rpuntaie many thanks for providing getmail6. To distribute your work it would be helpful if you sign your release tags with your gpg key.

Thanks in advance

can't concat str to bytes

Hi,

I moved from getmail 5 (which I have been using for many years) to getmail6 and I am getting a persistent error which does not allow me top receive emails:

2020-08-23 08:25:02 msg  1/47 (92343 bytes) msgid 1373023924/35200 from <SRS0=Fdaj=B6=bounce.mailing.tommy.com=bounce-25102_HTML-119607403-14328906-6407580-1763@mx.myprovider.com>
2020-08-23 08:25:03 Delivery error (mboxrd delivery 73644 error (127, mbox delivery process failed (can't concat str to bytes)))

The destination definition in the rc file looks like this:

[destination]
type = Mboxrd
path = /home/matthias/mailbox/matthias

The destination file does exists, it was used by getmail 5, and is over 100 MB in size.

I did not change any settings, just installed getmail6.

This is on Manjaro testing, with getmail6 package from AUR (version 6.03) and Python 3.8.5.

Version blurb should start mentioning others and keep year fresh

When running we see

getmail version 6.4
Copyright (C) 1998-2019 Charles Cazabon.  Licensed under the GNU GPL version 2.

Maybe this should start saying
Copyright (C) 1998-2020 Charles Cazabon and others. Licensed under the GNU GPL version 2.

Call for Debian getmail6 maintainer

I'm putting this call for a Debian getmail6 package maintainer here in this GitHub issue tracker, in the hopes that someone will see it one day and step forth an be the maintainer.