Code Monkey home page Code Monkey logo

mt940's Introduction

MT940

MT940 test status MT940 Pypi version MT940 code coverage

mt940 - A library to parse MT940 files and returns smart Python collections for statistics and manipulation.

Links

Install

To install the latest release:

pip install mt-940

Or if pip is not available:

easy_install mt-940

To install the latest development release:

git clone --branch develop https://github.com/WoLpH/mt940.git mt940
cd ./mt940
virtualenv .env
source .env/bin/activate
pip install -e .

To run the tests you can use the py.test command or just run tox to test everything in all supported python versions.

Usage

Basic parsing:

import mt940
import pprint

transactions = mt940.parse('mt940_tests/jejik/abnamro.sta')

print('Transactions:')
print(transactions)
pprint.pprint(transactions.data)

print()
for transaction in transactions:
    print('Transaction: ', transaction)
    pprint.pprint(transaction.data)

Set opening / closing balance information on each transaction:

import mt940
import pprint

mt940.tags.BalanceBase.scope = mt940.models.Transaction

# The currency has to be set manually when setting the BalanceBase scope to Transaction.
transactions = mt940.models.Transactions(processors=dict(
    pre_statement=[
        mt940.processors.add_currency_pre_processor('EUR'),
    ],
))

with open('mt940_tests/jejik/abnamro.sta') as f:
    data = f.read()

transactions.parse(data)

for transaction in transactions:
    print('Transaction: ', transaction)
    pprint.pprint(transaction.data)

Simple json encoding:

import json
import mt940


transactions = mt940.parse('mt940_tests/jejik/abnamro.sta')

print(json.dumps(transactions, indent=4, cls=mt940.JSONEncoder))

Parsing statements from the Dutch bank ASN where tag 61 does not follow the Swift specifications:

def ASNB_mt940_data():
    with open('mt940_tests/ASNB/0708271685_09022020_164516.940.txt') as fh:
        return fh.read()

def test_ASNB_tags(ASNB_mt940_data):
    tag_parser = mt940.tags.StatementASNB()
    trs = mt940.models.Transactions(tags={
        tag_parser.id: tag_parser
    })

trs.parse(ASNB_mt940_data)
trs_data = pprint.pformat(trs.data, sort_dicts=False)
print(trs_data)

Contributing

Help is greatly appreciated, just please remember to clone the development branch and to run tox before creating pull requests.

Travis tests for flake8 support and test coverage so it's always good to check those before creating a pull request.

Development branch: https://github.com/WoLpH/mt940/tree/develop

To run the tests:

pip install -r mt940_tests/requirements.txt
py.test

Or to run the tests on all available Python versions:

pip install tox
tox

Info

Python support Python 2.7, >= 3.3
Blog http://wol.ph/
Source https://github.com/WoLpH/mt940
Documentation http://mt940.rtfd.org
Changelog http://mt940.readthedocs.org/en/latest/history.html
API http://mt940.readthedocs.org/en/latest/modules.html
Issues/roadmap https://github.com/WoLpH/mt940/issues
Travis http://travis-ci.org/WoLpH/mt940
Test coverage https://coveralls.io/r/WoLpH/mt940
Pypi https://pypi.python.org/pypi/mt-940
Ohloh https://www.ohloh.net/p/mt-940
License BSD.
git repo
$ git clone https://github.com/WoLpH/mt940.git
install dev
$ git clone https://github.com/WoLpH/mt940.git mt940
$ cd ./mt940
$ virtualenv .env
$ source .env/bin/activate
$ pip install -e .
tests
$ py.test

mt940's People

Contributors

apro avatar benkonrath avatar bumi avatar cluck avatar djalexd avatar dotlambda avatar dr-duplo avatar fhachenberg avatar gpaulissen avatar philippeowagner avatar raphaelm avatar rdeknijf avatar sbi-c avatar sebschrader avatar sq9mev avatar stancline avatar sweh avatar wolph avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mt940's Issues

ING mt940 export assertionError

Traceback (most recent call last):
  File "mt940parse.py", line 5, in <module>
    for transaction in transactions:
  File "mt940/mt940/parser.py", line 21, in __iter__
    for transaction in Transaction.parse(transaction_collection):
  File "mt940/mt940/parser.py", line 81, in parse
    assert len(transactions) == len(transactions_detail)
AssertionError

Problem parsing a sample file

Hi,

in my project, I test the mt940 feature (that is based on your package) against the sample files of multiple banks.

Since version 4.0, the sample file from this document (page 34) can no longer be parsed.

I tried to find the problem myself but wasn't very successful until now. Do you see it?

The file:

:20:STARTUMS
:25:1222333444
:28:1/1
:NS:22Test GmbH
23Testkonto
240,800
25010102311202
3037010000
3190000022
:60F:C020315DEM0,00
:61:0203170320CM5000,00S05168790452
:NS:01Verwendungszweck 1
02Verwendungszweck 2
15Empfänger
17Buchungstext
1812345
191000
204711
:61:020322CM20000,00NCHG
:61:020322CM20000,00S051
:61:020322CM20000,00S051
:61:020322CM20000,00S051
:61:020322CM20000,00S051
:62M:C020315105000,00
:20:STARTUMS
:25:1222333444
:28:1/1
:NS:223037010000
:60M:C020315DEM105000,00
:61:020322CM20000,00S051
:61:020322CM20000,00S051
:62F:C020315145000,00
:20:STARTUMS
:25:3346780111
:28:2/1
:NS:22Meyer + Schneider
23Testkonto
3037010000
3187132101
:60F:C020324DEM145000,00
:61:020324DM50000,00S051
:NS:01bekannt
1812345
:62F:C02032495000,00

I get the following error:

RuntimeError: ('Unable to parse "<mt940.tags.Statement object at 0x7f96544eae10>" from "0203170320CM5000,00S05168790452\n:NS:01Verwendungszweck 1\n02Verwendungszweck 2\n15Empfänger\n17Buchungstext\n1812345\n191000\n204711"', <mt940.tags.Statement object at 0x7f96544eae10>, '0203170320CM5000,00S05168790452\n:NS:01Verwendungszweck 1\n02Verwendungszweck 2\n15Empfänger\n17Buchungstext\n1812345\n191000\n204711')

With the following output:

DEBUG 2016-05-04 16:32:57,596 mt940.tags.TransactionReferenceNumber tags matched (8) "STARTUMS" against "(?P<transaction_reference>.{0,16})", got: {'transaction_reference': 'STARTUMS'}
DEBUG 2016-05-04 16:32:57,596 mt940.tags.AccountIdentification tags matched (10) "1222333444" against "(?P<account_identification>.{0,35})", got: {'account_identification': '1222333444'}
DEBUG 2016-05-04 16:32:57,596 mt940.tags.StatementNumber tags matched (76) "1/1
:NS:22Test GmbH
23Testkonto
240,800
25010102311202
3037010000
3190000022" against "
    (?P<statement_number>\d{1,5})  # 5n
    (?:/(?P<sequence_number>\d{1,5}))?  # [/5n]
    ", got: {'sequence_number': '1', 'statement_number': '1'}
DEBUG 2016-05-04 16:32:57,596 mt940.tags.FinalOpeningBalance tags matched (14) "C020315DEM0,00" against "^
    (?P<status>[DC])  # 1!a Debit/Credit
    (?P<year>\d{2})  # 6!n Value Date (YYMMDD)
    (?P<month>\d{2})
    (?P<day>\d{2})
    (?P<currency>.{3})  # 3!a Currency
    (?P<amount>[0-9,]{0,16})  # 15d Amount (includes decimal sign, so 16)
    ", got: {'year': '02', 'month': '03', 'amount': '0,00', 'currency': 'DEM', 'day': '15', 'status': 'C'}
INFO 2016-05-04 16:32:57,596 mt940.tags.Statement tags matching (127) "0203170320CM5000,00S05168790452
:NS:01Verwendungszweck 1
02Verwendungszweck 2
15Empfänger
17Buchungstext
1812345
191000
204711" against "^
    (?P<year>\d{2})  # 6!n Value Date (YYMMDD)
    (?P<month>\d{2})
    (?P<day>\d{2})
    (?P<entry_month>\d{2})?  # [4!n] Entry Date (MMDD)
    (?P<entry_day>\d{2})?
    (?P<status>[A-Z]?[DC])  # 2a Debit/Credit Mark
    (?P<funds_code>[A-Z])? # [1!a] Funds Code (3rd character of the currency
                            # code, if needed)
    (?P<amount>[\d,]{1,15})  # 15d Amount
    (?P<id>[A-Z][A-Z0-9]{3})?  # 1!a3!c Transaction Type Identification Code
    (?P<customer_reference>.{0,16})  # 16x Customer Reference
    (//(?P<bank_reference>.{0,16}))?  # [//16x] Bank Reference
    (\n?(?P<extra_details>.{0,34}))?  # [34x] Supplementary Details
                                             # (this will be on a new/separate
                                             # line)
    $"

Feature: Multiple transaction description

Now the library reads the last line of :86: description multi lines as data['transaction_details'].

Would it be possible to parse all the :86: lines from a transaction and put them in an array or some sort in the parameter transaction_details?

That would be really helpful.

incorrect linebreaks sent by bank

When fetching transactions with the python-fints library the mt940 parser crashes because there are some non-valid line breaks in the MT940 sent by our bank (Ostsächsiche Sparkasse Dresden).

We found the following two types of superfluous line breaks:

  1. Line break in a line starting with the :61: tag:
:61:1811261126CR
30,00N062NONREF
  1. Line break in the :86: tag:
:
86:166?00GUTSCHR. UEBERWEISUNG?081542?20SVWZ+Test

I know that this isn't an issue of this library. But since contacting our bank resulted in a reference to their banking software and the information that third-party products are not supported, it might be the best solution to accept those line breaks. Maybe also other Sparkassen are affected.

Parse GV codes when purpose does not start with GV codes

Some banks use GV codes in the purpose field, but do not start the data with them:

'purpose': 'Cie., S.C.A.SEPA-BASISLASTSCHRIFTEREF+ 1234567890123 PAYPALMREF+ 12AB234CD6E7F CRED+ AB12CDE0000000000000000034 SVWZ+ . SPOTIFY, Ihr Einkaufbei SPOTIFY'

In this case the GV codes are not parsed at the moment, because purpose is expected to start with a GV code:

if purpose and purpose[:4] in GVC_KEYS:  # pragma: no branch
    result.update(_parse_mt940_gvcodes(result['purpose']))

In my personal use parsing is fine when doing it unconditionally:

if purpose:  # pragma: no branch
    result.update(_parse_mt940_gvcodes(result['purpose']))

This breaks a number of tests that I have not looked into, yet.

What would be a preferred solution for this situation?

  • Parse GV codes unconditionally
  • Parse when GV code is found anywhere in purpose
  • Make behavior configurable/separate processor
  • ???

I would be willing to prepare a PR when agreed on a way forward.

Extra Details?

I cannot see how to get the extra details from any mt940 file. I understood extra_details to be the optional text line at the end of the :61 block. I could be missing something obvious, but can you please tell how to get the text "FT148090949341 FOREIGN U" from the example mt940 block below?

:61:1404010401D1,NMSCNONREF
FT148090949341 FOREIGN U
:86:SD

What I get back is:
{'amount': <-1 EUR>,
'bank_reference': None,
'currency': 'EUR',
'customer_reference': 'NONREF',
'date': Date(2014, 4, 1),
'entry_date': Date(2014, 4, 1),
'extra_details': '',
'funds_code': None,
'id': 'NMSC',
'status': 'D',
'transaction_details': 'SD'}

Overly long transaction details crash _parse_mt940_details (?)

I'm investigating a crash while parsing a VR Bank MT940 file. Here's the stack trace:

~/my/stuff.py in check_payments(cls, mt940_file)
    568 			and returns a list of result records for each transaction in the mt940 file """
    569                 mt940_data = mt940_file.read().decode('latin1')
--> 570                 transactions = mt940.parse(mt940_data).transactions
    571                 results = []
    572                 for transaction in transactions:

~/somewhere/virtualenv-3.5.4/lib/python3.5/site-packages/mt940/parser.py in parse(src)
     50 
     51     transactions = mt940.models.Transactions()
---> 52     transactions.parse(data)
     53 
     54     return transactions

~/somewhere/virtualenv-3.5.4/lib/python3.5/site-packages/mt940/models.py in parse(self, data)
    369 
    370             for processor in self.processors.get('post_%s' % tag.slug):
--> 371                 result = processor(self, tag, tag_dict, result)
    372 
    373             # Creating a new transaction for :20: and :61: tags allows the

~/somewhere/virtualenv-3.5.4/lib/python3.5/site-packages/mt940/processors.py in transaction_details_post_processor(transactions, tag, tag_dict, result)
    198 
    199     if gvc.isdigit() and int(gvc) in gvcs and details[3:6] == '?00':
--> 200         result.update(_parse_mt940_details(details))
    201 
    202         purpose = result.get('purpose')

~/somewhere/virtualenv-3.5.4/lib/python3.5/site-packages/mt940/processors.py in _parse_mt940_details(detail_str)
    139             continue
    140         tmp[segment_type] = segment if not segment_type else segment[2:]
--> 141         segment_type = detail_str[index + 1] + detail_str[index + 2]
    142         segment = ''
    143 

IndexError: string index out of range

The immediate problem is that detail_str ends in a question mark. After digging further, I found that detail_str contains a truncated version of the data I see in the original MT940 file. I looked at the regex in TransactionDetails and I believe it says that a transaction detail field can only have up to six lines, with a max line length of 65. However, the relevant part of the MT940 file looks like this (I changed a few digits in an attempt to hide any private account numbers):

:86:105?00Basislastschrift?10931?20EREF+LS-FP2-840-1000-282163
?210623 ?22MREF+POSTAG0053000001669688?23CRED+DE65ZZZ00000210259 
?24PURP+OTHR andere ?25SVWZ+KD9191919195 919191912
?263 REFSAP3147898009/EFI RE74?2763339623  DAT04.10.2017 VIE
?28LEN DANK, IHRE EFILIALE 314?919191919 EREF: LS-FP2-840-10
?30HYVEDEMM?31DE84700202700015179032?32Deutsche Post AG?34992
?6000-2821919123 MREF: POSTAG0?61053000001669688 CRED: DE65Z
?62ZZ00000210259 IBAN: DE84700?63202700015179032 BIC: HYVEDE

Is that "out of spec", and can I send a PR to allow for more lines?

Differentiating between 62M and 62F

Hi there,

Sorry, I got another one:

I can see in the docs, that the field 62 is the "closing_balance". However that's not always true, as that would only be the case for 62F, while often there may be intra-day balances with field 62M.

Does this field not see a difference between the two or is there some magic going on that already works with 62F only?

Thanks very much.

Error using mt940 in fints: module 'mt940' has no attribute 'tags'

I use python-fints to retrieve transactions from a bank account. The latest mt940 version (v4.17.0) is installed.
Executing this line (from fints/utils.py, l. 18) transactions = mt940.models.Transactions() results in the following error:

WARNING:fints.client:Dialog response: 3060 - Bitte beachten Sie die enthaltenen Warnungen/Hinweise.
INFO:fints.client:Dialog response: 0020 - Auftrag ausgeführt.
WARNING:fints.client:Dialog response: 3040 - Weitere Informationen liegen vor. (['1f*4i*j7r45*j85io*1353.42'])
INFO:fints.client:Fetching more results (7)...
WARNING:fints.client:Dialog response: 3060 - Bitte beachten Sie die enthaltenen Warnungen/Hinweise.
INFO:fints.client:Dialog response: 0020 - Auftrag ausgeführt.
WARNING:fints.client:Dialog response: 3040 - Weitere Informationen liegen vor. (['1*1e*j7r45*j85io*990.26'])
INFO:fints.client:Fetching more results (8)...
INFO:fints.client:Dialog response: 0010 - Nachricht entgegengenommen.
INFO:fints.client:Dialog response: 0020 - Auftrag ausgeführt.
INFO:fints.client:Fetching done.
Traceback (most recent call last):
  File "/Users/timo/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-C/ch-0/182.4505.26/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 1664, in <module>
    main()
  File "/Users/timo/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-C/ch-0/182.4505.26/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 1658, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/Users/timo/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-C/ch-0/182.4505.26/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 1068, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Users/timo/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-C/ch-0/182.4505.26/PyCharm CE.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/timo/workspace/Spent/_fiddles/Fints.py", line 140, in <module>
    download_transactions(from_date=datetime.date(2018, 1, 1))
  File "/Users/timo/workspace/Spent/_fiddles/Fints.py", line 40, in download_transactions
    transactions = f.get_transactions(account, from_date, balance.date)
  File "/Users/timo/anaconda3/envs/spent/lib/python3.7/site-packages/fints/client.py", line 521, in get_transactions
    statement = mt940_to_array(''.join([seg.statement_booked.decode('iso-8859-1') for seg in responses]))
  File "/Users/timo/anaconda3/envs/spent/lib/python3.7/site-packages/fints/utils.py", line 18, in mt940_to_array
    transactions = mt940.models.Transactions()
  File "/Users/timo/anaconda3/envs/spent/lib/python3.7/site-packages/mt940/models.py", line 277, in __init__
    self.tags = Transactions.defaultTags().copy()
  File "/Users/timo/anaconda3/envs/spent/lib/python3.7/site-packages/mt940/models.py", line 309, in defaultTags
    return mt940.tags.TAG_BY_ID
AttributeError: module 'mt940' has no attribute 'tags'

ValueError: stat: path too long for Windows

An error pop-up once trying to pass string like mt940.parse(f.read())
In line 44 is a check:
elif os.path.isfile(src):
which crach once provide str under Windows.

Solution is to move it after: elif hasattr(src, 'decode'):

fixed code looks like:

if hasattr(src, 'read'):  # pragma: no branch
        data = src.read()
    elif hasattr(src, 'decode'):
        exception = None
        encodings = [encoding, 'utf-8', 'cp852', 'iso8859-15', 'latin1']

        for encoding in encodings:  # pragma: no branch
            if not encoding:
                continue

            try:
                data = src.decode(encoding)
                break
            except UnicodeDecodeError as e:
                exception = e
        else:  # pragma: no cover
            raise exception
    elif os.path.isfile(src):
        return parse(open(src, 'rb').read())
    else:  # pragma: no cover
        data = src

reading transaction closing balance (record :62F:)

Is it possible to access the transaction closing balance (record :62F:)? I don't see this in transaction.data? I've read through the docs but I'm still not sure how to get this piece of information. I did find mt940.models.Balance for parsing this information but I'm not sure how to use this to get the information in record :62F:. Thanks for your help.

Wrong reprtitions count in TransactionDetails tag

class TransactionDetails(Tag):                                                  
                                                                                
    '''Transaction details                                                      
                                                                                
    Pattern: 6x65x                                                              
    '''                                                                         
    id = 86                                                                     
    scope = models.Transaction                                                  
    pattern = r'(?P<transaction_details>[\s\S]{0,330})' 

https://github.com/WoLpH/mt940/blob/develop/mt940/tags.py#L310

6x65 != 330; 6x65 is 390.

Is there any reasonable cause for using 330 in that regex?

transaction_details seems are truncated, unfortunately for my mBank stmts, tnr (bank unique id) is at the very end of 86 lines… 86 line are not unicode in my case.

Transaction details parsing error if content is unfortunately wrapped

Hi guys,

I discovered a problem with the parsing method of the transaction model.
Actually this error prevents me fetching transactions with fints.
In the example below (obfuscated) the transaction details ":86:" contain an ISO timetamp which is
unfortunately wrapped to the next line. The ":26:" is no tag, but the minutes of the ISO timestamp.

:61:000000D8,47NMSCNONREF
:86:106?000000/661?20EREF+VZ0000000000000000?21MREF+000000?22CRED+XX0
000000000000000?23ABCDEFGHIJKLMNOPQRSTUVW?24/PL 12-09-2014T16
:26:37 Fo?25lgenr. 007

The parser raises an exception since it doesn't know the tag ":26:" (it doesn't exist either).
The root cause (I guess) is the method used to extract the tags (models.py:333).
Basically it searches for lines starting with ':XX:' which includes also not existing 'tags'.

It better way may be to search for known tags by listing them in the regex like:
r'^:(?P<full_tag>61|86|...|...):'

This still would lead to possible parsing errors for tags below 60, so maybe another parsing strategy is more appropriate.

Parsing error: "type 'exceptions.AttributeError'"

This is an error introduced by the fix in 08f52ab.

As mentioned in #12, I upgraded the package, but now I'm getting the following error for every file it tries to parse:

<type 'exceptions.AttributeError'>

Not sure where it occurs or why.
Let me know what I can get you to help pinpoint this regression.

Cheers.

Python 2.7 support?

Hi there,

under Python2.7 I get the following error:

$ virtualenv --python=python2.7 test
$ bin/pip install mt-940
Successfully installed mt-940-4.10.0 pyyaml-3.12
$ bin/pip install enum
Successfully installed enum-0.4.6
$ bin/python
>>> import mt940
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/sweh/dev/test/lib/python2.7/site-packages/mt940/__init__.py", line 3, in <module>
    from . import tags
  File "/Users/sweh/dev/test/lib/python2.7/site-packages/mt940/tags.py", line 369, in <module>
    @enum.unique
AttributeError: 'module' object has no attribute 'unique'

mt-940 does not declare enum as a dependency for Python2.7, so one has to install it by hand (which is not a big deal though).

If you hint me to a solution I could hand over a pull request.

Kind regards,
Sebastian

Test mt940.models.FixedOffset fails on Python 3.7

______________________ [doctest] mt940.models.FixedOffset ______________________
017 Fixed time offset based on the Python docs
018     Source: https://docs.python.org/2/library/datetime.html#tzinfo-objects
019
020     >>> offset = FixedOffset(60)
021     >>> offset.utcoffset(None)
Expected:
    datetime.timedelta(0, 3600)
Got:
    datetime.timedelta(seconds=3600)

/build/source/mt940/models.py:21: DocTestFailure

post processor too eager to strip whitespace

Not sure how to provide the raw test data but I think the processor to parse transaction details is too eager stripping whitespace:

 'transaction_details': '820?00UMBUCHUNG?109392?20ERSTATTUNG '
                        'AUSLANDSEINSATZE?21NTGELT\n'
                        ' FUER VISA 4998 XXXX?22XXXX '
                        '1234?3012030000?319003932309?32DEUTSC\n'
                        'HE KREDITBANK AG'

After processing this with mt940 + python-fints (latest git) I see this:

 'purpose': 'ERSTATTUNG AUSLANDSEINSATZENTGELTFUER VISA 4998 XXXXXXXX 1234',

As you can see in the original data there is a space before "FUER" which should be preserved.

Of course I'm ready to submit additional data and probably I could even come up with some tests but I need some pointers to get started...

MT940 structured

This is for mt940 structured file right?

I get a name key error in de list ?

Some have the regex name and some not. Do you know a solution?

It is really hard to find a good mt940/mt940 structured parser in Python that works..

Better Documentation

Thanks again for the release of a mt940 parser for Python.

Could you please give a code snippet of how to use this library?
How do I get the details of a Transaction? (booking date, id, amount, reference etc...)
Thanks!

AssertionError: Unknown tag 90in line: ':90D:'

Hi there,

I get the above error when running through some of my messages, which of cause is an exception failing the parse method.

I don't need this field, I just need 25 and 62, so ideally, I'd just like to be able to ignore those unknown fields.

What can I do?

Many thanks.

Loads of errors, am I using the wrong fixtures?

I am testing with the testfiles from here: https://github.com/sandermarechal/jejik-mt940/tree/master/tests/Jejik/Tests/MT940/Fixture/document, and I get a lot of errors. I don’t know if those test files are deprecated?
For instance: https://github.com/sandermarechal/jejik-mt940/blob/master/tests/Jejik/Tests/MT940/Fixture/document/abnamro.txt
parser.py", line 150, in handle_transaction_details
description, date = details[3].split('TRANSACTIEDATUM*')
ValueError: need more than 1 value to unpack

and https://github.com/sandermarechal/jejik-mt940/blob/master/tests/Jejik/Tests/MT940/Fixture/document/ing.txt
parser.py", line 100, in handle_transaction_data
''', data, re.VERBOSE).groupdict()
AttributeError: 'NoneType' object has no attribute 'groupdict'

Python 3.4.0

Falsely recognizes tags in a time value (field 86)

Hi,

I just found that if tag 86 contains a time formatted like hh:mm:ss the :mm: portion will be recognized as a tag and if incidentally that minute portion is not an actual tag, it will freak out:

Here's an excerpt of the message in question. Note the time portion in the first 86 tag:

:61:1507160716CD7500,00FMSC1971033191//IA004587508020
NONEREF
:86:eDeposit in Branch/Store 07/16/15 10:33:22 AM 1800 S NEVADA AVE C
OLORADO SPRINGS CO 5133
:61:1507160716DD8000,00FCHK4201//IA008510335677
NONEREF
:86:CHECK
:62F:C150716USD706403,50
:64:C150716USD706403,50
:65:C150717USD706403,50

Results in:
AssertionError: Unknown tag 33in line: ':33:'

error on import in IDLE3

I tried to test mt940 in IDLE and I got error:

import mt940
Traceback (most recent call last):
File "<pyshell#2>", line 1, in
import mt940
File "C:\Program Files\Python36\lib\site-packages\mt_940-4.11.0-py3.6.egg\mt940_init_.py", line 3, in
from . import tags
File "C:\Program Files\Python36\lib\site-packages\mt_940-4.11.0-py3.6.egg\mt940\tags.py", line 45, in
from . import models
File "C:\Program Files\Python36\lib\site-packages\mt_940-4.11.0-py3.6.egg\mt940\models.py", line 8, in
from . import _compat
File "C:\Program Files\Python36\lib\site-packages\mt_940-4.11.0-py3.6.egg\mt940_compat.py", line 114, in
console_encoding = sys.stdout.encoding
AttributeError: 'NoneType' object has no attribute 'encoding'

import in script works fine.
IDLE version:
Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit (AMD64)] on win32

`transaction_reference` only present in the first transaction.

The transaction_reference is only in the first transaction - and empty in all others.

Having the following swift file:

:20:2016101714551876
.....
:61:1610171017C123,FMSC01008//NONREF
20161014001442003300234000000XXX
:86:1017?61GIRO AUS KONTO XX-XXXXX-X NAME-XXXXX MARTIN UND XXXXXX M
USTERSTRASSE 123 4000 BASEL
:61:1610171017C123,FMSC01008//NONREF
20161014001442003300234000000XXX
:86:1017?61GIRO AUS KONTO XX-XXXXX-X NAME-XXXXX MARTIN UND XXXXXX M
USTERSTRASSE 123 4000 BASEL
:61:1610171017C123,FMSC01008//NONREF
20161014001442003300234000000XXX
:86:1017?61GIRO AUS KONTO XX-XXXXX-X NAME-XXXXX MARTIN UND XXXXXX M
USTERSTRASSE 123 4000 BASEL

Will result in...

>>> t[0].data.get('transaction_reference')
'2016101714551876'
>>> t[1].data.get('transaction_reference')
>>> t[2].data.get('transaction_reference')

Shouldn't the transaction_reference be part of each transaction record?

What about to pass `data`directly?

What about to pass data directly to parse(...)? Eg. testing for a no file path string or to add a data argument with default value =None.
We load the data from a database model and parse data on demand, so we do not use or have files. We could provide the PR if you like to merge it.

Thanks for sharing the library and maintaining it! 👍

Newline not recognized

Hi!

On trying to import from a german Bank called Sparkasse I have the problem that it looks like a new-line character is not recognized. This is the error:

  File "/usr/local/lib/python3.5/dist-packages/mt940/tags.py", line 84, in parse
    self, value)
RuntimeError: ('Unable to parse "<mt940.tags.StatementNumber object at 0x7fa570515a58>" from "00000/002\n:60M"', <mt940.tags.StatementNumber object at 0x7fa570515a58>, '00000/002\n:60M')

Have you had this in the past? Is there a way to solve it, or is this a problem of the bank?

Update:
It is a file with many transactions and it looks like, the bank is splitting it in various StatementNumbers:

:20:STARTUMSE
:25:44550045/0000177352
:28C:00000/001
.......
-
:20:STARTUMSE
:25:44550045/0000177352
:28C:00000/002
........

Are NS tags actually parsed as intended?

It seems to me that the NS tags are not parsed as intended. What puzzles me is that there is a unit test actually testing for this unexpected behaviour, so it would be great to get some comments about this.

The test file tests/self-provided/raphaelm.sta basically has this structure:

  • Transactions
    • :NS:22Test GmbH (and more lines)
    • Transaction 1
      • :NS:01Verwendungszweck 1 (and more lines)
    • Transaction 2-7
      • Some transactions have :NS: tags
    • Transaction 8
      • :NS:01bekannt, 1812345

mt940.parse() actually produces this:

  • Transactions
    • :NS:01bekannt, 1812345
    • Transaction 1-8
      • No NS records

So there is a clear mismatch of the scopes.

Python 3 support

Running into problems with Python3 (AttributeError: 'dict' object has no attribute 'iteritems')

p3512@Kaboom:~/Desktop$ which python3
/usr/bin/python3
p3512@Kaboom:~/Desktop$ mkvirtualenv mt-940 --python=/usr/bin/python3
Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in mt-940/bin/python3
Also creating executable in mt-940/bin/python
Installing setuptools, pip...done.
(mt-940)p3512@Kaboom:~/Desktop$ pip install mt-940
Downloading/unpacking mt-940
  Downloading mt-940-2.2.tar.gz
  Running setup.py (path:/home/p3512/.virtualenvs/mt-940/build/mt-940/setup.py) egg_info for package mt-940

Downloading/unpacking enum34 (from mt-940)
  Downloading enum34-1.0.4.tar.gz
  Running setup.py (path:/home/p3512/.virtualenvs/mt-940/build/enum34/setup.py) egg_info for package enum34

Installing collected packages: mt-940, enum34
  Running setup.py install for mt-940

  Running setup.py install for enum34

Successfully installed mt-940 enum34
Cleaning up...
(mt-940)p3512@Kaboom:~/Desktop$ python
Python 3.4.0 (default, Jun 19 2015, 14:20:21) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import mt940
>>> import pprint
>>> transactions = mt940.parse('abnamro.sta')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/p3512/.virtualenvs/mt-940/lib/python3.4/site-packages/mt940/parser.py", line 45, in parse
    transactions.parse(data)
  File "/home/p3512/.virtualenvs/mt-940/lib/python3.4/site-packages/mt940/models.py", line 209, in parse
    for k, v in result.iteritems():
AttributeError: 'dict' object has no attribute 'iteritems'
>>> 

Multiple statements in a single MT940

Potentially not an issue: when exporting mt940 from several business accounts it will deliver blocks separated by an empty line with just '-'. Using mt940.parse(file) on these exported files will correctly yield a list of all transactions, but only the opening and closing balance of the last statement.

I am not sure if this was intentional, but ideally a list of MT940 statements, each containing opening/closing balance and transactions is produced (e.g. similar to in the CAMT.053 format).

Is this desirable?

Missing DETAIL_KEYS for "additional_purpose" in transaction details

In mt940.processors._parse_mt940_details, values for detail keys 20 thru 29 are concatenated since they together constitute the purpose field. The linked spec says keys 60 thru 63 should likewise be treated as the additional_purpose field, however only key 60 is considered. Is that on purpose (no pun intended ;-) , or may I fix it?

Improve handling of (strange) encoded statements

I need to process "strange" encoded mt940 statements.

Background: Data/statement if from Hungarian RZB, statement is generated by system running in RZB Austria and exported in Slovakia (on Windows 7). chardet suggest to use Turkish encoding for data ;-)

Basic attempt to use transactions = mt940.parse('raiffeisen-cmi.sta.txt') failed:

  File "C:\Program Files\Python36\lib\site-packages\mt_940-4.11.0-py3.6.egg\mt940\parser.py", line 52, in parse
    transactions.parse(data)
  File "C:\Program Files\Python36\lib\site-packages\mt_940-4.11.0-py3.6.egg\mt940\models.py", line 400, in parse
    tag_dict = tag.parse(self, tag_data)
  File "C:\Program Files\Python36\lib\site-packages\mt_940-4.11.0-py3.6.egg\mt940\tags.py", line 71, in parse
    self, value)
RuntimeError: ('Unable to parse "<mt940.tags.Statement object at 0x0000015B5FA95080>" from "180417DF14790,00N031\nElektronikus bankon bel\x81li \xa0tutal\xa0s"', <mt940.tags.Statement object at 0x0000015B5FA95080>, '180417DF14790,00N031\nElektronikus bankon bel\x81li \xa0tutal\xa0s')

At the moment I use this workaround (it "damages encoding" but I am able to process statement) by function:

def sta_data(filename):
    rawdata = open(filename, 'rb').read()
    result = chardet.detect(rawdata)
    charenc = result['encoding']
    return rawdata.decode(charenc, 'ignore')

I think it could make sense to solve it at mt940.tags.Statement.
Also it would make sense to implement option "encoding" for file reading or at least to document information about expected file data encoding.

Unfortunately I am not able to attach file to this issue for testing, but I can send it if needed.

Fault tollerancy of a parser.

Hi, I was testing the library for some of the turkish bank.
The problem with them is that that they a not really well formed (according to the standard).
Many banks, in general, don't really follow well the standard.
I was wondering if I can work on making a PR which makes the parser more fault-tolerant?
So the parser will give his best effort to parse the entry, but if it fails it does not return an error just the fields which it was able to parse.
One of the examples of best effort when one of the "mandatory" fields Is missing like field 28C ("Statement Number/Sequence Number").

:61: Statement format from Sberbank

Hi,
I have STA files from Sberbank which mt940 can not process.
It contains statements like this:

:61:1710111011DF2402,00S   X

It seems Sberbank does not fill Transaction Type Identification Code.
I have updated its regexp to allow spaces, will send merge request soon.

Determining the type of a (parsed) transaction

I might be missing something obvious here but once a models.Transaction object is created, how do I know what type it is? I would have guessed there would some sort of reference to a tags.* class/object/singleton in the Transaction.data dict but I'm not seeing it. Maybe I'm simply misunderstanding the architecture here ...

My use case is that I have full dump of an account but I only care about certain entries (specifically Statements). I understand I could check for the presence of certain keys (e.g. guessed_entry_date) but that seems backwards, error prone, and fragile to me.

Faulty? mt942 from mBank

I'ma trying to parse mt942 provided by polish mBank, which throws:

  File "/home/users/bociek/lmsmap/eggs/mt_940-4.3-py2.7.egg/mt940/tags.py", line 70, in parse
    self, value)
RuntimeError: ('Unable to parse "<mt940.tags.FloorLimitIndicator object at 0x7fab98295050>" from "PLN0"', <mt940.tags.FloorLimitIndicator object at 0x7fab98295050>, 'PLN0')

That is because of line:

:34F:PLN0

Which does not match https://github.com/WoLpH/mt940/blob/develop/mt940/tags.py#L162 regex - the line is lacking credit/debit mark.

I'm not sure if credit/debit mark is obligatory here, however there's one thing i know almost for sure - mBank will not fix it ;).

i can imagine two resolutions:

  1. make regrex [DC] optional, if it's optional indeed
  2. add support for another preparse_ processors, which could be run somewhere before https://github.com/WoLpH/mt940/blob/develop/mt940/models.py#L341

I need advice what is the better solutoin, anybody?

how to display mt940 on a text file with spaces between them

hi wolf, used to extract data from mt940 and it works perfectly but i having challenges when it comes to displaying the output to text file like wordpad or notepad. the problem is that when it prints unto the text file created, it comes without space between them-thus no indentation or new line . any help will be much appreciated.

Don't install `tests` as a global python package

The current setup.py configuration results in tests being shipped as a python package (the find_packages call only excludes docs/).

This implies that everyone installing mt940, or any package which depends on mt940 (like fints), is now unable to use import tests to access their own tests/ package without manual importlib intervention. In particular, this renders usage of nosetest impossible.

Encoding issue

Seems the current parsing for this library changed slightly.

Instead of mt940.parse I need to use a class:

obj = mt940.MT940("Downloads/20200110-1631282804-umsMT940.TXT")

This fails with missing positional parameter:

TypeError: __new__() missing 1 required positional argument: 'end_balance'

Question is what is the format of the end_balance parameter?

I tried with datetime and datetime.isoformat() but both failed.

Simple json encoding example does not export all data properly

I am trying to parse my bank's MT940 data with your library and example of JSON encoding is producing different results than "plain" data dump. This is what I see with plain dump:

{'amount': <-21.99 PLN>,
 'bank_reference': None,
 'currency': 'PLN',
 'customer_reference': '00000013586',
 'date': Date(2019, 3, 3),
 'entry_date': Date(2019, 3, 3),
 'extra_details': '',
 'funds_code': None,
 'guessed_entry_date': Date(2019, 3, 3),
 'id': 'S114',
 'status': 'D',
 'transaction_details': '114\n'
                        '114~00B114ZAKUP PRZY UŻYCIU KARTY\n'
                        '~20SALAD STORY SZCZECI/SZCZECI\n'
                        '~21N\n'
                        '~22                DATA TRANSA\n'
                        '~23KCJI: 2019-03-01\n'
                        '~24\n'
                        '~25\n'
                        '~26\n'
                        '~27\n'
                        '~28\n'
                        '~29\n'
                        '~30\n'
                        '~31\n'
                        '~32\n'
                        '~33\n'
                        '~34114\n'
                        '~38PL\n'
                        '~62\n'
                        '~63'}

and this is what JSON dump example outputs:

{
    "status": "D",
    "funds_code": null,
    "amount": null,
    "id": "S114",
    "customer_reference": "00000013586",
    "bank_reference": null,
    "extra_details": "",
    "currency": "PLN",
    "date": null,
    "entry_date": null,
    "guessed_entry_date": null,
    "transaction_details": "114\n114~00B114ZAKUP PRZY U\u017bYCIU KARTY\n~20SALAD STORY SZCZECI/SZCZECI\n~21N\n~22                DATA TRANSA\n~23KCJI: 2019-03-01\n~24\n~25\n~26\n~27\n~28\n~29\n~30\n~31\n~32\n~33\n~34114\n~38PL\n~62\n~63"
}

And relevant part of the source file:

:61:1903030303D21,99S11400000013586
:86:114
:86:114~00B114ZAKUP PRZY UŻYCIU KARTY
~20SALAD STORY SZCZECI/SZCZECI
~21N    
~22                DATA TRANSA
~23KCJI: 2019-03-01
~24
~25
~26
~27
~28
~29
~30
~31
~32
~33
~34114
~38PL
~62
~63

incorrect entry date for :61: statement line

Thanks for this library - it's really useful!

I found a problem with the entry date from in a mutation line from an ABN AMRO (NL) MT940 file. This line seems to be parsed incorrectly:

:61:1412310105C**,**N570NONREF

The entry date should be 5 Jan 2015, not 5 Jan 2014 as shown here:

'customer_reference': 'NONREF',
 'date': Date(2014, 12, 31),
 'entry_date': Date(2014, 1, 5),

Since the entry date doesn't have a year, I wonder if it should always be assumed to be in the future. Any thoughts on this? Thanks.

DateTimeIndication parse

Hello I've created a python script which uses the FinTs package.
Last week it worked perfectly. Now today the script crashes with the following output:

INFO:mt940.tags.DateTimeIndication:matching`` (10) "1703271108" against "^
    (?P<year>\d{2})
    (?P<month>\d{2})
    (?P<day>\d{2})
    (?P<hour>\d{2})
    (?P<minute>\d{2})\+(?P<offset>\d{4})
    "
Traceback (most recent call last):
  File "test.py", line 17, in <module>
    statement = fintsClient.get_statement(accounts[0], date(2017, 1, 1), date.to
day())
  File "C:\Python\lib\site-packages\fints\client.py", line 84, in get_statement
    statement += mt940_to_array(m.group(2))
  File "C:\Python\lib\site-packages\fints\utils.py", line 7, in mt940_to_array
    return transactions.parse(data)
  File "C:\Python\lib\site-packages\mt940\models.py", line 357, in parse
    tag_dict = tag.parse(self, tag_data)
  File "C:\Python\lib\site-packages\mt940\tags.py", line 70, in parse
    self, value)
RuntimeError: ('Unable to parse "<mt940.tags.DateTimeIndication object at 0x0332
75F0>" from "1703271108"', <mt940.tags.DateTimeIndication object at 0x033275F0>,
 '1703271108')

Is this a problem of my bank? Could this be related to the daylight saving change on sunday?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.