Overly long transaction details crash _parse_mt940_details (?) about mt940 HOT 10 CLOSED

It's definitely out of spec for every MT940 specification file I've seen so far. But if it's out there.. let's fix it :)

Pull requests are more than welcome, assuming they don't break anything else of course :)
Simply run tox to test everything (or wait for travis to run after creating the pull request).

from mt940.

I modified the max number of lines locally and am running it against 10 months of MT940. Looks good so far, and from what I know about the format the change should not break anything. I hope I can make a pull request later today.

from mt940.

I expect to be able to review it this weekend, thanks for all the help :)

from mt940.

Ok... didn't manage that weekend but I've been able to review it now and I've merged your changes.

Thank you so much for all the help!

from mt940.

I'm getting this bug again using version 4.12.2:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-8-457c9efd8af9> in <module>
      4     pprint.pprint(data)
----> 5     p = mt940.parse(data)
      6 
      7     for transaction in p:

c:\users\edadmin\documents\risk-edzwo\.venv\lib\site-packages\mt940\parser.py in parse(src, encoding)
     64 
     65     transactions = mt940.models.Transactions()
---> 66     transactions.parse(data)
     67 
     68     return transactions

c:\users\edadmin\documents\risk-edzwo\.venv\lib\site-packages\mt940\models.py in parse(self, data)
    417 
    418             for processor in self.processors.get('post_%s' % tag.slug, []):
--> 419                 result = processor(self, tag, tag_dict, result)
    420 
    421             # Creating a new transaction for :20: and :61: tags allows the

c:\users\edadmin\documents\risk-edzwo\.venv\lib\site-packages\mt940\processors.py in transaction_details_post_processor(transactions, tag, tag_dict, result)
    199     # check for e.g. 103?00...
    200     if re.match(r'^\d{3}\?\d{2}', details):
--> 201         result.update(_parse_mt940_details(details))
    202 
    203         purpose = result.get('purpose')

c:\users\edadmin\documents\risk-edzwo\.venv\lib\site-packages\mt940\processors.py in _parse_mt940_details(detail_str)
    139             continue
    140         tmp[segment_type] = segment if not segment_type else segment[2:]
--> 141         segment_type = detail_str[index + 1] + detail_str[index + 2]
    142         segment = ''
    143 

IndexError: string index out of range

:86:105?00FOLGELASTSCHRIFT?109250?20EREF+31HA07BC81305866B2CA4A?2
1529648D0D2?22MREF+3034.1710.5760?23CRED+DE92XXX12312319957?24SVW
Z+3722.9795.1964 HAFEL-L?25INGPING GMBH CO. KG LEXING?26E RECHNU
NG FINANZEN MONATSP?27REIS GLAUB-ID DE34ZXX123345?644095 ID
A9044ED4-A8876-?ABC123-90D?30PBBBDEFFXXX?31DE9846948540984758434?
32HAFL-LINGPING GMBH CO. KG?34992?60ABWA+HAFEL-LINGPING GMBH CO.?6

the complete mt940 has several :61 and :86 chunks, but deleting this one allows mt940.parse to do its job. The :86 is only 6 lines long though, so I'm not sure if I'm looking at the same problem here as @sbi . Running mt940.parse against a large number of transactions has resulted in this error for several of them.

from mt940.

It looks like you have a similar but different issue. Instead of being out of spec with too many lines it's got too many characters (at least with the last line).

I'll have to investigate what's going on here though. In any case line 141 of processors.py does not look defensive enough in my opinion, assumptions that multiple parts are there can be wrong.

from mt940.

thanks for your answer! I got a couple of examples like the one above, let me know how I can provide help with this issue.

from mt940.

If you can provide some (anonymized) data files that would help a lot to keep it running in the future. But I think the snippet above should already help :)

from mt940.

I got a similar issue, but this time the line ends with a ? without segment type.
A 2 liner could make _parse_mt940_details() a little bit more defensive on that kind of problem- The downside would be to silently miss some data (probably).

Should I make pull request?

from mt940.

Pull requests are always welcome :)

from mt940.