hoffie / dkb-visa Goto Github PK
View Code? Open in Web Editor NEWDKB transaction download and conversion (CSV, QIF)
License: GNU Affero General Public License v3.0
DKB transaction download and conversion (CSV, QIF)
License: GNU Affero General Public License v3.0
Last week and today (19.9.2019) I tried to obtain my transactions and got the following error message and stack trace:
richi@yang:~/Coding/dkb-visa$ ./dkb.py --userid XXXX --cardid YYYY --output dkb_visa_card_umsaetze.qif
Start time: 21.06.2019
PIN:
Starting login as user XXXX...
Navigating to 'Umsätze'...
Selecting transactions in time frame 21.06.2019 - 18.09.2019...
Traceback (most recent call last):
File "./dkb.py", line 493, in <module>
fetcher.select_transactions(args.cardid, from_date, args.to_date)
File "./dkb.py", line 242, in select_transactions
br.form = form = self._get_transaction_selection_form()
File "./dkb.py", line 168, in _get_transaction_selection_form
raise RuntimeError("Unable to find transaction selection form")
RuntimeError: Unable to find transaction selection form
Since yesterday, I get this error message:
Start time: 25.08.2017
PIN:
Starting login as user xxx...
Navigating to 'Umsätze'...
Selecting transactions in time frame 25.08.2017 - 01.09.2017...
Traceback (most recent call last):
File "/usr/bin/dkb.py", line 416, in <module>
fetcher.select_transactions(args.cardid, from_date, args.to_date)
File "/usr/bin/dkb.py", line 173, in select_transactions
br.form = form = self._get_transaction_selection_form()
File "/usr/bin/dkb.py", line 99, in _get_transaction_selection_form
raise RuntimeError("Unable to find transaction selection form")
RuntimeError: Unable to find transaction selection form
Did not find the time to look into this in detail, but maybe DKB changed the HTML structure.
Running dkb.py as usual results in:
Navigating to 'Umsätze'...
Selecting transactions in time frame 03.03.2017 - 28.05.2017...
Traceback (most recent call last):
File "./dkb.py", line 427, in <module>
fetcher.select_transactions(args.cardid, from_date, args.to_date)
File "./dkb.py", line 190, in select_transactions
self._select_all_transactions_from(form, from_date, to_date)
File "./dkb.py", line 115, in _select_all_transactions_from
raise RuntimeError("Unable to find search period radio box")
RuntimeError: Unable to find search period radio box
Note that sometimes the original csv-file includes information, formatted via html.
I solved it like this:
# check that unparseable lines are not included as transactions
# dkb should really fix their csv export, they should have the
# money for that, right?
csv_text = csv_text.splitlines()
csv_text_filtered = []
for line in csv_text:
if "<br />" in line:
continue
else:
csv_text_filtered.append(line)
csv_text = "\n".join(csv_text_filtered)
This will exclude transactions that have <br />
in their description.
Everytime I run this tool it emits the following warning:
/usr/lib/python2.7/dist-packages/bs4/__init__.py:166: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.
To get rid of this warning, change this:
BeautifulSoup([your markup])
to this:
BeautifulSoup([your markup], "lxml")
markup_type=markup_type))
Does anyone else get this too?
I tested this via dkbweb.py which calls dkb.py, so this should apply to dkb.py as well.
If you specify a from_date like 1.1.1970, you won't get an error from mechanize. But it triggers another issue: The csv export will only contain the initially visible transactions because the date is not submittable. This means that the page won't update and the csv-file won't be generated with the new dates.
I "fixed" this for myself by fetching all the transactions from the last year every time and later I filter the results via sets. See: https://github.com/inktrap/dkb-export/blob/master/dkbweb.py#L215. Btw.: I also checked that BASEURL is an https-url. :)
Edit: Just to clarify: The allowed from_date is set server side and constant.
I get the following trace when starting:
Traceback (most recent call last):
File "/home/uwe/dkb-banking/dkb-visa/dkb.py", line 413, in
fetcher.login(args.userid, pin)
File "/home/uwe/dkb-banking/dkb-visa/dkb.py", line 61, in login
br.form = list(br.forms())[0]
IndexError: list index out of range
Hello,
it looks like mechanize
is not available for python3.
I'm currently trying to integrate dkb-visa into my own script, which unfortunately is written in python3. Do you have any plans to migrate to python3? How could I help with the migration?
The following StackOverflow comment mentions some mechanize alternatives for python3, which might be helpful: https://stackoverflow.com/a/31774959
Anway, thank you for the awesome script!
Hi, can you add the following topics to this repository?
Thanks!
Since some days (did not use dkb-visa for some months in before) I can not get any data entries at all, anymore. Maybe webpage form changed again.
Running dkb-visa with pdb debugger and checking the form input IMHO the request URL is fine but the response has an empty block like
…
<input name="$$$event_search" type="hidden" value="search"/>
</form>
</div>
</div>
</div>
</div>
</div>
<div class="clearfix richfooter_wrap" id="richfooter_wrap">
…
where I would actually expect some data as I can see them with the firefox inspector. Does anyone still have success on dkb webpage?
An entry like the following one has an empty COL_VALUTA_DATE
, because it does not show a
transaction, only the Soll or Haben (that you currently owe or have). So this
should not be one of the transactions.
"Nein";"01.01.2015";"";"Soll";"-00,01";"";
However filtering and matching transactions to accounts should happen later, so
I used this simple but ugly fix:
if len(line[self.COL_VALUTA_DATE].strip()) == 0:
return re.sub('.*?(\d{1,2})\.(\d{1,2})\.(\d{2,4}).*?', r'\2/\1/\3',
line[self.COL_DATE])
else:
return re.sub('.*?(\d{1,2})\.(\d{1,2})\.(\d{2,4}).*?', r'\2/\1/\3',
line[self.COL_VALUTA_DATE])
Otherwise the generated qif-file won't be importabable
It seems the option "qif-account" is never used. I tried to use it to be able to distinguish between two different accounts when merging both into a common gnucash document and saw no effect. Also, I could confirm in the source code that the value written by the ArgumentParser is never used.
Anyway, thanks for the great rewrite after "web_bank.py" does not work anymore. As a workaround I proposed a feature to web_bank.py and used it myself just doing the conversion from manually downloaded csv files to qif: okurz/web_bank@4175542
The command line download start date (--to-date) takes the value dates and not the booking dates. Not really sure what the intention is but from a use case perspective, it might not make sense:
Problem:
If a value date is two days ago or from last week or last month (because a supplier books very late or international transactions or whatever) or there is a refund (which is usually the value date of the original booking), I would have to download all transactions from the last X months.
Only solution I can think of, is do download all transactions from last year and then filter the file by booking date. Maybe this behaviour can be enabled by an additional flag.
Let me know what you think. Am happy to provide the fix.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.