Python tool that mimics cURL, but performs a login and handles any Cross-Site Request Forgery (CSRF) tokens. Useful for scraping HTML normally only accessible when logged in.
License: GNU General Public License v3.0
Hi,
The script works great in determining all the input fields and hidden fields that need to be submitted. However when making the POST on form submission it seems that there are some warnings from the python urllib3 library about InsecureRequestWarnings. Also the script logic thinks that the login was successful, however all that happens is a 302 redirect back to the login page happens. :-(
I would also mention I think it's because we aren't submitting the cookies back to the server.
A snippet of log
DEBUG:root:Data dictionary = {'username': ['foobar'], '_csrf': ['4ff30131-8184-4572-b301-b4321f8d2541'], 'password': [foobarspassword']}
DEBUG:root:Calculating action URL ...
INFO:root:Calculated action_url as https://sslcerts.billymadison.com/sslrequest/sslrequest/login
INFO:root:Performing POST on form submission ...
DEBUG:requests.packages.urllib3.connectionpool:Resetting dropped connection: sslcertsbillymadison.com
/Users/foobar/Library/Python/2.7/lib/python/site-packages/requests/packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
DEBUG:requests.packages.urllib3.connectionpool:https://sslcerts.billymadison.com:443 "POST /sslrequest/sslrequest/login HTTP/1.1" 302 0
DEBUG:requests.packages.urllib3.connectionpool:Resetting dropped connection: sslcerts.billymadison.com
/Users/foobar/Library/Python/2.7/lib/python/site-packages/requests/packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
DEBUG:requests.packages.urllib3.connectionpool:https://sslcerts.billymadison.com:443 "GET /sslrequest/login HTTP/1.1" 200 None
INFO:root:Request result = 200
INFO:root:Result URL after login = https://sslcerts.billymadison.com/sslrequest/login
INFO:root:Login was successful
INFO:root:Making requests of interest ...
INFO:root:Performing GET on https://sslcerts.millymadison.com/sslrequest/request/list ...
DEBUG:requests.packages.urllib3.connectionpool:Resetting dropped connection: sslcerts.billymadison.com
/Users/foobar/Library/Python/2.7/lib/python/site-packages/requests/packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
DEBUG:requests.packages.urllib3.connectionpool:https://sslcerts.billymadison.com:443 "GET /sslrequest/request/list HTTP/1.1" 302 0
DEBUG:requests.packages.urllib3.connectionpool:Resetting dropped connection: sslcerts.billymadison.com
/Users/foobar/Library/Python/2.7/lib/python/site-packages/requests/packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
Hi there and thanks for a great script!
I ran into this today, which was solved by passing encoding="UTF-8" to the FileType constructor used by the metavar FILE (-o file option) and forcing the script to run in python3.
Traceback (most recent call last):
File "./curl-auth-csrf.py", line 230, in <module>
main()
File "./curl-auth-csrf.py", line 217, in main
args.output.write(result.content.decode('utf-8'))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe5' in position 182: ordinal not in range(128)```
If I echo a password to this script on windows it doesnt appear to work, it does if it ignored and the password is prompted for...
strangely, i looks as if when echoing the password the initial auth passes and then when it calls the post auth url it gets back a permission denied.
DEBUG:urllib3.connectionpool:https://monitor.example.com:443 "POST /authentication/login HTTP/1.1" 200 None
INFO:root:Request result = 200
INFO:root:Result URL after login = https://monitor.example.com/authentication/login
INFO:root:Login was successful
Looks like we passed auth...
INFO:root:Making requests of interest ...
Requests the post auth url...
INFO:root:Performing GET on https://monitor.example.com/reporting/report/show?report=Host%20SLA&hostgroup=Customer%20Servers&timeframes=one_week&download=1 ...
DEBUG:urllib3.connectionpool:https://monitor.example.com:443 "GET /reporting/report/show?report=Host%20SLA&hostgroup=Customer%20Servers&timeframes=one_week&download=1 HTTP/1.1" 302 None
DEBUG:urllib3.connectionpool:https://monitor.example.com:443 "GET /authentication/login?redirect=reporting%2Freport%2Fshow%3Freport%3DHost%2520SLA%26hostgroup%3DCustomer%2520Servers%26timeframes%3Done_week%26download%3D1 HTTP/1.1" 200 None
INFO:root:Request result = 200
Got back permission denied and redirected to the login page.
This works fine when run on linux (or when the password is prompted for on windows)
Python: Python 3.5.3
Form string:
<form id="loginform" class="form-horizontal" role="form" action="/login" method="POST">Call:
echo password | ./curl-auth-csrf.py -i http://127.0.0.1:8000/login -d username=testusr http://127.0.0.1:8000/secretAttributeError: module 'string' has no attribute 'lstrip'
After change string ➡️ str:
INFO:root:Result URL after login = http://127.0.0.1:8000///login
(Extra slashes)
Right now, it's hard-baked to WARNING level. Is there a Pythonic way to set this from the CLI? If not, consider adding an argparse option to optionally set debug mode to DEBUG (or an arbitrary level?).
See discussion in README. pass adds a trailing newline after the password text. Removing it requires shenanigans like this:
pass foobar.com | tr '\n' 'x' | sed 's/x$//' | ./curl-auth-csrf.py ...
Consider adding an argparse option to optionally remove trailing newline from passwords originating from pass.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.