Code Monkey home page Code Monkey logo

Comments (14)

wkeeling avatar wkeeling commented on May 20, 2024

@appunni-dishq Do you have logging turned up to DEBUG level - for example:

import logging
logging.basicConfig(level=logging.DEBUG)

from seleniumwire import webdriver

...

That may give some clues as to why Selenium Wire is not getting through the upstream proxy.

Are you able to share the configuration you're passing to the webdriver constructor?

from selenium-wire.

appunni-dishq avatar appunni-dishq commented on May 20, 2024 

                        settings.CHROME_WEBDRIVER,
                        desired_capabilities=options.to_capabilities(),
                        seleniumwire_options={
                            'proxy': {
                                'http': settings.ENV_HTTP_PROXY,
                                'https': settings.ENV_HTTPS_PROXY,
                                'no_proxy': 'localhost,127.0.0.1'
                            }
                        }
                    )```
And both proxies are same http proxy

from selenium-wire.

appunni-dishq avatar appunni-dishq commented on May 20, 2024

@wkeeling I did turn up DEBUG logging but didn't give much information except the status code and stuff. And Selenium-wire throws error when http or https are None, which should not be the case as many implementations including in that of request module is to override proxy to empty if passed None. Also when http proxy is used in https I get 301 always.

from selenium-wire.

appunni-dishq avatar appunni-dishq commented on May 20, 2024

format of Proxy is this http://x.x.x.x:port
, earlier I was using ATS but as it didn't support SSL Termination, I switched Squid.
curl -x http://x.x.x.x:port/ -iLk https://ipchicken.com works fine, without option k. It throws insecure request error in curl. Now I added a flag in chrome options "--ignore-certificate-errors" but it didn't make much difference.

from selenium-wire.

appunni-dishq avatar appunni-dishq commented on May 20, 2024

@wkeeling I think that issue was because of configuration issue. As Selenium-wire send insecure proxy request, while I was expecting insecure request.

from selenium-wire.

wkeeling avatar wkeeling commented on May 20, 2024

@appunni-dishq - OK thanks.

So just to be clear, you're using a proxy configuration such as this:

seleniumwire_options={
    'proxy': {
        'http': http://x.x.x.x:port,
        'https': http://x.x.x.x:port,
        'no_proxy': 'localhost,127.0.0.1'
     }
}

So the https option is using an http proxy URL?

Regarding the use of None for http and https, you are right, Selenium Wire should handle this and not error. I will raise a separate issue for that.

from selenium-wire.

appunni-dishq avatar appunni-dishq commented on May 20, 2024

I did different implementations http proxy as well as https proxy. When http proxy is used I usually get 301 status code on all request. When I use https I am getting 400. Using curl proxy-insecure option was also giving me similar responses

from selenium-wire.

wkeeling avatar wkeeling commented on May 20, 2024

OK thanks. I'll see if I can reproduce the issue with a Squid server.

from selenium-wire.

appunni-dishq avatar appunni-dishq commented on May 20, 2024

I am using squid 4.4. This is the configuration . :

#
# Recommended minimum configuration:
#

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 172.12.0.0/16		# RFC 1918 local private network (LAN)
acl localnet src fc00::/7       	# RFC 4193 local private network range
acl localnet src fe80::/10      	# RFC 4291 link-local (directly plugged) machines
acl localhost src 127.0.0.1
acl localhost src localhost

acl SSL_ports port 443
acl Safe_ports port 80		# http
acl Safe_ports port 21		# ftp
acl Safe_ports port 443		# https
acl Safe_ports port 70		# gopher
acl Safe_ports port 210		# wais
acl Safe_ports port 1025-65535	# unregistered ports
acl Safe_ports port 280		# http-mgmt
acl Safe_ports port 488		# gss-http
acl Safe_ports port 591		# filemaker
acl Safe_ports port 777		# multiling http
acl CONNECT method CONNECT

#
# Recommended minimum Access Permission configuration:
#
# Deny requests to certain unsafe ports
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
# http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost

# And finally deny all other access to this proxy
http_access deny all

sslcrtd_program /usr/lib64/squid/security_file_certgen -s /var/lib/ssl_db -M 4MB


http_port 3129 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB  cert=/etc/squid/ssl_cert/rootCA.pem key=/etc/squid/ssl_cert/rootCA.key

https_port 3128 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB  cert=/etc/squid/ssl_cert/rootCA.pem key=/etc/squid/ssl_cert/rootCA.key options=SINGLE_DH_USE,SINGLE_ECDH_USE tls-dh=/etc/squid/ssl_cert/dhparam.pem

acl DiscoverSNIHost at_step SslBump1
# acl NoSSLIntercept ssl::server_name_regex -i "/etc/squid/server-regex.nobump"
# ssl_bump splice NoSSLIntercept
ssl_bump peek DiscoverSNIHost
#ssl_bump peek step1
ssl_bump bump all
# https_port 3129 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_cert/rootCA.crt key=/etc/squid/ssl_cert/rootCA.key options=SINGLE_DH_USE,SINGLE_ECDH_USE tls-dh=/etc/squid/ssl_cert/dhparam.pem

tls_outgoing_options options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE

tls_outgoing_options cafile=/usr/local/openssl/cabundle.file
# hardening
tls_outgoing_options cipher=HIGH:MEDIUM:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS

# Uncomment and adjust the following to add a disk cache directory.
cache_dir ufs /var/cache/squid 100 16 256

# Leave coredumps in the first cache dir
coredump_dir /var/cache/squid


#
# Add any of your own refresh_pattern entries above these.
#
refresh_pattern ^ftp:		1440	20%	10080
refresh_pattern ^gopher:	1440	0%	1440
refresh_pattern -i (/cgi-bin/|\?) 0	0%	0
refresh_pattern .		0	20%	4320


debug_options ALL,1 9,5

from selenium-wire.

wkeeling avatar wkeeling commented on May 20, 2024

@appunni-dishq Thanks for that. I'll see if I can reproduce and report back here.

from selenium-wire.

wkeeling avatar wkeeling commented on May 20, 2024

@appunni-dishq Just an update on this. Running Selenium Wire through a local Squid 4.4 using Squid's default config file works without any problem for me with the following options:

options = {
    'proxy': {
        'http': 'http://localhost:3128',
        'https': 'https://localhost:3128'
    }
}
driver = webdriver.Firefox(seleniumwire_options=options)

However, using your example config file above is giving me some errors on the ssl-bump configuration. I'll do some further investigation into that, but I'm suspecting that this overall issue is probably related to ssl-bump in some way.

from selenium-wire.

appunni-dishq avatar appunni-dishq commented on May 20, 2024

I am using chrome. Could be related to chrome's latest security updates. Can you please share the config file. I will try to replicate. @wkeeling Yes ssl-bump has issues. Not enough documentation regarding ssl-bump in latest tls implementations. Without ssl bump, I already have HAproxy which is more than good enough. As almost all requests are HTTPS.

from selenium-wire.

wkeeling avatar wkeeling commented on May 20, 2024

Chrome (Version 70.0.3538.77) also seems to work fine using the default config file and same options specified above, although it does display a message about the page not being secure.

The config file is:

#
# Recommended minimum configuration:
#

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 0.0.0.1-0.255.255.255	# RFC 1122 "this" network (LAN)
acl localnet src 10.0.0.0/8		# RFC 1918 local private network (LAN)
acl localnet src 100.64.0.0/10		# RFC 6598 shared address space (CGN)
acl localnet src 169.254.0.0/16 	# RFC 3927 link-local (directly plugged) machines
acl localnet src 172.16.0.0/12		# RFC 1918 local private network (LAN)
acl localnet src 192.168.0.0/16		# RFC 1918 local private network (LAN)
acl localnet src fc00::/7       	# RFC 4193 local private network range
acl localnet src fe80::/10      	# RFC 4291 link-local (directly plugged) machines

acl SSL_ports port 443
acl Safe_ports port 80		# http
acl Safe_ports port 21		# ftp
acl Safe_ports port 443		# https
acl Safe_ports port 70		# gopher
acl Safe_ports port 210		# wais
acl Safe_ports port 1025-65535	# unregistered ports
acl Safe_ports port 280		# http-mgmt
acl Safe_ports port 488		# gss-http
acl Safe_ports port 591		# filemaker
acl Safe_ports port 777		# multiling http
acl CONNECT method CONNECT

#
# Recommended minimum Access Permission configuration:
#
# Deny requests to certain unsafe ports
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost

# And finally deny all other access to this proxy
http_access deny all

# Squid normally listens to port 3128
http_port 3128

# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /usr/local/squid/var/cache/squid 100 16 256

# Leave coredumps in the first cache dir
coredump_dir /usr/local/squid/var/cache/squid

#
# Add any of your own refresh_pattern entries above these.
#
refresh_pattern ^ftp:		1440	20%	10080
refresh_pattern ^gopher:	1440	0%	1440
refresh_pattern -i (/cgi-bin/|\?) 0	0%	0
refresh_pattern .		0	20%	4320

from selenium-wire.

appunni-dishq avatar appunni-dishq commented on May 20, 2024

#49 I think these are related

from selenium-wire.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.