"set_proxy" doesn't seem to have an effect for external resources about apparition HOT 12 CLOSED

@thbar This is definitely a puffing-billy issue - The certificate chain it builds isn't compatible with Chrome for some reason. Not sure exactly why, I would guess it's using an out of date encryption but haven't actually dug far enough to confirm. If you just stop it from building its own cert chain (and pass the ignore_https_errors: true option to the apparition driver - which would have been needed for self signed cert on the proxy anyway) then it works fine. You can stop puffing-billy from using it's own cert by changing start_tls(certificate_chain(url)) to start_tls() in https://github.com/oesmith/puffing-billy/blob/master/lib/billy/proxy_connection.rb

from apparition.

@thbar Ok, this is purely a self-signed cert issue. You can either figure out how to set up the certs as mentioned in the puffing-billy readme - https://github.com/oesmith/puffing-billy#ssl-usage - or an easier solution is just to set the ignore-certificate-errors command line option for Chrome. You can do that in the driver registration like this

Capybara.register_driver :apparition_with_puffing_billy do |app|
  options = {
    window_size: [1280, 1024],
    headless: true,
    js_errors: true,
    ignore_https_errors: true,
    browser: {
      'ignore-certificate-errors' => nil
    }
  }
  Capybara::Apparition::Driver.new(app, options).tap do |driver|
    driver.set_proxy(Billy.proxy.host, Billy.proxy.port)
  end
end

from apparition.

hmmm... I'll look at this in a bit but how exactly are you configuring the proxy - like this https://github.com/oesmith/puffing-billy/pull/258/files ? Are you sure it's not something you've configured in puffing-billy (IIRC poltergeist/phantomjs had an issue around proxying localhost so maybe you implemented some workaround for that which is interfering)?

from apparition.

@thbar Ok - I set up a proxy locally and used the following driver registration to configure for it (also tried with headless: true)

Capybara.register_driver(:apparition) do |app|
   Capybara::Apparition::Driver.new(app, ignore_https_errors: true, headless: false).tap do |driver|
     driver.set_proxy('127.0.0.1', 8888)
   end
end

It made the initial request, the https redirect, and all the resource requests through the proxy. I'm going to need more details on your exact configuration to figure out what is happening for you.

from apparition.

With a bit of work, I've been able to rebuild a self-contained reproduction here (much smaller than my whole Rails app!):

https://github.com/thbar/repro-bug-proxy (with CI build!)

This is a bit different than my app, because in the reproduction, I'm only asserting that a <script> tag referring to external resources is called (or not), while in my app, I have actually JSONP calls occurring to external domains (but this was simpler to achieve this instead & enough to reproduce the behaviour).

I implemented 3 tests: one for poltergeist (and indeed I saw what you mentioned, that the proxying to localhost is apparently not working, but it's not a problem for my case), one for apparition and also one for the cuprite driver.

It's interesting that both apparition and cuprite fail similarly here: I presume it's because both drivers use CDP, and maybe there is something special (e.g. specific flags) to pass to Chrome via CDP to make this works. It could be some form of security restriction.

All in all - this is how far I went for now. If I find more information (I'll dig a bit), I'll update the issue!

from apparition.

I wonder if this is an issue in Chrome itself (at least there is a suspicion of this in oesmith/puffing-billy#259).

from apparition.

So it's definitely because of the https but I don't think it's an issue with Chrome - I think it's an issue or misconfiguration of puffing-billy (the request is making it to puffing-billy just not being handled). Can't prove that yet, but I'll take a more in depth look later.

from apparition.

It could also be that Chrome just doesn't like the self signed cert PB returns, and that when a proxy is in use the CDP command doesn't have a chance to ignore it.

from apparition.

@twalpole many thanks - indeed this fixes it on the reproduction!

I will verify the behaviour on the actual app & report back.

from apparition.

For some reason, the problem still occurs with the real app.

I will investigate further & re-open if I have more information.

from apparition.

@twalpole after upgrading to puffing-billy master & reviewing everything from scratch, I can confirm that your fix is working on the real app.

Again many thanks - and thanks for your work on apparition!

from apparition.

@thbar This is definitely a puffing-billy issue - The certificate chain it builds isn't compatible with Chrome for some reason. Not sure exactly why, I would guess it's using an out of date encryption but haven't actually dug far enough to confirm. If you just stop it from building its own cert chain (and pass the ignore_https_errors: true option to the apparition driver - which would have been needed for self signed cert on the proxy anyway) then it works fine. You can stop puffing-billy from using it's own cert by changing start_tls(certificate_chain(url)) to start_tls() in https://github.com/oesmith/puffing-billy/blob/master/lib/billy/proxy_connection.rb

I can confirm this fixes an issue for us locally
MacOS 10.14.3
Ruby 2.4.0
where we saw the following errors:

Assertion failed: (e > 0), function SslContext_t, file ssl.cpp

from apparition.