Code Monkey home page Code Monkey logo

fantoccini's Introduction

fantoccini

Crates.io Documentation codecov Gitter chat

A high-level API for programmatically interacting with web pages through WebDriver.

This crate uses the WebDriver protocol to drive a conforming (potentially headless) browser through relatively high-level operations such as "click this element", "submit this form", etc.

Most interactions are driven by using CSS selectors. With most WebDriver-compatible browser being fairly recent, the more expressive levels of the CSS standard are also supported, giving fairly powerful operators.

Forms are managed by first calling Client::form, and then using the methods on Form to manipulate the form's fields and eventually submitting it.

For low-level access to the page, Client::source can be used to fetch the full page HTML source code, and Client::raw_client_for to build a raw HTTP request for a particular URL.

Examples

These examples all assume that you have a WebDriver compatible process running on port 4444. A quick way to get one is to run geckodriver at the command line.

Let's start out clicking around on Wikipedia:

use fantoccini::{ClientBuilder, Locator};

// let's set up the sequence of steps we want the browser to take
#[tokio::main]
async fn main() -> Result<(), fantoccini::error::CmdError> {
    let c = ClientBuilder::native().connect("http://localhost:4444").await.expect("failed to connect to WebDriver");

    // first, go to the Wikipedia page for Foobar
    c.goto("https://en.wikipedia.org/wiki/Foobar").await?;
    let url = c.current_url().await?;
    assert_eq!(url.as_ref(), "https://en.wikipedia.org/wiki/Foobar");

    // click "Foo (disambiguation)"
    c.find(Locator::Css(".mw-disambig")).await?.click().await?;

    // click "Foo Lake"
    c.find(Locator::LinkText("Foo Lake")).await?.click().await?;

    let url = c.current_url().await?;
    assert_eq!(url.as_ref(), "https://en.wikipedia.org/wiki/Foo_Lake");

    c.close().await
}

How did we get to the Foobar page in the first place? We did a search! Let's make the program do that for us instead:

// -- snip wrapper code --
// go to the Wikipedia frontpage this time
c.goto("https://www.wikipedia.org/").await?;
// find the search form, fill it out, and submit it
let f = c.form(Locator::Css("#search-form")).await?;
f.set_by_name("search", "foobar").await?
 .submit().await?;

// we should now have ended up in the right place
let url = c.current_url().await?;
assert_eq!(url.as_ref(), "https://en.wikipedia.org/wiki/Foobar");

// -- snip wrapper code --

What if we want to download a raw file? Fantoccini has you covered:

// -- snip wrapper code --
// go back to the frontpage
c.goto("https://www.wikipedia.org/").await?;
// find the source for the Wikipedia globe
let img = c.find(Locator::Css("img.central-featured-logo")).await?;
let src = img.attr("src").await?.expect("image should have a src");
// now build a raw HTTP client request (which also has all current cookies)
let raw = img.client().raw_client_for(fantoccini::Method::GET, &src).await?;

// we then read out the image bytes
use futures_util::TryStreamExt;
let pixels = raw
    .into_body()
    .try_fold(Vec::new(), |mut data, chunk| async move {
        data.extend_from_slice(&chunk);
        Ok(data)
    })
    .await
    .map_err(fantoccini::error::CmdError::from)?;
// and voilà, we now have the bytes for the Wikipedia logo!
assert!(pixels.len() > 0);
println!("Wikipedia logo is {}b", pixels.len());

// -- snip wrapper code --

For more examples, take a look at the examples/ directory.

Contributing to fantoccini

The following information applies only to developers interested in contributing to this project. If you simply want to use it to automate web browsers you can skip this section.

How to run tests

The tests assume that you have chromedriver and geckodriver already running on your system. You can download them using the links above. Then run them from separate tabs in your terminal. They will stay running until terminated with Ctrl+C or until the terminal session is closed.

Then run cargo test from this project directory.

fantoccini's People

Contributors

aaronshim avatar aksiksi avatar bramhaag avatar chasecares avatar complexspaces avatar ctron avatar dependabot[bot] avatar evdokimovs avatar follpvosten avatar hgzimmerman avatar ipkiss42 avatar jonhoo avatar knnmran avatar kraai avatar kubastick avatar mathiaspius avatar maxcountryman avatar mpalmer avatar nkconnor avatar paolobarbolini avatar phrohdoh avatar quadrupleslap avatar roblabla avatar soerenmeier avatar stevepryde avatar tomaspecl avatar travisbrown avatar tyranron avatar x0f5c3 avatar zhiburt avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fantoccini's Issues

support interfacing with input elements directly

From the API it looks like you can only interface with forms. However, some web pages don't use them.

I would be nice to be able to interface directly with input elements like input and textarea

Request: Add an API for interacting w/ Cookies

Full disclaimer: I'm not a web expert and I'm new to using WebDriver so this might be possible using functionality that lives outside of Fantoccini, so feel free to close if that's the case

Currently, the Client does not support an easy way to interact with the current Session's cookie store unless you make a raw request and interact with the request builder/response which will only give you the cookies for that particular domain.

It would be nice to be able to interact with a Session's cookie store from the high level API to enable things like serializing/deserializing persistent cookies in applications where you may want to remember things like user log-in state. Examples of where this could be useful might include improving CI/CD test times by not requiring repeated log-ins. It could also be useful in scraper-backed user-facing applications that use technologies like OAuth instead of Basic Auth, so that the user doesn't have to repeatedly authorize the login every time they launch the application (even if it's automated).

More ergonomic API for writing tests that always terminate

The wait_for_* family of functions as they exist now are super useful, but if the element never appears, my automated tests spin forever.

Would it be possible to have an API that looks something like:

let client = fantoccini::Client::new("http://127.0.0.1/").await?;

// Tries once and fails
client.find(Locator::Css("#foo")).await?;

// Retries forever
client.find(Locator::Css("#foo")).retry_forever().await?;

// Retries for a certain amount of time
client.find(Locator::Css("#foo")).retry_for(Duration::from_secs(1)).await?;

// Retries until a deadline
let start = Instant::now();
let timeout = Duration::from_secs(10);
client.find(Locator::Css("#foo")).retry_until(timeout + start).await?;

Blocking tasks on fantoccini seems to prevent parallel execution

First: thank you for making fantoccini, it made getting this little app together a breeze!

I guess the million dollar question is: is it meant to work in parallel with multiple clients spawned?

More context here: bikeshedder/deadpool#31

In summary: I'm using fantoccini, tokio, and deadpool. deadpool seems to do the right thing when I wrote weird code to tickle it. A mock client that @bikeshedder made seemed to exhibit the correct behavior.

My profile shows a lot of blocking on the tokio thread pool, so I'm assuming I'm blocking on something somewhere that is effectively preventing parallel operation even when I have oodles of threads available.

Add a way to reconnect to an existing session

I don't think this is implemented yet, and I also don't really need it yet, but it would be nice to have; in my project, I may want to re-use the same session if the long-running process controlling it is restarted at some point.

At the moment, if I persist a session with Client::persist and then try to reconnect to the session (using Client::with_capabilities or Client::new), I get a SessionNotCreated(WebDriverError { error: SessionNotCreated, message: "Session is already started", stack: "", delete_session: false }).

I suppose the way to handle this would be adding a method like Client::with_existing_session(webdriver: &str, session_id: &str) and a corresponding Session::from_id(webdriver: &str, id: &str) or similar, which tries to connect to the given session using its ID (no idea how that's gonna work internally right now, but that would likely be the API). Then we can store the session ID and reconnect later.

I'll look into implementing this when I can find the time :)

Move to tokio crate

hyper is slowly but surely getting support for the new tokio crate, and we should start to migrate over to that. The interfaces are nicer, and it comes with some nice dependency version bumps.

Add API support for selecting between tabs / windows

I believe this is part of WebDriver, and I would use this in my project were it available, but for now I just wanted to log the feature request.

If it becomes more urgent I'll try to take a crack at implementing this.

(Thanks for the great work on fantoccini!)

Are explicit `drop`s necessary?

Writing explicit drop invocations is pretty counter-intuitive.

Is the fin function in your examples performing cleanup that is only possible if c has been dropped?

Change tests to use local HTML files

#75 added the necessary infrastructure for running tests against locally hosted HTML files rather than relying on wikipedia.org staying static. We should move pretty much all the tests to use this infrastructure so that the test suite can be run without an internet connection, and so that we do not run the risk of the tests failing if the relevant Wikipedia HTML should change.

chromedriver not respecting `--headless`

Hi! Fantoccini has been exactly what I've needed so far. Thank you so much!

I'm trying to run chromedriver with the headless and disable-gpu arguments passed through the capabilities object, using both the documented chromeOptions flag and the goog:chromeOptions flag that current selenium seems to support. Here's the code I'm using to create the capabilities:

    let mut caps = serde_json::map::Map::new();
    let chrome_opts = serde_json::json!({ "args": ["--headless", "--disable-gpu"] });
    caps.insert("goog:chromeOptions".to_string(), chrome_opts.clone());
    caps.insert("chromeOptions".to_string(), chrome_opts);

Any advice on how to debug this? I've tried increasing log verbosity for fantoccini but haven't seen any messages come up.

Loop?

Is there any way to build a loop with the client?

This does not work:

let f = client.goto("http://url.tld")
    .and_then(move |_| {
        loop {
            client.by_selector('element') // <- never gets executed
            ...
        }
    });

But I am pretty sure there is some way to use futures to create a loop, I just don't really know where to start...

Multi-threaded pooling of clients results in BrokenPipe

I've got code that looks like this:

    tokio::spawn(async move {
        scrape_worker(scraping.clone()).await;
    }
    );

if I change it to await the scrape_worker without spawning a task it works fine. The error message I get right now is:


[2020-01-16T20:49:06Z DEBUG scraping_rs] Starting scrape of url: "https://phantomjs.org/"
thread 'tokio-runtime-worker' panicked at 'gai background task failed: JoinError::Cancelled', /home/callen/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.13.1/src/client/connect/dns.rs:144:30
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
[2020-01-16T20:49:06Z ERROR scraping_rs] Error instantiating client error was: Lost(Custom { kind: BrokenPipe, error: "WebDriver session was closed while waiting" })
thread 'tokio-runtime-worker' panicked at 'Failed to pull a client from the pool: Backend(NotJson("Couldn\'t instantiate client"))', src/libcore/result.rs:1165:5

With the relevant error being: Lost(Custom { kind: BrokenPipe, error: "WebDriver session was closed while waiting" }).

It reproduces with Chromedriver 79.0.3945.36 and geckodriver 0.26.

Based on https://github.com/jonhoo/fantoccini/blob/master/tests/common.rs I feel like multi-threaded operation was intended here but I'm not sure what I'm doing wrong. I'm using deadpool to pool the clients:

#[async_trait]
impl deadpool::Manager<Client, Error> for Manager {
    async fn create(&self) -> Result<Client, Error> {
        let mut client = Client::new("http://localhost:9515").await.map_err(|err| {
            error!("Error instantiating client error was: {:?}", err);
            fantoccini::error::CmdError::NotJson("Couldn't instantiate client".to_string())
        })?;
        client.persist().await?;
        Ok(client)
    }
    async fn recycle(&self, conn: &mut Client) -> deadpool::RecycleResult<Error> {
        Ok(())
    }
}

It dies on the very first URL, never spawns a browser window.

chromeOptions is not the name of a known capability or extension capability

I download and installed geckodriver

wget https://github.com/mozilla/geckodriver/releases/{latest}linux64.tar.gz
tar -xvzf geckodriver*
chmod +x geckodriver
geckodriver
(running on port 4444)

Used this code:

extern crate tokio;
extern crate futures;
extern crate fantoccini;

fn main() {
    use fantoccini::{Client, Locator};
    use futures::future::Future;
    let c = Client::new("http://localhost:4444");
   
    // let's set up the sequence of steps we want the browser to take
    tokio::run(
        c
            .map_err(|e| {
                unimplemented!("failed to connect to WebDriver: {:?}", e)
            })
            .and_then(|c| {
                // first, go to the Wikipedia page for Foobar
                c.goto("https://en.wikipedia.org/wiki/Foobar")
            })
            .and_then(|mut c| c.current_url().map(move |url| (c, url)))
            .and_then(|(mut c, url)| {
                assert_eq!(url.as_ref(), "https://en.wikipedia.org/wiki/Foobar");
                // click "Foo (disambiguation)"
                c.find(Locator::Css(".mw-disambig"))
            })
            .and_then(|e| e.click())
            .and_then(|mut c| {
                // click "Foo Lake"
                c.find(Locator::LinkText("Foo Lake"))
            })
            .and_then(|e| e.click())
            .and_then(|mut c| c.current_url())
            .and_then(|url| {
                assert_eq!(url.as_ref(), "https://en.wikipedia.org/wiki/Foo_Lake");
                Ok(())
            })
            .map_err(|e| {
                panic!("a WebDriver command failed: {:?}", e);
            })
    );
}

Result:

$ cargo run   
   Compiling web_ui_testing v0.1.0 (file:///home/rochacbruno/Projects/rust/i_am_learning_rust/web_ui_testing)           
    Finished dev [unoptimized + debuginfo] target(s) in 2.62s
     Running `target/debug/web_ui_testing`
thread 'tokio-runtime-worker-2' panicked at 'unexpected webdriver error; webdriver returned error: chromeOptions is not the name of a known capability or extension capability', /home/rochacbruno/.cargo/registry/src/github.com-1ecc6299db9ec823/fantoccini-0.11.2/src/session.rs:314:17
note: Run with `RUST_BACKTRACE=1` for a backtrace.
thread 'tokio-runtime-worker-2' panicked at 'called `Option::unwrap()` on a `None` value', libcore/option.rs:345:21

Expose "Find Element From Element" WebDriver API call

I'm currently trying out fantoccini, and this is one (most likely easily fixable) issue I've found: When working with HTML elements that don't have any clear identifier attached, it can be kind of hard to impossible to find their child elements while still keeping context.

Example:

<div class="table">
  <div class="row">
    <div class="col name-col">Thing 1</div>
    <div class="col action-col"><button>Some action for Thing 1</button></div>
  </div>
  <div class="row">
    <div class="col name-col">Thing 2</div>
    <div class="col action-col"><button>Some action for Thing 2</button></div>
  </div>
</div>

Now what I want is:

  1. Find each div.row. That is easily done with find_all.
  2. Now, get each row's div.name-col text and div.action-col > button while keeping their relation and an Element on the button so I can invoke events on it.

At the moment, I can't see how I would perform the second step without falling back to invoking jQuery or something and parsing the result, or relying on indices (which is never a good idea).
I think the right WebDriver API call would be Find Element From Element; the webdriver crate has it as WebDriverCommand::FindElementElement.

API idea: Most WebDriver clients expose this call by having a "find element" method on their web element type; that would be fantoccini::Element::find here (and fantoccini::Element::find_all for "Find Elements From Element" to stay consistent with the Client API), which would internally invoke something like fantoccini::Client::find_from/fantoccini::Client::find_all_from. Those could also be pub in fantoccini, they work like find/find_all, but take a fantoccini::Element as the first argument.

It would work like this:

let rows = client.find_all(Locator::Css("div.table > div.row")).await?;
let mut col_groups = Vec::new();
for row in rows.iter() {
    let name = row.find(Css("div.name-col")).await?.text().await?;
    let button = row.find(Css("div.action-col > button")).await?;
    col_groups.push((name, button));
}
// Now I can present the user with the options and they can choose what to click on

I would really like to contribute and implement this myself as well, but as I only want to work with the .await API, I'd prefer to start working on it when the await PR is merged :)

Full example

Although there is some example code on how to use the library there isn't a full example that covers the setup. For example I wasn't really aware I would have to add the futures library as well as tokio in order to use this. So I think a full working example would be nice.

Funny comment in README

The examples will be using unwrap generously

None of the examples use unwrap at all. 😃

Fix interaction with WebKitDriver

Originally reported in #36.
Running fantoccini against WebKitDriver currently fails with

SessionNotCreated(WebDriverError {
  error: SessionNotCreated,
  message: "Failed to match capabilities",
  stack: "",
  delete_session: false
})

This seems to stem from this line in the WebKitDriver source (GH mirror link), but I cannot for the life of me see why matchedCapabilitiesList would be empty. Setting browserName or other parameters in the alwaysMatch capabilities list does not seem to make a difference. I think this might be a bug in WebKitDriver, but some more investigation is probably warranted.

The basic example might be broken?

I have the geckodriver running on port 4444 but cargo run in the example/basic_setup gives the following errors:

~/tmp/fan/exa/basic_setup:master> cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.03s
     Running `target/debug/basic_setup`
thread 'thread 'tokio-runtime-worker-2tokio-runtime-worker-0' panicked at '' panicked at 'not yet implemented: failed to connect to WebDriver: SessionNotCreated(WebDriverError { error: SessionNotCreated, message: "Expected browser binary location, but unable to find binary in default location, no \'moz:firefoxOptions.binary\' capability provided, and no binary flag set on the command line", stack: "", delete_session: false })called `Option::unwrap()` on a `None` value', ', src/main.rssrc/libcore/option.rs::16378::2621

I have tried it on macOS and linux and I got the same error. Any idea what is wrong here? Thx

nightly-x86_64-apple-darwin (default)
rustc 1.39.0-nightly (f0b58fcf0 2019-09-11)

Improve wait_* style methods

Currently, wait_for_find just spins trying to look for the element in question. A much better strategy would be to use MutationObserver (specifically observe on the document root) to wait for new elements to be added to the page, and only re-execute the webdriver find when something new has appeared.

Request: Add ability to switch frames

I'm trying to test a webpage that makes use of an iframe using this library.
Unfortunately, in order to interact with elements inside of iframes, the Client needs to switch into the iframe.
There exists command enums in the webdriver crate for performing the necessary actions, so this should be possible:

  • WebDriverCommand::SwitchToFrame
  • WebDriverCommand::SwitchToParentFrame

I have tried to do this myself in my fork, but I could not get my changes to work, with errors I don't quite recall being returned.
The PR with the relevant changes is available here: https://github.com/hgzimmerman/fantoccini/pull/1/files#diff-b4aea3e418ccdb71239b96952d9cddb6R765 (due to vendoring and rust version lock at work, I am basing this off of the v0.11.9 release, but the changes shouldn't be materially different against master).

This very well may be a bug in the webdriver crate, but since they lack a github page for the library, I'm filing the issue here with the hope that someone more familiar with that library may be able to assist.

firefox version: Mozilla Firefox 72.0.1
geckodriver version: geckodriver 0.26.0
OS version: Ubuntu 18.04.3 LTS
library version: v0.11.9 (fork)

EDIT:
This is the error following calling any of my frame functions:

WebDriverErr(Lost(Custom { kind: BrokenPipe, error: "WebDriver session was closed while waiting"}))

Add synchronous API wrapper

As observed here, there are some pretty good reasons to make the interface to interact with browsers synchronous. Or at least give the option of having the nicer synchronous interface when you don't need the async. We should be able to provide synchronous wrapper types for most operations pretty easily, so that would be a good addition!

Wrapping capabilities for initialization

Hi I want to use with_capabilities to create a new client, but not sure if doing it right. This is what I'm doing:

extern crate tokio;
extern crate futures;
extern crate fantoccini;
extern crate webdriver;
extern crate serde_json;

use fantoccini::{Client, Locator};
use futures::future::Future;
use webdriver::capabilities::Capabilities;

fn main() {
    let mut cap = Capabilities::new();
    let arg = serde_json::from_str("{\"args\": [\"-headless\"]}").unwrap();
    cap.insert("moz:firefoxOptions".to_string(), arg);
    let c = Client::with_capabilities("http://localhost:4444", cap);
   
   // rest of the code down here...
}

If this is the right way to do it I was wondering if we could receive in the capabilities in another way that didn't involve me knowing about webdriver and serde_json and having to add them to my dependencies.

I'm open to making a PR if you prefer to discuss over actual code.

`Element::click()` not being processed by `safaridriver`

Running a simple future along the lines of:

        client.goto("<url>").map_err()
        .and_then(|mut client| client.find(Locator::Id("login-link")))
        .and_then(|link| link.click())

When using safaridriver as the WebDriver implementation will result in the found element not being clicked. Adding some diagnostic prints after the find() indicates that the element does in fact get found, but looking at the logs generated by safaridriver indicates that the click event is being sent to coord (0,0). It's possible that this is a bug in the WebDriver crate and not Fantoccini itself but I'm admittedly having a tricky time debugging it on my own because I'm still very very new to tokio/futures and they're a little hard to understand at first.

Here is a snippet of the log from safaridriver showing the successful element request followed by the click going to (0,0) [the line with Automation.performInteractionSequence]:

2019/10/31 13:22:59:396  HTTPServer: POST http://localhost:4444/session/02F989FF-31C2-4279-B68D-2B2E70B61FB5/element
2019/10/31 13:22:59:396  HTTPServer: {"using":"xpath","value":"//*[@id=\"login-link\"]"}
2019/10/31 13:22:59:396  AutomationProtocol: SEND {"method":"Automation.isShowingJavaScriptDialog","params":{"browsingContextHandle":"page-994B2B31-57A0-471E-89E0-DEEF68E2DDBE"},"id":11}
2019/10/31 13:22:59:397  AutomationProtocol: RECV {"result":{"result":false},"id":11}
2019/10/31 13:22:59:397  AutomationProtocol: SEND {"method":"Automation.evaluateJavaScriptFunction","params":{"arguments":["\"xpath\"","null","\"\\/\\/*[@id=\\\"login-link\\\"]\"","true","0"],"frameHandle":"","expectsImplicitCallbackArgument":true,"function":"//# sourceURL=__InjectedScript_WDFindNodes.js\nfunction(strategy,ancestorElement,query,firstResultOnly,timeoutDuration,callback){ancestorElement=ancestorElement||document;switch(strategy){case\"id\":strategy=\"css selector\";query=\"[id=\\\"\"+escape(query)+\"\\\"]\";break;case\"name\":strategy=\"css selector\";query=\"[name=\\\"\"+escape(query)+\"\\\"]\";break;}\nswitch(strategy){case\"css selector\":case\"link text\":case\"partial link text\":case\"tag name\":case\"class name\":case\"xpath\":break;default:\n throw{name:\"InvalidParameter\",message:(\"Unsupported locator strategy: \"+strategy+\".\")};}\nfunction escape(string){return string.replace(/\\\\/g,\"\\\\\\\\\").replace(/\"/g,\"\\\\\\\"\");}\nfunction tryToFindNode(){try{switch(strategy){case\"css selector\":if(firstResultOnly)\nreturn ancestorElement.querySelector(query)||null;return Array.from(ancestorElement.querySelectorAll(query));case\"link text\":let linkTextResult=[];for(let link of ancestorElement.getElementsByTagName(\"a\")){if(link.text.trim()==query){linkTextResult.push(link);if(firstResultOnly)\nbreak;}}\nif(firstResultOnly)\nreturn linkTextResult[0]||null;return linkTextResult;case\"partial link text\":let partialLinkResult=[];for(let link of ancestorElement.getElementsByTagName(\"a\")){if(link.text.includes(query)){partialLinkResult.push(link);if(firstResultOnly)\nbreak;}}\nif(firstResultOnly)\nreturn partialLinkResult[0]||null;return partialLinkResult;case\"tag name\":let tagNameResult=ancestorElement.getElementsByTagName(query);if(firstResultOnly)\nreturn tagNameResult[0]||null;return Array.from(tagNameResult);case\"class name\":let classNameResult=ancestorElement.getElementsByClassName(query);if(firstResultOnly)\nreturn classNameResult[0]||null;return Array.from(classNameResult);case\"xpath\":if(firstResultOnly){let xpathResult=document.evaluate(query,ancestorElement,null,XPathResult.FIRST_ORDERED_NODE_TYPE,null);if(!xpathResult)\nreturn null;return xpathResult.singleNodeValue;}\nlet xpathResult=document.evaluate(query,ancestorElement,null,XPathResult.ORDERED_NODE_SNAPSHOT_TYPE,null);if(!xpathResult||!xpathResult.snapshotLength)\nreturn[];let arrayResult=[];for(let i=0;i\u003CxpathResult.snapshotLength;++i)\narrayResult.push(xpathResult.snapshotItem(i));return arrayResult;}}catch(error){\n throw{name:\"InvalidSelector\",message:error.message};}}\nconst pollInterval=50;let pollUntil=performance.now()+timeoutDuration;function pollForNode(){let result=tryToFindNode();if(typeof result===\"string\"||result instanceof Node||(result instanceof Array&&result.length)){callback(result);return;}\nlet durationRemaining=pollUntil-performance.now();if(durationRemaining\u003CpollInterval){callback(firstResultOnly?null:[]);return;}\nsetTimeout(pollForNode,pollInterval);}\npollForNode();}\n","browsingContextHandle":"page-994B2B31-57A0-471E-89E0-DEEF68E2DDBE","callbackTimeout":0},"id":12}
2019/10/31 13:22:59:407  AutomationProtocol: RECV {"result":{"result":"{\"session-node-02F989FF-31C2-4279-B68D-2B2E70B61FB5\":\"node-19CDF7E9-2E53-45D1-AFE3-5354727A0D2E\"}"},"id":12}
2019/10/31 13:22:59:407  HTTPServer: --> HTTP 200
2019/10/31 13:22:59:408  HTTPServer: --> {"value":{"element-6066-11e4-a52e-4f735466cecf":"node-19CDF7E9-2E53-45D1-AFE3-5354727A0D2E"}}
2019/10/31 13:22:59:418  HTTPServer: POST http://localhost:4444/session/02F989FF-31C2-4279-B68D-2B2E70B61FB5/element/node-19CDF7E9-2E53-45D1-AFE3-5354727A0D2E/click
2019/10/31 13:22:59:418  HTTPServer: {}
2019/10/31 13:22:59:418  AutomationProtocol: SEND {"method":"Automation.isShowingJavaScriptDialog","params":{"browsingContextHandle":"page-994B2B31-57A0-471E-89E0-DEEF68E2DDBE"},"id":13}
2019/10/31 13:22:59:419  AutomationProtocol: RECV {"result":{"result":false},"id":13}
2019/10/31 13:22:59:419  AutomationProtocol: SEND {"method":"Automation.evaluateJavaScriptFunction","params":{"arguments":["{\"session-node-02F989FF-31C2-4279-B68D-2B2E70B61FB5\":\"node-19CDF7E9-2E53-45D1-AFE3-5354727A0D2E\"}"],"frameHandle":"","expectsImplicitCallbackArgument":false,"function":"// sourceURL=__InjectedScript_WebDriver_ElementFetchAttributes.js\n//# sourceURL=__InjectedScript_WDElementFetchAttributes.js\nfunction fetchTagNameAndAttributesOfElement(element){\"use strict\";let tagName=element.tagName.toUpperCase();let attributes={};for(let i=0;i\u003Celement.attributes.length;++i){let name=element.attributes[i].name.toLowerCase();let value=element.attributes[i].value;if(!value.length)\nvalue=null;attributes[name]=value;}\nreturn{tagName,attributes};}\n","browsingContextHandle":"page-994B2B31-57A0-471E-89E0-DEEF68E2DDBE","callbackTimeout":30000},"id":14}
2019/10/31 13:22:59:465  AutomationProtocol: RECV {"result":{"result":"{\"tagName\":\"A\",\"attributes\":{\"href\":\"/login\",\"id\":\"login-link\",\"class\":\"user-interactions-action-signin user-interactions-action\"}}"},"id":14}
2019/10/31 13:22:59:466  AutomationProtocol: SEND {"method":"Automation.evaluateJavaScriptFunction","params":{"arguments":["{\"session-node-02F989FF-31C2-4279-B68D-2B2E70B61FB5\":\"node-19CDF7E9-2E53-45D1-AFE3-5354727A0D2E\"}"],"frameHandle":"","expectsImplicitCallbackArgument":false,"function":"// sourceURL=__InjectedScript_WebDriver_ScrollIntoView.js\nfunction(element) { element.scrollIntoViewIfNeeded(); }","browsingContextHandle":"page-994B2B31-57A0-471E-89E0-DEEF68E2DDBE","callbackTimeout":30000},"id":15}
2019/10/31 13:22:59:523  AutomationProtocol: RECV {"result":{"result":"null"},"id":15}
2019/10/31 13:22:59:524  AutomationProtocol: SEND {"method":"Automation.evaluateJavaScriptFunction","params":{"arguments":["{\"session-node-02F989FF-31C2-4279-B68D-2B2E70B61FB5\":\"node-19CDF7E9-2E53-45D1-AFE3-5354727A0D2E\"}"],"frameHandle":"","expectsImplicitCallbackArgument":false,"function":"// sourceURL=__InjectedScript_WebDriver_ElementDisplayed.js\n//# sourceURL=__InjectedScript_WDElementDisplayed.js\nfunction isShown(element){\"use strict\";function nodeIsElement(node){if(!node)\nreturn false;switch(node.nodeType){case Node.ELEMENT_NODE:case Node.DOCUMENT_NODE:case Node.DOCUMENT_FRAGMENT_NODE:return true;default:return false;}}\nfunction parentElementForElement(element){if(!element)\nreturn null;return enclosingNodeOrSelfMatchingPredicate(element.parentNode,nodeIsElement);}\nfunction enclosingNodeOrSelfMatchingPredicate(targetNode,predicate){for(let node=targetNode;node&&node!==targetNode.ownerDocument;node=node.parentNode)\nif(predicate(node))\nreturn node;return null;}\nfunction enclosingElementOrSelfMatchingPredicate(targetElement,predicate){for(let element=targetElement;element&&element!==targetElement.ownerDocument;element=parentElementForElement(element))\nif(predicate(element))\nreturn element;return null;}\nfunction cascadedStylePropertyForElement(element,property){if(!element||!property)\nreturn null;let computedStyle=window.getComputedStyle(element);let computedStyleProperty=computedStyle.getPropertyValue(property);if(computedStyleProperty&&computedStyleProperty!==\"inherit\")\nreturn computedStyleProperty;\n \nlet parentElement=parentElementForElement(element);return cascadedStylePropertyForElement(parentElement,property);}\nfunction elementSubtreeHasNonZeroDimensions(element){let boundingBox=element.getBoundingClientRect();if(boundingBox.width\u003E0&&boundingBox.height\u003E0)\nreturn true;if(element.tagName.toUpperCase()===\"PATH\"&&boundingBox.width+boundingBox.height\u003E0){let strokeWidth=cascadedStylePropertyForElement(element,\"stroke-width\");return!!strokeWidth&&(parseInt(strokeWidth,10)\u003E0);}\nlet cascadedOverflow=cascadedStylePropertyForElement(element,\"overflow\");if(cascadedOverflow===\"hidden\")\nreturn false;\nreturn Array.from(element.childNodes).some((childNode)=\u003E{if(childNode.nodeType===Node.TEXT_NODE)\nreturn true;if(nodeIsElement(childNode))\nreturn elementSubtreeHasNonZeroDimensions(childNode);return false;});}\nfunction elementOverflowsContainer(element){let cascadedOverflow=cascadedStylePropertyForElement(element,\"overflow\");if(cascadedOverflow!==\"hidden\")\nreturn false;return true;}\nfunction isElementSubtreeHiddenByOverflow(element){if(!element)\nreturn false;if(!elementOverflowsContainer(element))\nreturn false;if(!element.childNodes.length)\nreturn false;return Array.from(element.childNodes).every((childNode)=\u003E{if(!nodeIsElement(childNode))\nreturn true;if(!elementSubtreeHasNonZeroDimensions(childNode))\nreturn true;return isElementSubtreeHiddenByOverflow(childNode);});}\nif(!(element instanceof Element))\nthrow new Error(\"Cannot check the displayedness of a non-Element argument.\");if(!document.contains(element))\nreturn false;switch(element.tagName.toUpperCase()){case\"BODY\":return true;case\"SCRIPT\":case\"NOSCRIPT\":return false;case\"OPTGROUP\":case\"OPTION\":let enclosingSelectElement=enclosingNodeOrSelfMatchingPredicate(element,(e)=\u003Ee.tagName.toUpperCase()===\"SELECT\");return isShown(enclosingSelectElement);case\"INPUT\":if(element.type===\"hidden\")\nreturn false;break;case\"MAP\":default:break;}\nif(cascadedStylePropertyForElement(element,\"visibility\")!==\"visible\")\nreturn false;let hasAncestorWithZeroOpacity=!!enclosingElementOrSelfMatchingPredicate(element,(e)=\u003E{return Number(cascadedStylePropertyForElement(e,\"opacity\"))===0;});let hasAncestorWithDisplayNone=!!enclosingElementOrSelfMatchingPredicate(element,(e)=\u003E{return cascadedStylePropertyForElement(e,\"display\")===\"none\";});if(hasAncestorWithZeroOpacity||hasAncestorWithDisplayNone)\nreturn false;if(!elementSubtreeHasNonZeroDimensions(element))\nreturn false;if(isElementSubtreeHiddenByOverflow(element))\nreturn false;return true;}\n","browsingContextHandle":"page-994B2B31-57A0-471E-89E0-DEEF68E2DDBE","callbackTimeout":30000},"id":16}
2019/10/31 13:22:59:610  AutomationProtocol: RECV {"result":{"result":"true"},"id":16}
2019/10/31 13:22:59:610  AutomationProtocol: SEND {"method":"Automation.computeElementLayout","params":{"nodeHandle":"node-19CDF7E9-2E53-45D1-AFE3-5354727A0D2E","frameHandle":"","coordinateSystem":"LayoutViewport","browsingContextHandle":"page-994B2B31-57A0-471E-89E0-DEEF68E2DDBE","scrollIntoViewIfNeeded":false},"id":17}
2019/10/31 13:22:59:636  AutomationProtocol: RECV {"result":{"rect":{"origin":{"x":1323,"y":52},"size":{"width":80,"height":36}},"inViewCenterPoint":{"x":1362,"y":70},"isObscured":false},"id":17}
2019/10/31 13:22:59:637  AutomationProtocol: SEND {"method":"Automation.performInteractionSequence","params":{"frameHandle":"","inputSources":[{"sourceId":"mouse","sourceType":"Mouse"}],"steps":[{"states":[{"sourceId":"mouse","location":{"x":0,"y":0},"origin":"Element","nodeHandle":"node-19CDF7E9-2E53-45D1-AFE3-5354727A0D2E"}]},{"states":[{"sourceId":"mouse","pressedButton":"Left","location":{"x":0,"y":0},"origin":"Element","nodeHandle":"node-19CDF7E9-2E53-45D1-AFE3-5354727A0D2E"}]},{"states":[{"sourceId":"mouse","origin":"Pointer","nodeHandle":"node-19CDF7E9-2E53-45D1-AFE3-5354727A0D2E"}]}],"handle":"page-994B2B31-57A0-471E-89E0-DEEF68E2DDBE"},"id":18}
2019/10/31 13:22:59:665  AutomationProtocol: RECV {"result":{},"id":18}
2019/10/31 13:22:59:665  AutomationProtocol: SEND {"method":"Automation.waitForNavigationToComplete","params":{"frameHandle":"","pageLoadTimeout":300000,"pageLoadStrategy":"Normal","browsingContextHandle":"page-994B2B31-57A0-471E-89E0-DEEF68E2DDBE"},"id":19}
2019/10/31 13:22:59:666  AutomationProtocol: RECV {"result":{},"id":19}

Request: Add functionality to run a script every time you navigate before the page loads

Hi all, I need to run a script every time i load a page that occurs before the page actually loads. I am currently looking into a solution for this but if anyone knows how its done I would love to know how. I have currently tried running client.execute() before client.goto() but that doesn't seem to do the trick (which makes sense). Will continue searching for a way to do this and will update here if I find one.

Futures 0.3

Since futures will be stable in 1.36, and the latest futures crate version has already Updated to stabilized futures_api as per https://github.com/rust-lang-nursery/futures-rs/releases I think it would be a big ergonomic upgrade for the users if Fantoccini updated to futures 0.3.

This would allow us to use the nightly async await feature without having to convert between 0.1 and 0.3 before every await call.

This would alleviate issues such as #43

Is this change something Fantoccini would be willing to embrace?

Request: A way to connect to an already created session by passing through a session ID

Hi all, I am currently trying to do automation with chromeDriver and I am having a tough time with passing through args for my sessions. A way around this I thought would be to initialize the session through curl then connect to it and control it with Fantoccini. I may just be missing the point here but I have been struggling with this for a while and I thought this would be a potential fix. Feel free to leave comments/suggestions :)

BadURL(RelativeUrlWithoutBase) when using `safaridriver` w/ Safari 13/macOS Mojave on fantoccini stable (0.11.9)

I'm working on a scraper for a site that doesn't have a proper OAuth API and I'd like to be able to support multiple browsers. I've been prototyping w/ geckodriver without issue but today I tried to use safaridriver and I'm not even able to issue a simple client.goto() or run the basic example as I immediately error out with the error in the issue title. Apple claims that safaridriver on Safari 12 and later supports the W3C API and not the Selenium API (https://developer.apple.com/documentation/webkit/macos_webdriver_commands_for_safari_12_and_later) but given their track record with docs in the last few years I wouldn't be totally shocked if this is a half-truth.

It's not immediately obvious to me that there's a way to extract the offending URL from the CmdError or ParseError to provide more information. I'd love to help debug if you don't have easy access to macOS, maybe you could point me to a place in the internals that I should start digging/instrumenting?

conservative_impl_trait is stable

warning: this feature has been stable since 1.26.0. Attribute no longer needed
   --> src/lib.rs:189:12
    |
189 | #![feature(conservative_impl_trait)]

How to run it?

What should be running on the port 4444? chromedriver? Mine is running on 9515, but it's not available on localhost:9515

$ chromedriver 
Starting ChromeDriver 2.29 on port 9515
Only local connections are allowed.

And thus this:

let mut c = Client::new("http://localhost:9515").unwrap();
throws an exception:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Failed(Io(Error { repr: Os { code: 111, message: "Connection refused" } }))', src/libcore/result.rs:859
note: Run with `RUST_BACKTRACE=1` for a backtrace.

And I ran this:

$ geckodriver
1497632367412   geckodriver     INFO    Listening on 127.0.0.1:4444

and still got:

  let mut c = Client::new("127.0.0.1:4444").unwrap();
  
  ===>

  thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Failed(Io(Error { repr: Os { code: 
  111, message: "Connection refused" } }))', src/libcore/result.rs:859
  note: Run with `RUST_BACKTRACE=1` for a backtrace.

the trait bound `std::option::Option<impl futures::Future>: futures::Future` is not satisfied

Hi Jonhoo,

I have followed your first code example of "clicking around on Wikipedia". However, I got the following issue:

error[E0277]: the trait bound `std::option::Option<impl futures::Future>: futures::Future` is not satisfied
  --> src/main.rs:46:10
   |
46 |     core.run(fin).unwrap();
   |          ^^^ the trait `futures::Future` is not implemented for `std::option::Option<impl futures::Future>`

error: aborting due to previous error

Does not seem to work with multiple redirects

I am trying to use fantoccini to automate the login and fecthing of some data from an energi provider website.

The main landing page is redirecting to an external page for authentication but it does that by going through an intermediary page.

This intermediary page is not even visible in the browser when doing the whole thing manually butfantoccinialways reports this intermediary page as current_url() even if on the screen I can see that it reached the authentication page.

So, it looks like fantoccini gets locked on the first redirect url. Is there a way to make it advance to the current page that it's displayed on the screen?

I did several tries with wait_for_navigation() and wait_for_find() but I always end up in a deadlock.

If needed, I can add some of the code I've tried.

Looking forward for your feedback.

Thanks

Real-time discussion channel

Having a channel (IRC, gitter, ...) to host real-time discussions could be advantageous to the project as developers could discuss features and implementations before work is started then reference conversations during code review, etc.

In my particular case I need to discuss some semantics with you (though I could create an issue for this, too) related to #6.

Docs link broken

Hey,

I'm looking at your crate for a project but the doc.rs link seems to be broken.
Since I'm here I may as well ask, would I be able to take screenshots using fantoccini? I know WebDriver supports it, does fantoccini cover all the WebDriver API?

Thanks!

Expand all docs to have examples (including HTML)

I'd like all the methods we provide to have doctests, and ideally to also give the HTML that they are operating over. I'd be happy to alos host those HTML snippets somewhere so that we don't have all the tests use wikipedia.org (which could also change).

It's a little sad that we need to disable these tests by default. We should probably get phantomjs or slimerjs running on travis so that it can run all the test cases too!

/cc @phrohdoh

How to get page source after page load?

Is it the case that I only have to await Client::goto?
And is it also the case that Client::source always returns the current source (modified by any js code that ran before)?

compiler failures in tests

Failed during command: rs-test -- --ignored --test-threads=1

failures:
---- src/error.rs - error::CmdError::is_miss (line 112) stdout ----
        error[E0433]: failed to resolve. Use of undeclared type or module `CmdError`
 --> src/error.rs:113:22
  |
3 | let is_miss = if let CmdError::NoSuchElement(..) = e {
  |                      ^^^^^^^^ Use of undeclared type or module `CmdError`
error[E0425]: cannot find value `e` in this scope
 --> src/error.rs:113:52
  |
3 | let is_miss = if let CmdError::NoSuchElement(..) = e {
  |                                                    ^ not found in this scope

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.