Code Monkey home page Code Monkey logo

webtools's Introduction

WebTools (formerly WebUnit)

Note for existing users of this package: As of March 2, 2018 I am renaming this package from WebUnit to WebTools to avoid naming conflicts with future releases of the Wolfram Language (which will include parts of the WebUnit functionality).

WebTools is a package written in the Wolfram Language which automates interaction with a web browser using the JSON wire protocol.

Supported functionality includes:

  • Starting and stopping a web browser
  • Opening web pages and switching to web pages in different browser tabs
  • Typing in input fields and clicking links
  • Reading web page content and taking screen shots
  • Executing javascript in the current browser session and returning javascript function results

To install this package:

  • Navigate to the 'Releases' area of this repository: https://github.com/arnoudbuzing/webtools/releases
  • Download the most recent WebTools-x.y.z.paclet file from this page
  • Start the Wolfram Language and evaluate the following command (replacing x.y.z with the actual version digits):

PacletInstall["<path to downloads>/WebTools-x.y.z.paclet"]

Then, open the usage.nb notebook to see basic usage for the supported functions.

webtools's People

Contributors

arnoudbuzing avatar erws avatar wjxway avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

webtools's Issues

A tricky solution to Chinese directory names

In the switch case in InstallWebUnit[driver_], I suggest to change the code for windows to:

"Windows-x86-64", 
  If[
    Complement[Characters[dir],Join[CharacterRange["0", "z"], Characters@"-."]]=!={},
    SystemOpen[FileNameJoin[{ dir, "chromedriver.exe" }]],
    process = StartProcess[ $SystemShell ]; 
    WriteLine[ process, "start /b " <> FileNameJoin[{ dir, "chromedriver.exe" }] ]
  ]

Or else Chinese characters in dir would lead to initialization failure.

WebUnit crash after opening certain webpages

Whenever I try to open a webpage with login panel, WebUnit crashes.
Even if I open the webpage manually, after executing arbitrary webpage manipulation program, driver crashes.( Before I execute anything, http://localhost:9515/status returns proper result, however, after execution, which would fail, the localhost webpage becomes unavailable )
I checked this behavior on baidu.com(I'm in China, so Google is unavailable......)/my school bbs/my school course system all result in the crash of WebUnit. What's intriguing is that baidu.com use a pop-up window to display login panel, before I click to show this pop-up window, everything is fine, but after clicking, WebUnit crashes, however, URL of the webpage doesn't change in the progress.
How could this possibly happen and how could I fix this?
Thanks again!

I checked Google, when I open google's login page, WebUnit crashes as well.

Starting Chrome using StartExternalSession with SessionOptions

How do you start chrome using StartExternalSession["WebDriver-Chrome"] with SessionOptions that will disable the message 'Chrome is bring controlled by automated test software". The basic issue is that the automated test window is not accepted as a login browser by certain financial websites. I would like a chrome window that does not use the argument "--enable-automation".

EdgeDriver's high OS dependency

Edge Driver only works under corresponding Windows build. If the build number and Edge Driver's version is not exactly the same, driver would crash. Is it possible to create a function to check driver vapidity and download corresponding driver?

Suggestion: verify connection to browser after InstallWebUnit[]

Today I found an issue where WebUnit would throw mysterious errors. After some digging, it turned out that the system running WebUnit did not have permissions to use the command line, so it could never launch the web driver.

In the code of InstallWebUnit[] there is the line Quiet[URLFetch[webDriverBaseURL<>"/status"]] that checks if the process is already running. Maybe this check should be performed again after launching the driver to check if it really did launch.

Get specified HTML element

It would be great if I can write something like:

GetPageHTML[Id[*******]]
or
GetPageHTML[XPath[*******]]

Now I'll have to do StringCases[GetPageHTML[****],***] which is kind of redundant and of low efficiency.

Also, WebUnit crash sometimes when opening webpage, returning the following info:
"value" /. ImportString[$Failed, "JSON"], and don't work afterward even if I change webpage manually.
How can I fix this?

BTW: I LOVE WebUnit, It's so helpful when dealing with web stuffs, thanks a lot~

Import Source

It would be great to add a function to import the webpage because some pages can be only accessed after logging in and one can't simply use standard Import function to read the page.
I think the most useful would be to get a source of current page.
Thank you in advance!

Download Image using WebUnit

Is it possible for WebUnit to download images on the page?
Note that Fetching URL is not always an option, as sometimes image differs between calls even if the URLs are the same(E.G. verification code images).

Misspelling in `QueryMethod` and `QueryValue`?

I thought it should be:

QueryMethod[ wtId[s_String] ] ^:= s;
QueryMethod[ wtName[s_String] ] ^:= s;
QueryValue[ wtId[s_String] ] ^:= s;
QueryValue[ wtName[s_String] ] ^:= s;

instead of:

QueryMethod[ Id[s_String] ] ^:= s;
QueryMethod[ Name[s_String] ] ^:= s;
QueryValue[ Id[s_String] ] ^:= s;
QueryValue[ Name[s_String] ] ^:= s;

How to catch the full cookies?

How to get cookies from a controlled explorer?

I tried to run some js but missed something important ones.

qq 20180312133206

Then I found that means Http-Only cookies, which can't get from javascript.

So how can I catch them with the Mathematica?

Unable to get CaptureWebPage[] function to work

I am able to navigate to a web page using the WebUnit package, however when I execute the function:

CaptureWebPage[]

it returns the error:

ImportString[{"message" ->
"unknown error: cannot get automation extension
from unknown error: page could not be found:
chrome-extension://aapnijgdinlhnhlmodcfapnahmbfebeb/generated
background_page.html
(Session info: chrome=57.0.2987.133)
(Driver info: chromedriver=2.25.426935
(820a95b0b81d33e42712f9198c215f703412e1a1),platform=Mac OS X 10.10.5
x86_64)"}, "Base64"]


If I have to supply an argument to CaptureWebPage[], what is the correct syntax and data type?
Picture of my test below.
Thanks!
example-webunit

StartWebSession not working?

I installed the new WebTools package, but now StartWebSession won't work......
Also when <<WebTools, a warning is generated which said:
Set::wrsym: Symbol $SupportedWebDrivers is Protected.

Am I installing the package in the wrong way?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.