Comments (9)
jlipps
commented on July 18, 2024
It looks like the issue is that the connection to WDA times out while the classifier is running. I would therefore recommend exploring the XCUITest driver docs to discover whether there is a capability that can modulate that timeout. At any rate nothing in this log looks like a bug in this plugin, so closing here.
from appium-classifier-plugin.
jlipps
commented on July 18, 2024
From https://github.com/appium/appium-xcuitest-driver it looks like wdaConnectionTimeout could be increased. If you're still running into issues, please provide a full log at the Appium issue tracker.
from appium-classifier-plugin.
ericsavoiewaltz
commented on July 18, 2024
Cool thanks
from appium-classifier-plugin.
ericsavoiewaltz
commented on July 18, 2024
I have some appium output here that shows that the WDA losing connection doesn't impact the custom plugin's ability to find an element. You can see it lose connection and find the QR code with a confidence interval of 0.9+ and then fail to find the send icon with a confidence interval of 0.3+. The default threshold is at 0.2.
I don't think this is just solved by the time out. Can you please take another look?
[debug] [W3C (db475602)] Calling AppiumDriver.findElement() with args: ["-custom","ai:qr code","db475602-5ad2-4bd1-8865-c0d8c53a8ca9"]
[debug] [XCUITest] Executing command 'findElement'
[debug] [BaseDriver] Find plugin 'ai' requested; will attempt to use it from 'test-ai-classifier'
[debug] [BaseDriver] Waiting up to 20000 ms for condition
[ai] Retrieving current settings to check element response attributes
[ai] We will need to update settings to include element response attributes
[debug] [JSONWP Proxy] Matched '/appium/settings' to command name 'updateSettings'
[debug] [JSONWP Proxy] Proxying [POST /appium/settings] to [POST http://localhost:8001/session/3C65A71B-BE78-4150-8321-40AC8549680E/appium/settings] with body: {"settings":{"elementResponseAttributes":"rect"}}
[debug] [JSONWP Proxy] Got response with status 200: {"value":{"elementResponseAttributes":"rect","shouldUseCompactResponses":false},"sessionId":"3C65A71B-BE78-4150-8321-40AC8549680E","status":0}
[ai] Setting confidence threshold to default value of 0.2
[ai] Retrieving data for all leaf-node elements on screen
[debug] [BaseDriver] Valid locator strategies for this request: xpath, id, name, class name, -ios predicate string, -ios class chain, accessibility id
[debug] [BaseDriver] Waiting up to 20000 ms for condition
[debug] [JSONWP Proxy] Matched '/elements' to command name 'findElements'
[debug] [JSONWP Proxy] Proxying [POST /elements] to [POST http://localhost:8001/session/3C65A71B-BE78-4150-8321-40AC8549680E/elements] with body: {"using":"xpath","value":"//*[not(child::*)]"}
[debug] [XCUITest] Connection to WDA timed out
[debug] [iProxy] recv failed: Operation not permitted
[debug] [JSONWP Proxy] Got response with status 200: {"value":[{"ELEMENT":"84000000-0000-0000-7401-000000000000","rect":{"y":20,"x":8,"width":55,"height":44}},{"ELEMENT":"85000000-0000-0000-7401-000000000000","rect":{"y":26,"x":125,"width":125,"height":32}},{"ELEMENT":"AB000000-0000-0000-7401-000000000000","rect":{"y":80,"x":-279,"width":50,"height":50}},{"ELEMENT":"AC000000-0000-0000-7401-000000000000","rect":{"y":80,"x":-213,"width":197,"height":21}},{"ELEMENT":"AD000000-0000-0000-7401-000000000000","rect":{"y":108,"x":-213,"width":197,"height":22}},{"ELEMENT":"B1000000-0000-0000-7401-000000000000","rect":{"y":205,"x":-295,"width":295,"height":1}},{"ELEMENT":"B0000000-0000-0000-7401-000000000000","rect":{"y":157,"x":-280,"width":36,"height":36}},{"ELEMENT":"B2000000-0000-0000-7401-000000000000","rect":{"y":146,"x":-229,"width":214,"height":60}},{"ELEMENT":"B5000000-0000-0000-7401-000000000000","rect":{"y":265,"x":-295,"width":295,"height":1}},{"ELEMENT":"B4000000-0000-0000-7401-000000000000","rect":{"y":217,"x":-280,"width":36,"height":36}},{"ELEMENT":"B60...
[ai] Getting window size in case we need to scale screenshot
[ai] Getting screenshot to use for classifier
[debug] [XCUITest] Taking screenshot with WDA
[debug] [JSONWP Proxy] Matched '/screenshot' to command name 'getScreenshot'
[debug] [JSONWP Proxy] Proxying [GET /screenshot] to [GET http://localhost:8001/session/3C65A71B-BE78-4150-8321-40AC8549680E/screenshot] with no body
[debug] [XCUITest] Connection to WDA timed out
[debug] [iProxy] recv failed: Operation not permitted
[debug] [JSONWP Proxy] Got response with status 200: "{\n \"value\" : \"iVBORw0KGgoAAAANSUhEUgAAAu4AAAU2CAIAAABFtaRRAAAAAXNSR0IArs4c6QAA\\r\\nABxpRE9UAAAAAgAAAAAAAAKbAAAAKAAAApsAAAKbAAGayJ52Y+gAAEAASURBVHgB\\r\\n7L1ldFzn1f7txrYYLJOYjGIyyJY00ohli1myLIvB0LRp0zA1aZq0aZ+AHXLiGGJm\\r\\nlGUxMzMzWmZ2DPd77fuMFCVpHft51v\\/LG2XtddY9Z86AlZlzfnPta+89TSSVIZqZ\\r\\nJZqZJ5pRIJqZQyGdK5LOtJM+bydzTiSTKpLOFs0sE82sEPaLp1eKZ5TgAIStVBnC\\r\\nXibHQSbLfmaReHqZ44xKx5mluOkgk24nm4lwkM77WYhl8hHCTmE9eesgm+sgU+gg\\r\\nXUUhm+8gl+Ugl4EQy2dSKFwQy6eL5fLEsqVimWqxdJ1IrkQsV+Aim+4uk+4uneMq\\r\\nVYS77OVK7OWzHeUyXWTy3GeWuElVuMhU2CsU2yrlu8gWusjlO8nnIhzlih3lSh3k\\r\\nKcQKZWKFEieFAieFPGf5AoSrfC4PLBB548FvypU4y5U5y5U7y1XSVr4Ex7so5rgo\\r\\nZTnJ1TrJVboolLooFLoo5mPrJl\\/qJldOIV\\/qrJjvrJTnqJTvqFjsqFTpqFjrMivD\\r\\nZVami3K+i1KRi1KJi2IZbZWKXJXyXZVyXZWzaYu1YpG7Qpm7fIWrYgnCTan0V8Nd\\r\\nuQwx+TBhz9NvPWaVP0WUesySxBqVsmeKtbPLJ0J44MTNJyw851QghAOE9ZO3XnMr\\r\\nEcIxwvrJW+95Vf\\/H8Jlf\\/avhq1rzfww\\/tVoEf5I6P7VGHrV+6lW+GmUIP\\/UKfkCj\\r\\nn2obxfwOf7VWhK9aI8JPowHhr17nr45tk79ai596C+3UrPTTKvXTKsLWX6PSX6NG\\r\\nOCZIvTl...
[ai] Turning screenshot into HTML image for use with canvas
[ai] Screenshot and screen size did not match. Screen size is 375x667 but screenshot size is 750x1334. Scaling screenshot to match screen size
[ai] Getting screenshot slices for each element
[debug] [XCUITest] Connection to WDA timed out
[debug] [iProxy] recv failed: Operation not permitted
[ai] Making label predictions based on element images
[ai] Found 1 matching elements
[ai] Highest confidence of any element for desired label 'qr code' was 0.9949374198913574
[ai] Resetting element response attribute setting to original value: "type,label"
[debug] [JSONWP Proxy] Matched '/appium/settings' to command name 'updateSettings'
[debug] [JSONWP Proxy] Proxying [POST /appium/settings] to [POST http://localhost:8001/session/3C65A71B-BE78-4150-8321-40AC8549680E/appium/settings] with body: {"settings":{"elementResponseAttributes":"type,label"}}
[debug] [JSONWP Proxy] Got response with status 200: {"value":{"elementResponseAttributes":"type,label","shouldUseCompactResponses":false},"sessionId":"3C65A71B-BE78-4150-8321-40AC8549680E","status":0}
[debug] [XCUITest] Connection to WDA timed out
[debug] [iProxy] recv failed: Operation not permitted
[debug] [W3C (db475602)] Responding to client with driver.findElement() result: {"element-6066-11e4-a52e-4f735466cecf":"CD000000-0000-0000-7401-000000000000","rect":{"y":106,"x":31,"width":313,"height":312}}
[HTTP] <-- POST /wd/hub/session/db475602-5ad2-4bd1-8865-c0d8c53a8ca9/element 200 49155 ms - 137
[HTTP]
[HTTP] --> POST /wd/hub/session/db475602-5ad2-4bd1-8865-c0d8c53a8ca9/element
[HTTP] {"using":"-custom","value":"ai:send"}
[debug] [W3C (db475602)] Calling AppiumDriver.findElement() with args: ["-custom","ai:send","db475602-5ad2-4bd1-8865-c0d8c53a8ca9"]
[debug] [XCUITest] Executing command 'findElement'
[debug] [BaseDriver] Find plugin 'ai' requested; will attempt to use it from 'test-ai-classifier'
[debug] [BaseDriver] Waiting up to 20000 ms for condition
[ai] Retrieving current settings to check element response attributes
[ai] We will need to update settings to include element response attributes
[debug] [JSONWP Proxy] Matched '/appium/settings' to command name 'updateSettings'
[debug] [JSONWP Proxy] Proxying [POST /appium/settings] to [POST http://localhost:8001/session/3C65A71B-BE78-4150-8321-40AC8549680E/appium/settings] with body: {"settings":{"elementResponseAttributes":"rect"}}
[debug] [JSONWP Proxy] Got response with status 200: {"value":{"elementResponseAttributes":"rect","shouldUseCompactResponses":false},"sessionId":"3C65A71B-BE78-4150-8321-40AC8549680E","status":0}
[ai] Setting confidence threshold to default value of 0.2
[ai] Retrieving data for all leaf-node elements on screen
[debug] [BaseDriver] Valid locator strategies for this request: xpath, id, name, class name, -ios predicate string, -ios class chain, accessibility id
[debug] [BaseDriver] Waiting up to 20000 ms for condition
[debug] [JSONWP Proxy] Matched '/elements' to command name 'findElements'
[debug] [JSONWP Proxy] Proxying [POST /elements] to [POST http://localhost:8001/session/3C65A71B-BE78-4150-8321-40AC8549680E/elements] with body: {"using":"xpath","value":"//*[not(child::*)]"}
[debug] [XCUITest] Connection to WDA timed out
[debug] [iProxy] recv failed: Operation not permitted
[debug] [XCUITest] Connection to WDA timed out
[debug] [iProxy] recv failed: Operation not permitted
[debug] [JSONWP Proxy] Got response with status 200: {"value":[{"ELEMENT":"84000000-0000-0000-7401-000000000000","rect":{"y":20,"x":8,"width":55,"height":44}},{"ELEMENT":"85000000-0000-0000-7401-000000000000","rect":{"y":26,"x":125,"width":125,"height":32}},{"ELEMENT":"CD000000-0000-0000-7401-000000000000","rect":{"y":106,"x":31,"width":313,"height":312}},{"ELEMENT":"CE000000-0000-0000-7401-000000000000","rect":{"y":468,"x":26,"width":323,"height":41}},{"ELEMENT":"D0000000-0000-0000-7401-000000000000","rect":{"y":517,"x":116,"width":143,"height":142}},{"ELEMENT":"D8000000-0000-0000-7401-000000000000","rect":{"y":64,"x":0,"width":375,"height":603}},{"ELEMENT":"AB000000-0000-0000-7401-000000000000","rect":{"y":80,"x":16,"width":50,"height":50}},{"ELEMENT":"AC000000-0000-0000-7401-000000000000","rect":{"y":80,"x":82,"width":197,"height":21}},{"ELEMENT":"AD000000-0000-0000-7401-000000000000","rect":{"y":108,"x":82,"width":197,"height":22}},{"ELEMENT":"B1000000-0000-0000-7401-000000000000","rect":{"y":205,"x":0,"width":295,"height":1}},{"ELEMENT":"B0000000-0000-0...
[ai] Getting window size in case we need to scale screenshot
[ai] Getting screenshot to use for classifier
[debug] [XCUITest] Taking screenshot with WDA
[debug] [JSONWP Proxy] Matched '/screenshot' to command name 'getScreenshot'
[debug] [JSONWP Proxy] Proxying [GET /screenshot] to [GET http://localhost:8001/session/3C65A71B-BE78-4150-8321-40AC8549680E/screenshot] with no body
[debug] [JSONWP Proxy] Got response with status 200: "{\n \"value\" : \"iVBORw0KGgoAAAANSUhEUgAAAu4AAAU2CAIAAABFtaRRAAAAAXNSR0IArs4c6QAA\\r\\nABxpRE9UAAAAAgAAAAAAAAKbAAAAKAAAApsAAAKbAAL\\/ghKz6t0AAEAASURBVHgB\\r\\n7L11eFTn1v7PKRAXgsUFjxsSSDLJxEPchRDiArQ9Pdaeek\\/l9HgPhRotUCjuGkLc\\r\\n3d1dCe7Fnt+9nj1JU6Pwvtf7\\/ePXcK3rufbs2ZmZhJm9P3Ove601TSSVIZqZJZqZ\\r\\nJ5pRIJqZQyGdK5LOtJU+bytzTiSTKpLOFs0sE82sEPaLp1eKZ5TgAISNVBnCTibH\\r\\nXibLbmaReHqZw4xKh5mluGkvk24rm4mwl877QYhl8hHCTmF78movm2svU2gvXUUh\\r\\nm28vl2Uvl4EQy2dSKFwQy6eL5fLEsqVimWqxdJ1IrkQsV+Asm+4mk+4mneMiVYS7\\r\\n7ORK7OSzHeQynWXy3GaWuEpVOMtU2CkU2yjlO8sWOsvlO8rnIhzkih3kSu3lKcQK\\r\\nZWKFEkeFAkeFPCf5AoSLfC4PbCDyxoPflCtxkitzkit3kqukVb4Exzsr5jgrZTnK\\r\\n1TrKVTorlDorFDor5mN1lS91lSunkC91Usx3UspzUMp3UCx2UKp0UKx1npXhPCvT\\r\\nWTnfWanIWanEWbGMVqUiF6V8F6VcF+VsWrGtWOSmUOYmX+GiWIJwVSr9xXBTLkNM\\r\\nPkzY8\\/Sr+6zyp4hS91mSWKtS9kzhMbt8IoQfnLj5hA3PORUI4QBh+8mr19xKhHCM\\r\\nsP3k1Xte1f8yfOZX\\/2L4qtb8L8NPrRbBH6TOT7WFQq2RdqpX8aj+blujDHv81ev8\\r\\n1Zv85vX6zev2V2vFtr9GtSRwl1oL7cSK\\/eoNdLBGDcIPP6LREKBRjwhSbwxSbw5W\\r\\naw5Wrw\\/...
[ai] Turning screenshot into HTML image for use with canvas
[ai] Screenshot and screen size did not match. Screen size is 375x667 but screenshot size is 750x1334. Scaling screenshot to match screen size
[ai] Getting screenshot slices for each element
[debug] [XCUITest] Connection to WDA timed out
[debug] [iProxy] recv failed: Operation not permitted
[ai] Making label predictions based on element images
[ai] Found 0 matching elements
[ai] Highest confidence of any element for desired label 'send' was 0.3476186692714691
[ai] Resetting element response attribute setting to original value: "type,label"
from appium-classifier-plugin.
jlipps
commented on July 18, 2024
@ericsavoiewaltz I'd need to see the full log to comment any further. This log doesn't show the response to the client, for example.
from appium-classifier-plugin.
ericsavoiewaltz
commented on July 18, 2024
https://gist.github.com/ericsavoiewaltz/f35b235c46769c79a613597b87ce255a
from appium-classifier-plugin.
jlipps
commented on July 18, 2024
OK, thanks, will reopen and investigate more when I have a bit of time.
from appium-classifier-plugin.
jlipps
commented on July 18, 2024
@ericsavoiewaltz can you retry with v 2.1 of the classifier plugin, and let me know if this is still happening?
from appium-classifier-plugin.
pramodKarande
commented on July 18, 2024
@cnavrides @jlipps
while training , do we need to add images in specific resolution and format??
from appium-classifier-plugin.
Related Issues (20)
- trying to install appium classifer on mac getting error HOT 1
- Error: undefined symbol: TF_Version HOT 1
- node: symbol lookup error: /usr/local/lib/node_modules/test-ai-classifier/build/Release/test-ai-classifier.node: undefined symbol: TF_Version HOT 5
- ERROR： [email protected] install error->node-gyp rebuild,No such file or directory HOT 2
- ERROR: [email protected] run "Illegal instruction: 4"
- ERROR： [email protected] run "Illegal instruction: 4"
- Start Classifier server only ，run ERROR：test-ai-classifier: command not found HOT 3
- How to expand labels in the "lib/labels.js" file?
- It can take more than 20 seconds to find only one element. How to make it quicker?
- Getting error while installing test-ai-classifier in windows machine HOT 2
- Prediction result array had 107 elements but labels list had 106 elements. They need to match. (GRPC::Unknown)
- install error: fatal error: '../node_modules/@tensorflow/tfjs-node/deps/include/tensorflow/c/c_api.h' file not found HOT 3
- Error while installing the plugin : Error: [email protected] postinstall: `node ./postinstall.js`
- Model Returns no match/wrong matches via classifier client but identifies the image correctly when tested stand alone
- org.openqa.selenium.WebDriverException:Original error: Invalid format for the 'customFindModules' capability. It should be an object with keys corresponding to the short names and values corresponding to the full names of the element finding plugins HOT 1
- Questions - Please could you clarify these questions HOT 1
- The module '/usr/local/lib/node_modules/test-ai-classifier/node_modules/canvas/build/Release/canvas.node'
- [Question] Are there any plans to develop a C# client? HOT 2
- error: Element had a width or height of zero; cannot slice such an image
- Can we this for mobile browser in Javascript?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from appium-classifier-plugin.