Unable to run tests using Object Detect mode about appium-classifier-plugin HOT 7 OPEN

The attached screenshot is taken by the plugin and saved at
/usr/local/lib/node_modules/test-ai-classifier/build-js/lib

Desired Capabilities used -
HashMap<String, String> customFindModules = new HashMap<>();
customFindModules.put("ai", "test-ai-classifier");

    capabilities.setCapability("customFindModules", customFindModules);
    capabilities.setCapability("testaiFindMode", "object_detection");
    capabilities.setCapability("testaiObjectDetectionThreshold", 0.2);
    capabilities.setCapability("shouldUseCompactResponses", false);

Tried below functions to find the element in test case -
getAppDriver().findElementByCustom("search").click();
getAppDriver().findElement("-custom","ai:search").click();
getAppDriver().findElement(MobileBy.custom("ai:search")).click();

from appium-classifier-plugin.

two ideas:

1. did you try with normal mode (not object detection mode?)
2. did you try decreasing the threshold even further (make it 0.01 just to see what happens)?

from appium-classifier-plugin.

Thanks a lot @jlipps for your expert comments. I retried the tests & outcome are below -

1. did you try with normal mode (not object detection mode?) -
-- Yes, it worked with Element Lookup with value upto 0.3, but not able to find icon/element if its more than 0.4.

2. did you try decreasing the threshold even further (make it 0.01 just to see what happens)? -
-- Even after decreasing it to 0.01 in object detection mode, the plugin was not able to identify the search icon/element.

[ai] Will use object detection method for finding elements via AI
[ai] Setting confidence threshold to default value of 0.01
[ai] Getting window size in case we need to scale screenshot
[debug] [WD Proxy] Matched '/window/current/size' to command name 'getWindowSize'
[debug] [WD Proxy] Proxying [GET /window/current/size] to [GET http://localhost:8204/wd/hub/session/fd85ade4-f2d2-491e-9c01-fdd337788d6b/window/current/size] with body: {}
[debug] [WD Proxy] Got response with status 200: "{"sessionId":"fd85ade4-f2d2-491e-9c01-fdd337788d6b","status":0,"value":{"height":1504,"width":2560}}"
[ai] Getting screenshot to use for classifier
[debug] [WD Proxy] Matched '/screenshot' to command name 'getScreenshot'
[debug] [WD Proxy] Proxying [GET /screenshot] to [GET http://localhost:8204/wd/hub/session/fd85ade4-f2d2-491e-9c01-fdd337788d6b/screenshot] with no body
[debug] [WD Proxy] Got response with status 200: "{"sessionId":"fd85ade4-f2d2-491e-9c01-fdd337788d6b","status":0,"value":"iVBORw0KGgoAAAANSUhEUgAACgAAAAZACAYAAAC1rBvAAAAABHNCSVQICAgIfAhkiAAAIABJREFU\neJzUfXucX1V17\/c3Mwl5P0kIhISENwIKiqJoUbBV0baiaAGtQLXWeqtotdbqRUW04qNXW2it1FqL\ntFfQquD7dcH6QBAV0ICACAkJIQ\/ymDwmk8nMnPtHcn7u2bPXc+9zZrr48Mnvd87aa62993p8187J\n+XXOvXxRBSV10Ol+rlB1r9WfuXESj5W8MsNx9ecm7EvpmwxyJHnx9RQft261j0gyNLZxsiwyuTmU\nJGrtYp8DUGw9JBu81GRMePWWuketpzTnybq2JfOYJba1OaTEPQufJV5ydeXKLGGjV3cOXyqmAP08\nrHbk2F1yD3P8dyL80aszJ6fk1LgSlIsNLfZTvE35t5ZKYTOtv1OfOd0a\/V574mvWWp+rvwn5TVFT\nNSFHd0ksMxkpt3\/x8Le5pp467FkTKqfF1yzkwT6xLfHn1J85lJKfS9qznJq0ujVnCvG9pnqxUnIt\nc8qVpfXnpuaWy58Tj1xNLY2dLDnLihVLr2kpsvqxhLG0a+LJGx4ZGmq7B691evpazX5Z1zYHt1J1\nLnXN0zdzfQN13XOtBHbXkjcXWPOft+eaiHiQyNqTa9bJ65M59nrjrum+U6PHIgMY22uH3626NTXY\nmkNrezw1vwSVxIvUPY9ftJEH26zrbePFkjImQyxzMnPWMxcXeG3WfNeMLdn\/ae2W+HJ6DMvYJnqZ\n0phE4i+9tm31ayV0Sn2atbfV2Fe6rrS1L97+yiKzVG3W4JtSWMpiq8V2aaxVdxv7x9lcci4leWv+\n3hOeM+ty9YhocEmjUvLCa1p9E01eO6Vx8f0cPZokoJEzmQ4IOGpqT7z8Ob7c...
[ai] Turning screenshot into HTML image for use with canvas
[ai] Writing screenshot to /usr/local/lib/node_modules/test-ai-classifier/build-js/lib/Tue, 24 Dec 2019 20:24:27 GMT_classifiertest.png for use in object detection
[ai] Gathering object candidates from tensorflow native
2019-12-25 01:54:27.877062: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /usr/local/lib/node_modules/test-ai-classifier/model/obj_detection_model
2019-12-25 01:54:28.070844: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2019-12-25 01:54:28.569526: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2019-12-25 01:54:28.569581: I tensorflow/cc/saved_model/loader.cc:212] The specified SavedModel has no variables; no checkpoints were restored. File does not exist: /usr/local/lib/node_modules/test-ai-classifier/model/obj_detection_model/variables/variables.index
2019-12-25 01:54:28.569605: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 692541 microseconds.
[ai] Candidates retrieved
[ai] Making label predictions based on element images
This model execution did not contain any nodes with control flow or dynamic output shapes. You can use model.execute() instead.
[ai] Predictions for 47 element(s) took 0.944 seconds
[ai] Found 0 matching elements
[ai] Registering image elements with driver
[debug] [W3C (15e065c4)] Encountered internal error running command: NoSuchElementError: An element could not be located on the page using the given search parameters.
[debug] [W3C (15e065c4)] at AndroidUiautomator2Driver.findByCustom (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/commands/find.js:176:13)
[HTTP] <-- POST /wd/hub/session/15e065c4-730d-4a84-bec7-04882cfccb46/element 404 18848 ms - 397
[HTTP]

3. Another issue which I see is that even if I set the value in desired capability as below, the plugin always takes default value from classifier.js file inside (/usr/local/lib/node_modules/test-ai-classifier/build-js/lib). So I have to modify the classifier.js file everytime I need to change the threshold value. Am I missing something here ?? ->
E.g.
capabilities.setCapability("testaiObjectDetectionThreshold", "0.3");
const DEFAULT_CONFIDENCE_THRESHOLD = 0.2;

from appium-classifier-plugin.

Hi @jlipps - Appreciate if you can guide further on the above 3 result set, so that I can explore this plugin use cases in our project.

from appium-classifier-plugin.

hi @jlipps -

Could you also please suggest on how to integrate this plugin while running our tests on Sauce labs.

from appium-classifier-plugin.

hi @jlipps -

Could you also please suggest on how to integrate this plugin while running our tests on Sauce labs.

The above question is raised separately and closed.
#30

from appium-classifier-plugin.

@jlipps - Please confirm on above 3 points output as per your suggestion and what should be the next debugging steps ?

from appium-classifier-plugin.